Create ability to shard a table #10

Closed
gburgett opened this Issue Dec 27, 2012 · 1 comment

Projects

None yet

1 participant

@gburgett
Owner

Sharding is where one table is spread across multiple files, based on the values of an index. An easy-enough way to do this would be to have a ShardingEngine, which manages one sub-engine per shard file.

The ShardingEngine would spin up and spin down the sub engines using the same code the Database uses to detect when an engine transition is necessary. It would call into the appropriate engine based on its sharding strategy, which can be one of the following not-exhaustive list:

  1. Fixed Range
    • A RangeProvider would determine the high and low values for each Range of an IComparable key (based on an index). Each range would map to a sub engine, and the ShardingTable would redirect to the appropriate engine based on what range the value fits in, spinning it up if necessary.
  2. Balancing
    • A Balancer would determine the number of files (and therefore engines) necessary for the total number of rows. As the table grows, the Balancer would occasionally transfer rows from one sub-engine to another in order to balance them. The Balancer would then also update the ranges associated with each engine.
    • A fixed-number balancer would be the easiest to begin with, but we could improve this to add more balancing strategies.
  3. Other
    • some other sharding strategy we may come up with.
@gburgett gburgett added a commit that referenced this issue Jan 10, 2013
@gburgett working on #10 - sharded collections
Signed-off-by: gburgett <gordon.burgett@gmail.com>
86113f9
@gburgett gburgett was assigned Jan 12, 2013
@gburgett gburgett closed this Jan 12, 2013
@gburgett gburgett reopened this Jan 12, 2013
@gburgett
Owner

As the other issue states, this task is going to become just about sharding by ID.

@gburgett gburgett added a commit that referenced this issue Jan 12, 2013
@gburgett refs #10 - working on sharding
Signed-off-by: gburgett <gordon.burgett@gmail.com>
eb4fe8d
@gburgett gburgett added a commit that referenced this issue Jan 13, 2013
@gburgett continuing to work on #10
Signed-off-by: gburgett <gordon.burgett@gmail.com>
d4b6979
@gburgett gburgett added a commit that referenced this issue Jan 15, 2013
@gburgett Working on #10, added ValueSet.java which will be a set of intervals …
…describing the dissection of a query.

Signed-off-by: gburgett <gordon.burgett@gmail.com>
6a18f4e
@gburgett gburgett added a commit that referenced this issue Jan 15, 2013
@gburgett Still working on #10, ought to have it done wednesday.
Signed-off-by: gburgett <gordon.burgett@gmail.com>
73023ec
@gburgett gburgett added a commit that closed this issue Jan 19, 2013
@gburgett Sharding works!!!! Fixes #10
Signed-off-by: gburgett <gordon.burgett@gmail.com>
88840d0
@gburgett gburgett closed this in 88840d0 Jan 19, 2013
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment