Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Already on GitHub? Sign in to your account

make mongos batch inserts into larger chunks #912

Closed
leifwalsh opened this Issue Jan 23, 2014 · 1 comment

Comments

Projects
None yet
1 participant
Contributor

leifwalsh commented Jan 23, 2014

In 1.4.0, the mongos insert batch grouping algorithm takes intervals of consecutive documents from the insert batch that all have the same shard key, and inserts each of those intervals separately. With a hashed shard key (or a write batch with high entropy in the shard key), these intervals are tiny in expectation, compared to the total size of the insert batch.

Instead, we should divide the batch fully into separate groups, and then insert those groups, so that the batches sent to each shard are all bigger and amortize away more of the network/locking/transaction overhead.

For extra credit, we could parallelize sending the batches to their respective mongods.

@ghost ghost assigned leifwalsh Jan 23, 2014

@leifwalsh leifwalsh modified the milestones: 1.5.0, 1.4.1 Feb 28, 2014

Contributor

leifwalsh commented May 7, 2014

this is done by 2.6 sharding code and will be resolved by merging that code

@leifwalsh leifwalsh closed this May 7, 2014

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment