Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Already on GitHub? Sign in to your account

no sharding support #304

Closed
tecbot opened this Issue May 7, 2012 · 13 comments

Comments

Projects
None yet
8 participants
Contributor

tecbot commented May 7, 2012

the current version doesn't support sharding.
We need a new "Shard" key and add it to all update queries of a document.

See: http://www.mongodb.org/display/DOCS/Sharding+Limits
"All (non-multi)updates, upserts, and inserts must include the current shard key. This may cause issues for anyone using a mapping library since you don't have full control of updates."

Contributor

j commented May 31, 2012

+9999!

Contributor

tecbot commented Jun 1, 2012

@jstout24 i will implement this next week

Contributor

tystr commented Jun 1, 2012

yay! +1

+1 to this. If the pull request which implements it works, it should be merged as sharding is one of the key features of MongoDB.

lotar commented Jun 12, 2013

Any update on this one? Need help maybe?

Contributor

tecbot commented Jun 13, 2013

Because of this incomplete Pull Request: #511 I have not worked on this anymore.

Currently we use in our app a simply workaround for sharded collections. If a object should be persists/update than we check first if the object already persists in the database. If not we use the simply function of the document manager. If its already persist we use the following implementation to update a object:

$qb = $this->repository->createQueryBuilder();
$this->applyCriteria($qb, array_merge(array('id' => $document->getId()), $this->getShardCriteria($document)));

$this->dm->getUnitOfWork()->computeChangeSet($this->repository->getClassMetadata(), $document);
$changeSet = $this->dm->getUnitOfWork()->getDocumentChangeSet($document);

//don't save unchanged objects since they will be empty after execute
if (empty($changeSet)) {
    return;
}

foreach ($changeSet as $key => $changes) {
    $qb->field($key)->set($changes[1]);
}

$qb->update()->getQuery($options)->execute();

And every shardable manager will implement getShardCriteria. e.g to shard on an user:

protected function getShardCriteria($document)
{
    return array(
        'user' => $document->getUser(),
    );
}

The applyCriteria method is a simple function which walks over the array and add it to the query builder.
I hope this helps you, if you do not want to wait any longer.

lotar commented Jun 14, 2013

Thanks for the input

lotar commented Jun 14, 2013

@tecbot I've actually forked mongodb-odm repo and supported "full native" sharding using additional annotation "ShardKey" on document.

It still prolly needs some cleanup and maybe few more tests / bugfixes but for now it suits my needs for now.

If you think it's close enough, I might modify and test it a bit more and contribute to the original project.

Reference:
https://packagist.org/packages/backplane/mongodb-odm
lotar/mongodb-odm@86a7645

Contributor

tecbot commented Jun 15, 2013

@lotar I'm missing the query part in your implementation otherwise mongos needs do query all shards if you load a document. You need also implement it for removes, upserts, collections/proxy loading etc. and while this pr #511 is changing a lot you should wait for it to implement this otherwise you will rewrite many stuff ;)

lotar commented Jun 15, 2013

@tecbot for update, query parameter get's shard keys merged in here: lib/Doctrine/ODM/MongoDB/Persisters/DocumentPersister.php:276 (https://github.com/lotar/mongodb-odm/blob/86a7645855e8c1c80ed2aec173845d95d70b5c0c/lib/Doctrine/ODM/MongoDB/Persisters/DocumentPersister.php), rest of the implementation can be done in similar manner fairly quick.

I believe it wouldn't take me more than few days to get this thing done - to cover everything, but don't want to spend time on it if you're going to implement something different anytime soon.

On the final note - would really like to help with this since it's one of the core reasons why people are using Mongo in the first place.

Contributor

tecbot commented Jun 15, 2013

Currently I have no plan to implement this, because our simple workaround which I have posted works for us good and I have currently no time to contribute something expect bug fixes and improvements.

@alcaeus alcaeus added the has PR label Mar 14, 2016

@alcaeus alcaeus added this to the 1.1 milestone Mar 14, 2016

Member

malarzm commented May 30, 2016

Sharding is now supported thanks to #1385

@malarzm malarzm closed this May 30, 2016

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment