Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support shard key derivation [DATAMONGO-2341] #3197

Closed
spring-projects-issues opened this issue Aug 11, 2019 · 6 comments
Closed

Support shard key derivation [DATAMONGO-2341] #3197

spring-projects-issues opened this issue Aug 11, 2019 · 6 comments

Comments

@spring-projects-issues
Copy link

@spring-projects-issues spring-projects-issues commented Aug 11, 2019

Lars Duelfer opened DATAMONGO-2341 and commented

MongoDB suggests to use the shard key in all queries that are targeting a sharded collection. This also applies for the query section in an update command. Those are - for example - created for the "save" method of a MongoRepositoy.

Azure CosmosDB (used with the MongoDB API) even requires the shard key in every query section of an update command targeting a sharded collection. The command fails, if the shard key is not present.

With this improvement, we suggest to support the shard key in MongoRepository (or an enhanced interface thereof, including an enhanced version of the SimpleMongoRepository) so that every command includes the shard key and adds it to the update command, for example, sent to the database.

Further ideas are required where the shard key is specified. An idea would be to add it to the @Document annotation.

Discussion around this is very welcome. If we have a chance (timewise), we will try to think about a solution design for this the next weeks and propose it here as part of this ticket


Issue Links:

  • DATAMONGO-2463 Save operation on shard collection fails due to missing inclusion of shard key in replaceOne filter section

  • DATAMONGO-1961 Add shard in the @Document annotation

Referenced from: pull request #833

@spring-projects-issues
Copy link
Author

@spring-projects-issues spring-projects-issues commented Aug 13, 2019

Mark Paluch commented

Azure CosmosDB contains several specifics that are not part of the vanilla MongoDB server. In Spring Data MongoDB, we don't have even have the means to express the sharding key. Why don't you use Spring Data for Azure CosmosDB?

Generally spoken, we do not include specifics of alike data stores but stay pure with what the actual database provides/requires for which a specific Spring Data module was built

@spring-projects-issues
Copy link
Author

@spring-projects-issues spring-projects-issues commented Aug 13, 2019

Lars Duelfer commented

I must say, I am a bit surprised about the way that such an improve suggestion is handled. From an open-source project, where many people can and should contribute, I would have at least expected an open-minded discussion.

This improvement was NOT only suggested for the usage with CosmosDB. While we were using this, we just got aware of this fact. The documentation of MongoDB also highly suggests, that all queries for sharded collections to include the shard key. And this makes totally sense so that the server does not have to fan-out the queries

Taken from the following page:

[https://docs.mongodb.com/manual/sharding/]

 

If queries do not include the shard key or the prefix of a compound shard key, mongos performs a broadcast operation, querying all shards in the sharded cluster. These scatter/gather queries can be long running operations.

Since the command issued to the database for a save() method of a MongoRepository also contains a query section, it would from my perspective also makes sense to support this concept for pure usage with MongoDB.

So discussing this point is by no means specific to CosmosDB although it would help users of it a lot, too

@spring-projects-issues
Copy link
Author

@spring-projects-issues spring-projects-issues commented Aug 13, 2019

Mark Paluch commented

Thanks for clarifying. It wasn't clear from the description that this ticket aims for a general improvement but sounded rather specific to Azure CosmosDB and we generally don't want to pick up specifics of derivate products.

Care to elaborate for operations the shard key would be suitable? Right now, only the save method seems affected

@spring-projects-issues
Copy link
Author

@spring-projects-issues spring-projects-issues commented Aug 13, 2019

Lars Duelfer commented

That I understand, of course.

As we now go along with our implementation in our current project (we are currently implementing a CustomRepository as an extension of the SimpleMongoRepository), I will collect which operations are affected and post them here. I will also think about possible solution so we can discuss.

Is this ok for you? Thanks for re-opening this!

@spring-projects-issues
Copy link
Author

@spring-projects-issues spring-projects-issues commented Sep 2, 2019

Lars Duelfer commented

The save and delete methods are effected. Both would benefit from supporting the shard key. The findBy... whatever methods can include the shard key using the normal spring data conventions

@spring-projects-issues
Copy link
Author

@spring-projects-issues spring-projects-issues commented Feb 10, 2020

Christoph Strobl commented

Lars Duelfer we created a draft for initial shard support via PR #833 in case you wan't to have a look

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
2 participants