[Backport to 5.7] postgres performance improvements - Write path #6717
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
in key is omitted the bucket is listed and the object list is iterated with getObject Signed-off-by: Danny Zaken <dannyzaken@gmail.com> (cherry picked from commit 19dffca)
Signed-off-by: Danny Zaken <dannyzaken@gmail.com> (cherry picked from commit 2cf4050)
* Added option to enable query logging. this will print for each query
the time it took and if enabled the EXPLAIN result of the query
* fixes in encode_json and handle_ops_encoding
* encode mongo id as string to match indexes
* handle multiple ops in one query
* handle comparison ops
* in md_store - pass $exists in some queries to match indexes
* in system_server - cache result of list_undeletable_buckets
to avoid calling it on each read system
* postgres_client - fix sorts to use indexes
* access _id as text
* remove NULLS FIRST\LAST and use default as we do in the indexes
* added datachunk index by _id descending for iterate_all_chunks
* added $exists in MDStore.load_parts_objects_for_chunks
Signed-off-by: Danny Zaken <dannyzaken@gmail.com>
encode mongo.objectid as string in postgres_client encode_json
when mongo object id is passed as object to imongo-to-postgres lib, it will convert the query to access data->_id as a jsonb struct. this causes queries to not hit the indexes
Signed-off-by: Danny Zaken <dannyzaken@gmail.com>
added {$exists: true} to chunks query in load_blocks_for_chunks
this will cause the postgres query to match the filter in the index
Signed-off-by: Danny Zaken <dannyzaken@gmail.com>
added $exists for dedup_keys in find_chunks_by_dedup_key
* without the $exists the query does not hit the index
Signed-off-by: Danny Zaken <dannyzaken@gmail.com>
modified handle_ops_encoding to handle multiple ops
Signed-off-by: Danny Zaken <dannyzaken@gmail.com>
fixed find_all_parts_of_object to hit index
* added {$exists: true } to obj
* fixes handle_ops_encoding for comparison operators ($eq etc.)
Signed-off-by: Danny Zaken <dannyzaken@gmail.com>
cache results of bucket_server.list_undeletable_buckets() in read_system
Signed-off-by: Danny Zaken <dannyzaken@gmail.com>
db query log changes
Signed-off-by: Danny Zaken <dannyzaken@gmail.com>
Update postgres_client.js
Signed-off-by: Danny Zaken <dannyzaken@gmail.com>
use {disableContainmentQuery: true} option when calling mongo_to_pg
Signed-off-by: Danny Zaken <dannyzaken@gmail.com>
more postgres improvements
* postgres_client - fix sorts to use indexes
* access _id as text
* remove NULLS FIRST\LAST and use default as we do in the indexes
* added datachunk index by _id descending for iterate_all_chunks
* added $exists in MDStore.load_parts_objects_for_chunks
Signed-off-by: Danny Zaken <dannyzaken@gmail.com>
mark some indexes as postgres only and skip in mongo_client
Update postgres_client.js
Signed-off-by: Danny Zaken <dannyzaken@gmail.com>
Update md_store.js
(cherry picked from commit e5f4b6b)
Signed-off-by: Danny Zaken <dannyzaken@gmail.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Explain the changes
Issues: Fixed #xxx / Gap #xxx
Testing Instructions: