Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fixed map_common_prefixes queries #7040

Merged
merged 1 commit into from
Aug 18, 2022
Merged

Conversation

dannyzaken
Copy link
Contributor

Signed-off-by: Danny Zaken dannyzaken@gmail.com

Explain the changes

  • Changed pl/sql function map_common_prefixes to fetch all required entries in one call.
  • instead of querying for 1000 keys and then extracting the common prefixes from it (can be very few common prefixes if there are a lot of objects in that "dir"), it iterates over the objectmds until reaching the max keys or the end.
  • To avoid unnecessary scanning of objects - if we find a common prefix, we set the marker to be the next possible value for a common prefix (by incrementing the last character by 1)
  • These changes rely on the DB having collation of LC_COLLATE = 'C'. it will not work on DBs with utf8 collation

Issues: Fixed #xxx / Gap #xxx

  1. More cleanup can be done after this change. for example, it is probably unnecessary to redo the list_object queries in the object_server, since now it should return all of the required count
    while (!state.done) {
    const results = await MDStore.instance().list_objects(state);
    _list_add_results(state, results);
    }

Testing Instructions:

  • Doc added/updated
  • Tests added

@dannyzaken dannyzaken force-pushed the danny-list-func branch 3 times, most recently from 88b1cae to 62b62c7 Compare August 11, 2022 05:33
Copy link
Contributor

@liranmauda liranmauda left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@dannyzaken Have we tested Upgrade?
In the past, we had issues when we changed sql_functions

Copy link
Contributor

@liranmauda liranmauda left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Other then the comment, LGTM

* changed pl/sql function map_common_prefixes to fetch all required entries in one call.
* instead of querying for 1000 keys and then extracting the common prefixes from it (can be very few common prefixes if there are a lot of objects in that "dir"), it iterates over the objectmds untils reaching to the max keys or the end.
* To avoid unnecessary scanning of objects - if we find a common prefix, we set the marker to be the next possible value for a common prefix (by incrementing the last charecter by 1)
* These changes rely on the DB having collation of LC_COLLATE = 'C'. it will not work on DBs with utf8 collation

Signed-off-by: Danny Zaken <dannyzaken@gmail.com>
@baum baum merged commit 4233abe into noobaa:master Aug 18, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants