api: reducing db calls in repo list endpoints with quota enabled (PROJQUAY-6895)

[bcaton85]

Reducing the number of DB calls in the repo list endpoint with quota enabled by:

• Adding the id to RepositoryBaseElement when the repositories are initially fetched, removing the need to fetch the repository ID's again
• Fetching the repository sizes with a single DB call using the IN operator

[codecov]

Codecov Report

Attention: Patch coverage is 87.50000% with 3 lines in your changes are missing coverage. Please review.

Project coverage is 70.93%. Comparing base (bf766a8) to head (a9672f1).

Files	Patch %	Lines
data/model/repository.py	84.61%	1 Missing and 1 partial ⚠️
endpoints/api/repository_models_pre_oci.py	88.88%	0 Missing and 1 partial ⚠️

Additional details and impacted files

@@           Coverage Diff           @@
##           master    #2770   +/-   ##
=======================================
  Coverage   70.93%   70.93%           
=======================================
  Files         436      436           
  Lines       40701    40715   +14     
  Branches     5315     5320    +5     
=======================================
+ Hits        28870    28881   +11     
- Misses      10148    10152    +4     
+ Partials     1683     1682    -1     

Flag	Coverage Δ
unit	70.93% <87.50%> (+<0.01%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[kleesc]

What is this function used for, wouldn't we want a way to paginate this instead?
Are the repo_ids restricted to a single namespace or is this a full table lookup?

[bcaton85]

I could add pagination but for this particular use case it uses what ever the page size is for listing repositories. The repo_ids will belong to a single namespace the majority of the time but it's possible that they belong to separate namespaces.

I could also replace this with individual requests for fetching each repository size but it looked like this gave better performance.

[kleesc]

lgtm

[bcaton85]

/cherry-pick redhat-3.9

[openshift-cherrypick-robot]

@bcaton85: #2770 failed to apply on top of branch "redhat-3.9":

Applying: reducing db calls in repo list endpoints with quota enabled
Using index info to reconstruct a base tree...
M	data/model/repository.py
M	data/model/test/test_repository.py
M	endpoints/api/repository.py
M	endpoints/api/repository_models_interface.py
M	endpoints/api/repository_models_pre_oci.py
A	endpoints/api/test/test_repository_models_pre_oci.py
Falling back to patching base and 3-way merge...
CONFLICT (modify/delete): endpoints/api/test/test_repository_models_pre_oci.py deleted in HEAD and modified in reducing db calls in repo list endpoints with quota enabled. Version reducing db calls in repo list endpoints with quota enabled of endpoints/api/test/test_repository_models_pre_oci.py left in tree.
Auto-merging endpoints/api/repository_models_pre_oci.py
CONFLICT (content): Merge conflict in endpoints/api/repository_models_pre_oci.py
Auto-merging endpoints/api/repository_models_interface.py
Auto-merging endpoints/api/repository.py
CONFLICT (content): Merge conflict in endpoints/api/repository.py
Auto-merging data/model/test/test_repository.py
CONFLICT (content): Merge conflict in data/model/test/test_repository.py
Auto-merging data/model/repository.py
error: Failed to merge in the changes.
hint: Use 'git am --show-current-patch=diff' to see the failed patch
Patch failed at 0001 reducing db calls in repo list endpoints with quota enabled
When you have resolved this problem, run "git am --continue".
If you prefer to skip this patch, run "git am --skip" instead.
To restore the original branch and stop patching, run "git am --abort".

In response to this:

/cherry-pick redhat-3.9

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[bcaton85]

/cherry-pick redhat-3.11

[openshift-cherrypick-robot]

@bcaton85: #2770 failed to apply on top of branch "redhat-3.11":

Applying: reducing db calls in repo list endpoints with quota enabled
Using index info to reconstruct a base tree...
M	endpoints/api/repository.py
Falling back to patching base and 3-way merge...
Auto-merging endpoints/api/repository.py
CONFLICT (content): Merge conflict in endpoints/api/repository.py
error: Failed to merge in the changes.
hint: Use 'git am --show-current-patch=diff' to see the failed patch
Patch failed at 0001 reducing db calls in repo list endpoints with quota enabled
When you have resolved this problem, run "git am --continue".
If you prefer to skip this patch, run "git am --skip" instead.
To restore the original branch and stop patching, run "git am --abort".

In response to this:

/cherry-pick redhat-3.11

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.