-
Notifications
You must be signed in to change notification settings - Fork 107
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Export of large amount of packages will result in errors #5375
Comments
Actually Django still uses the client-side cursors by default. |
Just one more slight clarification of what was already said above: The bug triggers (in the first version) as soon as the export includes more than ~65k artifacts. This was observed for both deb and rpm content exports. In the Katello context, it is enough for all the repos in a content view combined to add up to those more than ~65k artifacts and then export that content view version. The second version of the bug occurs if there are less than ~65k artifacts but more than ~65k content units. For deb content there tends to be roughly twice as much content as actual artifacts because for each .deb package there is also a "package release component" content. |
I ran into the same issue with rpm with about ~40k artifacts in the export so its still an issue here too |
Does this issue mean we somehow need to batch the |
What happens if you tweak the |
@dralley We are trying out your suggestion (in a somewhat hackish way) here: https://github.com/ATIX-AG/pulpcore/blob/fix_large_exports/pulpcore/app/importexport.py#L118 See the diff of the change here. This seems to get us through the first version of the problem (we will most likely still fail on the second version, but we now need to wait about an hour for our test run to fail 😉) |
A note to recreate this behavior w/out needing 65K large rpm/deb artifacts:
(master) ~/github/Pulp3/pulp-fixtures $ git diff
diff --git a/Makefile b/Makefile
index 00fc396..5c6b89d 100644
--- a/Makefile
+++ b/Makefile
@@ -337,7 +337,7 @@ fixtures/file-perf: fixtures
# 100,000 files were exceeding Travis' 50 minute time limit
# https://pulp.plan.io/issues/6104
# file/gen-fixtures.sh $@ --number 100000 --file-size 50
- file/gen-fixtures.sh $@ --number 20000 --file-size 50
+ file/gen-fixtures.sh $@ --number 100000 --file-size 10
fixtures/file-dl-forward: fixtures
file/gen-fixtures.sh $@ --number 750 --file-size 50
Much kinder on one's available disk space (repo is ~400Mb instead of gigabytes...) |
closes pulp#5375 Co-authored-by: Tobias Grigo <56518487+hstct@users.noreply.github.com>
So we still don't know exactly what changed between Django 3 and Django 4 to break this? |
closes pulp#5375 Co-authored-by: Tobias Grigo <56518487+hstct@users.noreply.github.com>
closes pulp#5375 Co-authored-by: Tobias Grigo <56518487+hstct@users.noreply.github.com>
closes #5375 Co-authored-by: Tobias Grigo <56518487+hstct@users.noreply.github.com>
Version
Describe the bug
Exporting data (in this case Content Views in Katello) with large amount of packages will fail:
Error: sending query and params failed: number of parameters must be between 0 and 65535
There are actually at least two different issues both of which will throw the same error:
Debian and RPM content are both affected by this.
To Reproduce
Steps to reproduce the behavior:
hammer content-export complete version --chunk-size-gb=40 --content-view="CV_NAME" --organization-id=1 --version=1.0
Expected behavior
The export to be successful.
Additional context
We suspect that this has something to do with the upgrade to Django 4.2 or more specifically with the upgrade from psycopg2 to psycopg3. We found an issue that parallels our findings psycopg/psycopg#620
We can also confirm that on older systems that still use older pulpcore versions (pre django 4.2) the export still works on these types of Content Views.
For more context here are the pulp tasks:
The text was updated successfully, but these errors were encountered: