Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Can't rerun a failed content-import task if it was exported using chunks #3737

Closed
dralley opened this issue Apr 12, 2023 · 2 comments · Fixed by #3886 or pulp/pulp_file#969
Closed

Can't rerun a failed content-import task if it was exported using chunks #3737

dralley opened this issue Apr 12, 2023 · 2 comments · Fixed by #3886 or pulp/pulp_file#969
Assignees
@ggainey
Labels
Issue prio-list satellite

Comments

@dralley
Copy link
Contributor

dralley commented Apr 12, 2023

Version

All?

Describe the bug

When importing content that was exported in chunks, the importer process concatenate the chunks into a single file in order to import.

If that import task fails for some reason, after the chunks were already combined into a single file, the user can't simply re-run the same command to retry the import. Satellite will complain that the chunks are missing.

Checking the directory, we can see that the chunks are gone and only the master file is present (together with metadata.json and TOC file).

To Reproduce

Steps to Reproduce:

  1. Export something using chunks
  2. Start the import on another satellite. Monitor the data directory until all the chunk files are gone and only the main file is present. At this moment, kill the pulpcore-worker which is processing the import. Wait until the task returns with error.
  3. Repeat the same import command. Pulp will error out with an error like this:
Feb 27 16:38:56 reproducer-import pulpcore-worker-3[156191]: pulp [6c91f855-9959-43b1-864f-925393ae025a]: pulpcore.tasking.pulpcore_worker:INFO: Task ddf45eb1-adea-480e-9114-748baa7bd7bf failed ([ErrorDetail(string="Missing import-chunks named in table-of-contents: ['export-fc4273a4-6320-4fd5-98c9-6adfe9461781-20230227_2132.tar.gz.0000', 'export-fc4273a4-6320-4fd5-98c9-6adfe9461781-20230227_2132.tar.gz.0001', 'export-fc4273a4-6320-4fd5-98c9-6adfe9461781-20230227_2132.tar.gz.0002'].", code='invalid')])

Actual results:
Re-running same import fails, complaining about chunks missing. User needs to modify the TOC file manually OR split the master file in chunks again OR copy the files again in order to run the import.

Expected behavior

Pulp could be smart enough to identify either all the chunks are present OR the global file file. If the checksum matches, move forward with the import.

Additional context
https://bugzilla.redhat.com/show_bug.cgi?id=2173757
ggainey added a commit to ggainey/pulpcore that referenced this issue May 26, 2023
@ggainey
Taught PulpImport to recover from previously-failed attempt.
cab1216 
If an expected-reassembled-filename exists, ignore chunk-info and
proceed with that file.

Test-coverage will be added in pulp_file once this is in a released
version of pulpcore.

fixes pulp#3737.
[nocoverage]
@ggainey ggainey mentioned this issue May 26, 2023
Merged
@ggainey ggainey self-assigned this May 26, 2023
ggainey added a commit to ggainey/pulpcore that referenced this issue May 30, 2023
@ggainey
Taught PulpImport to recover from previously-failed attempt.
7ed17d4 
If an expected-reassembled-filename exists, ignore chunk-info and
proceed with that file.

Test-coverage will be added in pulp_file once this is in a released
version of pulpcore.

fixes pulp#3737.
[nocoverage]
ggainey added a commit to ggainey/pulpcore that referenced this issue May 30, 2023
@ggainey
Taught PulpImport to recover from previously-failed attempt.
a3a3e5a 
If an expected-reassembled-filename exists, ignore chunk-info and
proceed with that file.

Test-coverage will be added in pulp_file once this is in a released
version of pulpcore.

fixes pulp#3737.
[nocoverage]
mdellweg pushed a commit that referenced this issue Jun 5, 2023
@ggainey @mdellweg
Taught PulpImport to recover from previously-failed attempt.
596a72e 
If an expected-reassembled-filename exists, ignore chunk-info and
proceed with that file.

Test-coverage will be added in pulp_file once this is in a released
version of pulpcore.

fixes #3737.
[nocoverage]
patchback bot pushed a commit that referenced this issue Jun 5, 2023
@ggainey @patchback
Taught PulpImport to recover from previously-failed attempt.
80e6c3e 
If an expected-reassembled-filename exists, ignore chunk-info and
proceed with that file.

Test-coverage will be added in pulp_file once this is in a released
version of pulpcore.

fixes #3737.
[nocoverage]

(cherry picked from commit 596a72e)
@patchback patchback bot mentioned this issue Jun 5, 2023
Merged
patchback bot pushed a commit that referenced this issue Jun 5, 2023
@ggainey @patchback
Taught PulpImport to recover from previously-failed attempt.
4619688 
If an expected-reassembled-filename exists, ignore chunk-info and
proceed with that file.

Test-coverage will be added in pulp_file once this is in a released
version of pulpcore.

fixes #3737.
[nocoverage]

(cherry picked from commit 596a72e)
@patchback patchback bot mentioned this issue Jun 5, 2023
Merged
patchback bot pushed a commit that referenced this issue Jun 5, 2023
@ggainey @patchback
Taught PulpImport to recover from previously-failed attempt.
17ce7ea 
If an expected-reassembled-filename exists, ignore chunk-info and
proceed with that file.

Test-coverage will be added in pulp_file once this is in a released
version of pulpcore.

fixes #3737.
[nocoverage]

(cherry picked from commit 596a72e)
@patchback patchback bot mentioned this issue Jun 5, 2023
Merged
patchback bot pushed a commit that referenced this issue Jun 5, 2023
@ggainey @patchback
Taught PulpImport to recover from previously-failed attempt.
7adc3ca 
If an expected-reassembled-filename exists, ignore chunk-info and
proceed with that file.

Test-coverage will be added in pulp_file once this is in a released
version of pulpcore.

fixes #3737.
[nocoverage]

(cherry picked from commit 596a72e)
@patchback patchback bot mentioned this issue Jun 5, 2023
Merged
patchback bot pushed a commit that referenced this issue Jun 5, 2023
@ggainey @patchback
Taught PulpImport to recover from previously-failed attempt.
e38565d 
If an expected-reassembled-filename exists, ignore chunk-info and
proceed with that file.

Test-coverage will be added in pulp_file once this is in a released
version of pulpcore.

fixes #3737.
[nocoverage]

(cherry picked from commit 596a72e)
@patchback patchback bot mentioned this issue Jun 5, 2023
Merged
mdellweg pushed a commit that referenced this issue Jun 5, 2023
@ggainey @mdellweg
Taught PulpImport to recover from previously-failed attempt.
8a4d98e 
If an expected-reassembled-filename exists, ignore chunk-info and
proceed with that file.

Test-coverage will be added in pulp_file once this is in a released
version of pulpcore.

fixes #3737.
[nocoverage]

(cherry picked from commit 596a72e)
mdellweg pushed a commit that referenced this issue Jun 5, 2023
@ggainey @mdellweg
Taught PulpImport to recover from previously-failed attempt.
30d4b67 
If an expected-reassembled-filename exists, ignore chunk-info and
proceed with that file.

Test-coverage will be added in pulp_file once this is in a released
version of pulpcore.

fixes #3737.
[nocoverage]

(cherry picked from commit 596a72e)
mdellweg pushed a commit that referenced this issue Jun 5, 2023
@ggainey @mdellweg
Taught PulpImport to recover from previously-failed attempt.
e2d648e 
If an expected-reassembled-filename exists, ignore chunk-info and
proceed with that file.

Test-coverage will be added in pulp_file once this is in a released
version of pulpcore.

fixes #3737.
[nocoverage]

(cherry picked from commit 596a72e)
mdellweg pushed a commit that referenced this issue Jun 5, 2023
@ggainey @mdellweg
Taught PulpImport to recover from previously-failed attempt.
04851b4 
If an expected-reassembled-filename exists, ignore chunk-info and
proceed with that file.

Test-coverage will be added in pulp_file once this is in a released
version of pulpcore.

fixes #3737.
[nocoverage]

(cherry picked from commit 596a72e)
mdellweg pushed a commit that referenced this issue Jun 5, 2023
@ggainey @mdellweg
Taught PulpImport to recover from previously-failed attempt.
f6a8657 
If an expected-reassembled-filename exists, ignore chunk-info and
proceed with that file.

Test-coverage will be added in pulp_file once this is in a released
version of pulpcore.

fixes #3737.
[nocoverage]

(cherry picked from commit 596a72e)
ggainey added a commit to ggainey/pulp_file that referenced this issue Jun 13, 2023
@ggainey
Adds test for re-import after broken import-attempt.
155cbc2 
Tests the support added to fix pulp/pulpcore#3737

[noissue]
@ggainey ggainey mentioned this issue Jun 13, 2023
Merged
@pulpbot
Copy link
Member

pulpbot commented Jun 29, 2023

https://bugzilla.redhat.com/show_bug.cgi?id=2218659

patchback bot pushed a commit that referenced this issue Jul 29, 2023
@ggainey @patchback
Taught PulpImport to recover from previously-failed attempt.
d65ca3c 
If an expected-reassembled-filename exists, ignore chunk-info and
proceed with that file.

Test-coverage will be added in pulp_file once this is in a released
version of pulpcore.

fixes #3737.
[nocoverage]

(cherry picked from commit 596a72e)
@patchback patchback bot mentioned this issue Jul 29, 2023
Merged
dralley pushed a commit that referenced this issue Jul 29, 2023
@ggainey @dralley
Taught PulpImport to recover from previously-failed attempt.
0c45a67 
If an expected-reassembled-filename exists, ignore chunk-info and
proceed with that file.

Test-coverage will be added in pulp_file once this is in a released
version of pulpcore.

fixes #3737.
[nocoverage]

(cherry picked from commit 596a72e)
@pulpbot
Copy link
Member

pulpbot commented Jul 31, 2023

https://bugzilla.redhat.com/show_bug.cgi?id=2227903

ggainey added a commit to pulp/pulp_file that referenced this issue Aug 3, 2023
@ggainey
Adds test for re-import after broken import-attempt.
7d1e048 
Tests the support added to fix pulp/pulpcore#3737

[noissue]
@patchback patchback bot mentioned this issue Sep 13, 2023
Merged
patchback bot pushed a commit that referenced this issue Sep 13, 2023
@ggainey @patchback
Taught PulpImport to recover from previously-failed attempt.
dd36ccd 
If an expected-reassembled-filename exists, ignore chunk-info and
proceed with that file.

Test-coverage will be added in pulp_file once this is in a released
version of pulpcore.

fixes #3737.
[nocoverage]

(cherry picked from commit 596a72e)
dralley pushed a commit that referenced this issue Sep 13, 2023
@ggainey @dralley
Taught PulpImport to recover from previously-failed attempt.
ee87d6c 
If an expected-reassembled-filename exists, ignore chunk-info and
proceed with that file.

Test-coverage will be added in pulp_file once this is in a released
version of pulpcore.

fixes #3737.
[nocoverage]

(cherry picked from commit 596a72e)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@ggainey ggainey

Labels
Issue prio-list satellite
Projects
No open projects
RH Pulp Kanban board
Archived in project
Milestone
No milestone
3 participants
@dralley @ggainey @pulpbot