Skip to content
This repository has been archived by the owner on Jan 9, 2023. It is now read-only.

Add pulp_deb plugin support (Packages only for now) #149

Merged
merged 1 commit into from Nov 19, 2020
Merged

Add pulp_deb plugin support (Packages only for now) #149

merged 1 commit into from Nov 19, 2020

Conversation

quba42
Copy link
Contributor

@quba42 quba42 commented May 14, 2020 •
edited

@quba42 quba42 force-pushed the pulp_deb branch 3 times, most recently from d643caf to 1c4a49c Compare July 2, 2020 14:55
@quba42
Copy link
Contributor Author

quba42 commented Jul 2, 2020

The latest error I am stuck on is:

pulp: pulp_2to3_migration.pulp2.connection:INFO: Attempting to connect to localhost:27017
pulp: rq.worker:ERROR: ValueError: No declared artifact with relative path "tuxedo-cc-wmi_0.1.4_all.deb" for content "<Package: __>"
Traceback (most recent call last):
File "/usr/local/lib/pulp/lib64/python3.6/site-packages/rq/worker.py", line 883, in perform_job
File "/usr/local/lib/pulp/lib64/python3.6/site-packages/rq/job.py", line 657, in perform
File "/usr/local/lib/pulp/lib64/python3.6/site-packages/rq/job.py", line 663, in _execute
File "/home/vagrant/devel/pulp_2to3_migration/pulp_2to3_migration/app/tasks/migrate.py", line 140, in migrate_from_pulp2
File "/home/vagrant/devel/pulp_2to3_migration/pulp_2to3_migration/app/migration.py", line 36, in migrate_content
File "/home/vagrant/devel/pulp_2to3_migration/pulp_2to3_migration/app/plugin/deb/migrator.py", line 49, in migrate_content_to_pulp3
File "/usr/lib64/python3.6/asyncio/base_events.py", line 484, in run_until_complete
File "/home/vagrant/devel/pulp_2to3_migration/pulp_2to3_migration/app/plugin/content.py", line 84, in create
File "/home/vagrant/devel/pulpcore/pulpcore/plugin/stages/api.py", line 225, in create_pipeline
File "/home/vagrant/devel/pulpcore/pulpcore/plugin/stages/api.py", line 43, in __call__
File "/home/vagrant/devel/pulpcore/pulpcore/plugin/stages/artifact_stages.py", line 225, in run
File "/home/vagrant/devel/pulpcore/pulpcore/plugin/stages/artifact_stages.py", line 269, in _needed_remote_artifacts
ValueError: No declared artifact with relative path "tuxedo-cc-wmi_0.1.4_all.deb" for content "<Package: __>"
Traceback (most recent call last):
File "/usr/local/lib/pulp/lib64/python3.6/site-packages/rq/worker.py", line 883, in perform_job
File "/usr/local/lib/pulp/lib64/python3.6/site-packages/rq/job.py", line 657, in perform
File "/usr/local/lib/pulp/lib64/python3.6/site-packages/rq/job.py", line 663, in _execute
File "/home/vagrant/devel/pulp_2to3_migration/pulp_2to3_migration/app/tasks/migrate.py", line 140, in migrate_from_pulp2
File "/home/vagrant/devel/pulp_2to3_migration/pulp_2to3_migration/app/migration.py", line 36, in migrate_content
File "/home/vagrant/devel/pulp_2to3_migration/pulp_2to3_migration/app/plugin/deb/migrator.py", line 49, in migrate_content_to_pulp3
File "/usr/lib64/python3.6/asyncio/base_events.py", line 484, in run_until_complete
File "/home/vagrant/devel/pulp_2to3_migration/pulp_2to3_migration/app/plugin/content.py", line 84, in create
File "/home/vagrant/devel/pulpcore/pulpcore/plugin/stages/api.py", line 225, in create_pipeline
File "/home/vagrant/devel/pulpcore/pulpcore/plugin/stages/api.py", line 43, in __call__
File "/home/vagrant/devel/pulpcore/pulpcore/plugin/stages/artifact_stages.py", line 225, in run
File "/home/vagrant/devel/pulpcore/pulpcore/plugin/stages/artifact_stages.py", line 269, in _needed_remote_artifacts
ValueError: No declared artifact with relative path "tuxedo-cc-wmi_0.1.4_all.deb" for content "<Package: __>"

At this point I am thinking that my problem is that I just don't understand the central async pipelines that are being used...

@quba42 quba42 force-pushed the pulp_deb branch 3 times, most recently from 479f77d to 8d78f9a Compare November 16, 2020 16:38
@quba42 quba42 marked this pull request as ready for review November 16, 2020 16:39
@quba42
Copy link
Contributor Author

quba42 commented Nov 16, 2020 •
edited

I believe this is now ready for an initial review.
It has some limited but independently useful functionality.
The current state is able to migrate Debian packages from Pulp2 to Pulp3.
The so created repository is published using the simple publisher to create a usable Pulp3 publication. (I have not yet been able to look over the resulting publication due to some network configuration issue with my development environment...)

What is currently missing is all the content types required for structured publishing.
It might make sense to review and merge the current state, and then extend its functionality on separate issue branches (to avoid a single giant PR seeking to add everything at once).
However, in that case it needs to be made clear to any potential users that the current state is not yet full featured or ready for production.
How was this handled for other plugins?

@quba42
Copy link
Contributor Author

quba42 commented Nov 16, 2020

I would of course be happy to arrange a meeting/screenshare or similar on how I am currently testing this in my development environment.

@quba42 quba42 force-pushed the pulp_deb branch 3 times, most recently from 923c39a to ae43e2c Compare November 18, 2020 13:15
@quba42
Copy link
Contributor Author

quba42 commented Nov 18, 2020

Rebased to take advantage of Travis pipeline with pulp_deb installed in it.

goosemania
goosemania suggested changes Nov 18, 2020
View reviewed changes
Copy link
Member

@goosemania goosemania left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@quba42, I have just some minor comments, other than that it's ready to merge I believe. Thanks for working on it!

@goosemania
Copy link
Member

@quba42, and about the plugin support, I suggest to add a line to a README and to the docs.
Either that pulp_deb is in active development and currently supporting packages only, or mention that it's a tech preview with packages suppor tonly.

@quba42
Copy link
Contributor Author

quba42 commented Nov 19, 2020

I squished all commits into one and added an issue reference in the commit message.
I also added a line on plugin support to both the README and the main documentation.
In addition I added some more detailed description of what does and does not work to the workflow documentation.

@quba42
Copy link
Contributor Author

quba42 commented Nov 19, 2020

Not sure why the tests are now failing, looks a bit like a timeout issue to me.

goosemania
goosemania reviewed Nov 19, 2020
View reviewed changes
CHANGES/7863.feature
@@ -0,0 +1 @@
Added support to migrate Debian packages (tech preview).
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍

goosemania
goosemania approved these changes Nov 19, 2020
View reviewed changes
Copy link
Member

@goosemania goosemania left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great job!

@goosemania
Copy link
Member

Not sure why the tests are now failing, looks a bit like a timeout issue to me.

I restarted the job, we'll see.

@goosemania goosemania merged commit ba6b6f1 into pulp:master Nov 19, 2020
@quba42 quba42 deleted the pulp_deb branch November 19, 2020 11:47
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Reviewers

@quba quba quba left review comments

@dralley dralley dralley left review comments

@goosemania goosemania goosemania approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
4 participants
@quba42 @goosemania @quba @dralley