-
Notifications
You must be signed in to change notification settings - Fork 53
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add support for syncing from v3 #286
Conversation
Looks good to me from just visual inspection. Just started testing it though. |
IGNORE: See #286 (comment) @daviddavis Running this pulp_ansible pr against galaxy-dev, I run into errors when syncing NOTE: I suspect this is primarily caused by an incorrect url specified for the remote Creating a sync task:
For the remote 0fbd4044-4524-4895-980b-9f38f7738ab3 {
"ca_cert": "d770937c97e5e0db5a13da580c8b0724989533f4d8059e115c65683835f44ee5",
"client_cert": null,
"client_key": null,
"download_concurrency": 20,
"name": "foo_autohub_ci_pzzycdot",
"policy": "immediate",
"proxy_url": null,
"pulp_created": "2020-02-10T18:34:02.356662Z",
"pulp_href": "/pulp/api/v3/remotes/ansible/collection/0fbd4044-4524-4895-980b-9f38f7738ab3/",
"pulp_last_updated": "2020-02-10T18:34:02.356682Z",
"requirements_file": null,
"tls_validation": true,
"url": "https://ci.cloud.redhat.com/api/automation-hub/v3/collections/autohubtest2/collection_dep_a_pzzycdot"
}
The server side logs show tracebacks:
And the task result for the sync task:
{
"created_resources": [],
"error": {
"description": "URL: /api/automation-hub/v3/collections/autohubtest2/collection_dep_a_pzzycdot/versions/ not supported.",
"traceback": " File \"/venv/lib64/python3.6/site-packages/rq/worker.py\", line 884, in perform_job\n rv = job.perform()\n File \"/venv/lib64/python3.6/site-packages/rq/job.py\", line 664, in perform\n self._result = self._execute()\n File \"/venv/lib64/python3.6/site-packages/rq/job.py\", line 670, in _execute\n return self.func(*self.args, **self.kwargs)\n File \"/code/pulp_ansible/pulp_ansible/app/tasks/collections.py\", line 80, in sync\n d_version = AnsibleDeclarativeVersion(first_stage, repository, mirror=mirror)\n File \"/venv/lib64/python3.6/site-packages/pulpcore/plugin/stages/declarative_version.py\", line 149, in create\n loop.run_until_complete(pipeline)\n File \"/usr/lib64/python3.6/asyncio/base_events.py\", line 484, in run_until_complete\n return future.result()\n File \"/venv/lib64/python3.6/site-packages/pulpcore/plugin/stages/api.py\", line 228, in create_pipeline\n await asyncio.gather(*futures)\n File \"/venv/lib64/python3.6/site-packages/pulpcore/plugin/stages/api.py\", line 43, in __call__\n await self.run()\n File \"/code/pulp_ansible/pulp_ansible/app/tasks/collections.py\", line 274, in run\n with ProgressReport(message=\"Parsing Collection Metadata\", code=\"parsing.metadata\") as pb:\n File \"/code/pulp_ansible/pulp_ansible/app/tasks/collections.py\", line 351, in _fetch_collections\n File \"/venv/lib64/python3.6/site-packages/pulpcore/app/models/repository.py\", line 288, in get_downloader\n return self.download_factory.build(url, **kwargs)\n File \"/venv/lib64/python3.6/site-packages/pulpcore/download/factory.py\", line 140, in build\n raise ValueError(_('URL: {u} not supported.'.format(u=url)))\n"
},
"finished_at": "2020-02-25T15:10:13.217897Z",
"name": "pulp_ansible.app.tasks.collections.sync",
"progress_reports": [
{
"code": "parsing.metadata",
"done": 0,
"message": "Parsing Collection Metadata",
"state": "failed",
"suffix": null,
"total": null
},
{
"code": "parsing.collections",
"done": 0,
"message": "Parsing Galaxy Collections API",
"state": "failed",
"suffix": null,
"total": 1
},
{
"code": "downloading.artifacts",
"done": 0,
"message": "Downloading Artifacts",
"state": "canceled",
"suffix": null,
"total": null
},
{
"code": "associating.content",
"done": 0,
"message": "Associating Content",
"state": "canceled",
"suffix": null,
"total": null
}
],
"pulp_created": "2020-02-25T15:10:10.289509Z",
"pulp_href": "/pulp/api/v3/tasks/f969af22-3560-4ed1-8886-90cbb9263911/",
"reserved_resources_record": [
"/pulp/api/v3/remotes/ansible/collection/0fbd4044-4524-4895-980b-9f38f7738ab3/",
"/pulp/api/v3/repositories/ansible/ansible/5f4caaa4-3c65-41fd-8690-208a9a24d65c/"
],
"started_at": "2020-02-25T15:10:10.407881Z",
"state": "failed",
"worker": "/pulp/api/v3/workers/1ee9ab44-54fb-41f3-a441-fe33c9377a41/"
}
```~~~_ |
(The traceback from #286 (comment) but formatted for easier reading) Traceback (most recent call last):
File "/venv/lib64/python3.6/site-packages/pulpcore/download/factory.py", line 137, in build
builder = self._handler_map[scheme]
KeyError: ''
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/venv/lib64/python3.6/site-packages/rq/worker.py", line 884, in perform_job
rv = job.perform()
File "/venv/lib64/python3.6/site-packages/rq/job.py", line 664, in perform
self._result = self._execute()
File "/venv/lib64/python3.6/site-packages/rq/job.py", line 670, in _execute
return self.func(*self.args, **self.kwargs)
File "/code/pulp_ansible/pulp_ansible/app/tasks/collections.py", line 80, in sync
d_version = AnsibleDeclarativeVersion(first_stage, repository, mirror=mirror)
File "/venv/lib64/python3.6/site-packages/pulpcore/plugin/stages/declarative_version.py", line 149, in create
loop.run_until_complete(pipeline)
File "/usr/lib64/python3.6/asyncio/base_events.py", line 484, in run_until_complete
return future.result()
File "/venv/lib64/python3.6/site-packages/pulpcore/plugin/stages/api.py", line 228, in create_pipeline
await asyncio.gather(*futures)
File "/venv/lib64/python3.6/site-packages/pulpcore/plugin/stages/api.py", line 43, in __call__
await self.run()
File "/code/pulp_ansible/pulp_ansible/app/tasks/collections.py", line 274, in run
with ProgressReport(message="Parsing Collection Metadata", code="parsing.metadata") as pb:
File "/code/pulp_ansible/pulp_ansible/app/tasks/collections.py", line 351, in _fetch_collections
File "/venv/lib64/python3.6/site-packages/pulpcore/app/models/repository.py", line 288, in get_downloader
return self.download_factory.build(url, **kwargs)
File "/venv/lib64/python3.6/site-packages/pulpcore/download/factory.py", line 140, in build
raise ValueError(_('URL: {u} not supported.'.format(u=url)))
ValueError: URL: /api/automation-hub/v3/collections/autohubtest2/collection_dep_a_pzzycdot/versions/ not supported.
Traceback (most recent call last):
File "/venv/lib64/python3.6/site-packages/pulpcore/download/factory.py", line 137, in build
builder = self._handler_map[scheme]
KeyError: ''
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/venv/lib64/python3.6/site-packages/rq/worker.py", line 884, in perform_job
rv = job.perform()
File "/venv/lib64/python3.6/site-packages/rq/job.py", line 664, in perform
self._result = self._execute()
File "/venv/lib64/python3.6/site-packages/rq/job.py", line 670, in _execute
return self.func(*self.args, **self.kwargs)
File "/code/pulp_ansible/pulp_ansible/app/tasks/collections.py", line 80, in sync
d_version = AnsibleDeclarativeVersion(first_stage, repository, mirror=mirror)
File "/venv/lib64/python3.6/site-packages/pulpcore/plugin/stages/declarative_version.py", line 149, in create
loop.run_until_complete(pipeline)
File "/usr/lib64/python3.6/asyncio/base_events.py", line 484, in run_until_complete
return future.result()
File "/venv/lib64/python3.6/site-packages/pulpcore/plugin/stages/api.py", line 228, in create_pipeline
await asyncio.gather(*futures)
File "/venv/lib64/python3.6/site-packages/pulpcore/plugin/stages/api.py", line 43, in __call__
await self.run()
File "/code/pulp_ansible/pulp_ansible/app/tasks/collections.py", line 274, in run
with ProgressReport(message="Parsing Collection Metadata", code="parsing.metadata") as pb:
File "/code/pulp_ansible/pulp_ansible/app/tasks/collections.py", line 351, in _fetch_collections
File "/venv/lib64/python3.6/site-packages/pulpcore/app/models/repository.py", line 288, in get_downloader
return self.download_factory.build(url, **kwargs)
File "/venv/lib64/python3.6/site-packages/pulpcore/download/factory.py", line 140, in build
raise ValueError(_('URL: {u} not supported.'.format(u=url)))
ValueError: URL: /api/automation-hub/v3/collections/autohubtest2/collection_dep_a_pzzycdot/versions/ not supported. |
@daviddavis The errors above may boil down to the versions_url not being a full url. sync task result url http://localhost:5002/pulp/api/v3/tasks/b9c3b029-9958-4b40-89bf-a83b3dfdd734/
|
Which I guess is mostly stuff from master...alikins:collection_sync_6116 I forgot to pr. |
@daviddavis Looks like #286 (comment) through #286 (comment) were causes by django reloading, so a restart of django/pulp resolved that. |
7573030
to
d07c595
Compare
Hello @daviddavis! Thanks for updating this PR. We checked the lines you've touched for PEP 8 issues, and found: There are currently no PEP 8 issues detected in this Pull Request. Cheers! 🍻 Comment last updated at 2020-03-10 20:58:22 UTC |
70a8850
to
f7698aa
Compare
@bmbouter @alikins this should be good to go now. Note that I tried to also expose our v3 api to the ansible-galaxy cli but there are some discrepancies between our format of the v3 api vs Automation Hub's. I filed a follow up issue to get that fixed. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks @daviddavis !
fixes #6132
https://pulp.plan.io/issues/6132