Add support for syncing from v3 #286
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
Looks good to me from just visual inspection. Just started testing it though. |
|
IGNORE: See #286 (comment) @daviddavis Running this pulp_ansible pr against galaxy-dev, I run into errors when syncing NOTE: I suspect this is primarily caused by an incorrect url specified for the remote Creating a sync task: For the remote 0fbd4044-4524-4895-980b-9f38f7738ab3 {
"ca_cert": "d770937c97e5e0db5a13da580c8b0724989533f4d8059e115c65683835f44ee5",
"client_cert": null,
"client_key": null,
"download_concurrency": 20,
"name": "foo_autohub_ci_pzzycdot",
"policy": "immediate",
"proxy_url": null,
"pulp_created": "2020-02-10T18:34:02.356662Z",
"pulp_href": "/pulp/api/v3/remotes/ansible/collection/0fbd4044-4524-4895-980b-9f38f7738ab3/",
"pulp_last_updated": "2020-02-10T18:34:02.356682Z",
"requirements_file": null,
"tls_validation": true,
"url": "https://ci.cloud.redhat.com/api/automation-hub/v3/collections/autohubtest2/collection_dep_a_pzzycdot"
}
The server side logs show tracebacks: And the task result for the sync task:
{
"created_resources": [],
"error": {
"description": "URL: /api/automation-hub/v3/collections/autohubtest2/collection_dep_a_pzzycdot/versions/ not supported.",
"traceback": " File \"/venv/lib64/python3.6/site-packages/rq/worker.py\", line 884, in perform_job\n rv = job.perform()\n File \"/venv/lib64/python3.6/site-packages/rq/job.py\", line 664, in perform\n self._result = self._execute()\n File \"/venv/lib64/python3.6/site-packages/rq/job.py\", line 670, in _execute\n return self.func(*self.args, **self.kwargs)\n File \"/code/pulp_ansible/pulp_ansible/app/tasks/collections.py\", line 80, in sync\n d_version = AnsibleDeclarativeVersion(first_stage, repository, mirror=mirror)\n File \"/venv/lib64/python3.6/site-packages/pulpcore/plugin/stages/declarative_version.py\", line 149, in create\n loop.run_until_complete(pipeline)\n File \"/usr/lib64/python3.6/asyncio/base_events.py\", line 484, in run_until_complete\n return future.result()\n File \"/venv/lib64/python3.6/site-packages/pulpcore/plugin/stages/api.py\", line 228, in create_pipeline\n await asyncio.gather(*futures)\n File \"/venv/lib64/python3.6/site-packages/pulpcore/plugin/stages/api.py\", line 43, in __call__\n await self.run()\n File \"/code/pulp_ansible/pulp_ansible/app/tasks/collections.py\", line 274, in run\n with ProgressReport(message=\"Parsing Collection Metadata\", code=\"parsing.metadata\") as pb:\n File \"/code/pulp_ansible/pulp_ansible/app/tasks/collections.py\", line 351, in _fetch_collections\n File \"/venv/lib64/python3.6/site-packages/pulpcore/app/models/repository.py\", line 288, in get_downloader\n return self.download_factory.build(url, **kwargs)\n File \"/venv/lib64/python3.6/site-packages/pulpcore/download/factory.py\", line 140, in build\n raise ValueError(_('URL: {u} not supported.'.format(u=url)))\n"
},
"finished_at": "2020-02-25T15:10:13.217897Z",
"name": "pulp_ansible.app.tasks.collections.sync",
"progress_reports": [
{
"code": "parsing.metadata",
"done": 0,
"message": "Parsing Collection Metadata",
"state": "failed",
"suffix": null,
"total": null
},
{
"code": "parsing.collections",
"done": 0,
"message": "Parsing Galaxy Collections API",
"state": "failed",
"suffix": null,
"total": 1
},
{
"code": "downloading.artifacts",
"done": 0,
"message": "Downloading Artifacts",
"state": "canceled",
"suffix": null,
"total": null
},
{
"code": "associating.content",
"done": 0,
"message": "Associating Content",
"state": "canceled",
"suffix": null,
"total": null
}
],
"pulp_created": "2020-02-25T15:10:10.289509Z",
"pulp_href": "/pulp/api/v3/tasks/f969af22-3560-4ed1-8886-90cbb9263911/",
"reserved_resources_record": [
"/pulp/api/v3/remotes/ansible/collection/0fbd4044-4524-4895-980b-9f38f7738ab3/",
"/pulp/api/v3/repositories/ansible/ansible/5f4caaa4-3c65-41fd-8690-208a9a24d65c/"
],
"started_at": "2020-02-25T15:10:10.407881Z",
"state": "failed",
"worker": "/pulp/api/v3/workers/1ee9ab44-54fb-41f3-a441-fe33c9377a41/"
}
```~~~_ |
|
(The traceback from #286 (comment) but formatted for easier reading) Traceback (most recent call last):
File "/venv/lib64/python3.6/site-packages/pulpcore/download/factory.py", line 137, in build
builder = self._handler_map[scheme]
KeyError: ''
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/venv/lib64/python3.6/site-packages/rq/worker.py", line 884, in perform_job
rv = job.perform()
File "/venv/lib64/python3.6/site-packages/rq/job.py", line 664, in perform
self._result = self._execute()
File "/venv/lib64/python3.6/site-packages/rq/job.py", line 670, in _execute
return self.func(*self.args, **self.kwargs)
File "/code/pulp_ansible/pulp_ansible/app/tasks/collections.py", line 80, in sync
d_version = AnsibleDeclarativeVersion(first_stage, repository, mirror=mirror)
File "/venv/lib64/python3.6/site-packages/pulpcore/plugin/stages/declarative_version.py", line 149, in create
loop.run_until_complete(pipeline)
File "/usr/lib64/python3.6/asyncio/base_events.py", line 484, in run_until_complete
return future.result()
File "/venv/lib64/python3.6/site-packages/pulpcore/plugin/stages/api.py", line 228, in create_pipeline
await asyncio.gather(*futures)
File "/venv/lib64/python3.6/site-packages/pulpcore/plugin/stages/api.py", line 43, in __call__
await self.run()
File "/code/pulp_ansible/pulp_ansible/app/tasks/collections.py", line 274, in run
with ProgressReport(message="Parsing Collection Metadata", code="parsing.metadata") as pb:
File "/code/pulp_ansible/pulp_ansible/app/tasks/collections.py", line 351, in _fetch_collections
File "/venv/lib64/python3.6/site-packages/pulpcore/app/models/repository.py", line 288, in get_downloader
return self.download_factory.build(url, **kwargs)
File "/venv/lib64/python3.6/site-packages/pulpcore/download/factory.py", line 140, in build
raise ValueError(_('URL: {u} not supported.'.format(u=url)))
ValueError: URL: /api/automation-hub/v3/collections/autohubtest2/collection_dep_a_pzzycdot/versions/ not supported.
Traceback (most recent call last):
File "/venv/lib64/python3.6/site-packages/pulpcore/download/factory.py", line 137, in build
builder = self._handler_map[scheme]
KeyError: ''
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/venv/lib64/python3.6/site-packages/rq/worker.py", line 884, in perform_job
rv = job.perform()
File "/venv/lib64/python3.6/site-packages/rq/job.py", line 664, in perform
self._result = self._execute()
File "/venv/lib64/python3.6/site-packages/rq/job.py", line 670, in _execute
return self.func(*self.args, **self.kwargs)
File "/code/pulp_ansible/pulp_ansible/app/tasks/collections.py", line 80, in sync
d_version = AnsibleDeclarativeVersion(first_stage, repository, mirror=mirror)
File "/venv/lib64/python3.6/site-packages/pulpcore/plugin/stages/declarative_version.py", line 149, in create
loop.run_until_complete(pipeline)
File "/usr/lib64/python3.6/asyncio/base_events.py", line 484, in run_until_complete
return future.result()
File "/venv/lib64/python3.6/site-packages/pulpcore/plugin/stages/api.py", line 228, in create_pipeline
await asyncio.gather(*futures)
File "/venv/lib64/python3.6/site-packages/pulpcore/plugin/stages/api.py", line 43, in __call__
await self.run()
File "/code/pulp_ansible/pulp_ansible/app/tasks/collections.py", line 274, in run
with ProgressReport(message="Parsing Collection Metadata", code="parsing.metadata") as pb:
File "/code/pulp_ansible/pulp_ansible/app/tasks/collections.py", line 351, in _fetch_collections
File "/venv/lib64/python3.6/site-packages/pulpcore/app/models/repository.py", line 288, in get_downloader
return self.download_factory.build(url, **kwargs)
File "/venv/lib64/python3.6/site-packages/pulpcore/download/factory.py", line 140, in build
raise ValueError(_('URL: {u} not supported.'.format(u=url)))
ValueError: URL: /api/automation-hub/v3/collections/autohubtest2/collection_dep_a_pzzycdot/versions/ not supported. |
|
@daviddavis The errors above may boil down to the versions_url not being a full url. sync task result url http://localhost:5002/pulp/api/v3/tasks/b9c3b029-9958-4b40-89bf-a83b3dfdd734/ |
|
Which I guess is mostly stuff from master...alikins:collection_sync_6116 I forgot to pr. |
|
@daviddavis Looks like #286 (comment) through #286 (comment) were causes by django reloading, so a restart of django/pulp resolved that. |
daviddavis
force-pushed
the
issue6132
branch
2 times, most recently
from
7573030
to
d07c595
Compare
|
Hello @daviddavis! Thanks for updating this PR. We checked the lines you've touched for PEP 8 issues, and found: There are currently no PEP 8 issues detected in this Pull Request. Cheers! 🍻 Comment last updated at 2020-03-10 20:58:22 UTC |
daviddavis
force-pushed
the
issue6132
branch
5 times, most recently
from
70a8850
to
f7698aa
Compare
|
@bmbouter @alikins this should be good to go now. Note that I tried to also expose our v3 api to the ansible-galaxy cli but there are some discrepancies between our format of the v3 api vs Automation Hub's. I filed a follow up issue to get that fixed. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
fixes #6132
https://pulp.plan.io/issues/6132