Optimize collections sync #172
Conversation
| expected = ANSIBLE_COLLECTION_FIXTURE_SUMMARY | ||
| expected["ansible.collection_version"] = 2 | ||
| self.assertDictEqual(get_added_content_summary(repo), expected) | ||
| self.assertDictEqual(get_added_content_summary(repo), ANSIBLE_COLLECTION_FIXTURE_SUMMARY) |
There was a problem hiding this comment.
What about we update the usage of assertDictEqual? We could use another assertion a bit more strict. Removing the need of mock.ANY, for instance assertGreaterEqual(a, b). Since the end goal is to assure that content was synced properly, but we do not control that URL being synced.
|
@fabricio-aguiar I found the issue associated w/ this can you associate w/ the commit and add a bugfix release note? https://pulp.plan.io/issues/5236 |
| # Concurrent downloads are limited by aiohttp... | ||
| not_done = set( | ||
| remote.get_downloader(url=_get_url(page)).run() for page in range(2, page_count + 1) | ||
| remote.get_downloader(url=_get_url(page)).run() for page in range(1, page_count + 1) |
There was a problem hiding this comment.
This line could be broken up for readability.
There was a problem hiding this comment.
Black requires to put on the same line:
--- pulp_ansible/app/tasks/collections.py 2019-08-16 15:21:52.437710 +0000
+++ pulp_ansible/app/tasks/collections.py 2019-08-16 15:22:02.078186 +0000
@@ -258,12 +258,11 @@
progress_bar.total = count
progress_bar.save()
# Concurrent downloads are limited by aiohttp...
not_done = set(
- remote.get_downloader(url=_get_url(page)).run()
- for page in range(1, page_count + 1)
+ remote.get_downloader(url=_get_url(page)).run() for page in range(1, page_count + 1)
)
while not_done:
done, not_done = await asyncio.wait(not_done, return_when=asyncio.FIRST_COMPLETED)
for item in done:
would reformat pulp_ansible/app/tasks/collections.py
All done! 💥 💔 💥There was a problem hiding this comment.
I agree I was thinking of something like:
not_done = set()
for page in range(1, page_count + 1):
downloader = remote.get_downloader(url=_get_url(page))
not_done.add(downloader.run())
There was a problem hiding this comment.
this is great, thanks @bmbouter
closes #5236
closes #5236