🧪⇪ Bump `galaxy_ng` to v4.9.2 in tests by webknjaz · Pull Request #82396 · ansible/ansible

webknjaz · 2023-12-11T13:52:20Z

It updates the underlying pulp_ansible to v0.20.2, which fixes a race condition that is occasionally observable in our CI when provisioning the initial dummy collections for testing.

Refs:

https://github.com/ansible/galaxy_ng/releases/tag/4.9.0
update pulp_ansible to 0.20.2 galaxy_ng#1950
API Updates Broken: Duplicate key value violates unique constraint "unique_is_highest" galaxy#3267
Race condition on multiple nearly-simultaneous uploads of the same collection (different versions) pulp/pulp_ansible#1571
Fix constraint violation in is_latest pulp/pulp_ansible#1572
Add retry logic to is_higest calculation pulp/pulp_ansible#1595
is_highest update query can still fail since Django doesn't order the query for Postgres pulp/pulp_ansible#1623
Fix update_highest_version when versions were uploaded out-of-order pulp/pulp_ansible#1624

SUMMARY

$sbj.

ISSUE TYPE

Maintenance Pull Request
Test Pull Request

ADDITIONAL INFORMATION

N/A

webknjaz · 2023-12-11T15:29:05Z

The ansible-galaxy-collection integration test is broken in the CI runs:

03:20 TASK [ansible-galaxy-collection : ensure we can download the published collection - pulp_v2] ***
03:21 fatal: [testhost]: FAILED! => {"changed": true, "cmd": ["ansible-galaxy", "collection", "install", "-s", "pulp_v2", "-p", "/tmp/ansible.mk27mu05.test/publish/pulp_v2", "ansible_test.my_collection==1.0.0", "-v"], "delta": "0:00:00.470485", "end": "2023-12-11 13:56:52.273680", "msg": "non-zero return code", "rc": 1, "start": "2023-12-11 13:56:51.803195", "stderr": "\u001b[1;35m[WARNING]: The specified \"api_version\" configuration for the galaxy server\u001b[0m\n\u001b[1;35m\"pulp_v2\" is not a public configuration, and may be removed at any time without\u001b[0m\n\u001b[1;35mwarning.\u001b[0m\n\u001b[1;35m[WARNING]: The specified collections path\u001b[0m\n\u001b[1;35m'/tmp/ansible.mk27mu05.test/publish/pulp_v2' is not part of the configured\u001b[0m\n\u001b[1;35mAnsible collections paths\u001b[0m\n\u001b[1;35m'/root/.ansible/collections:/usr/share/ansible/collections'. The installed\u001b[0m\n\u001b[1;35mcollection will not be picked up in an Ansible run, unless within a playbook-\u001b[0m\n\u001b[1;35madjacent collections directory.\u001b[0m\n\u001b[0;31mERROR! Error when getting the collection info for ansible_test.my_collection from pulp_v2 (http://galaxy-pulp/pulp_ansible/galaxy/primary/api/) (HTTP Code: 500, Message: Internal Server Error Code: Unknown)\u001b[0m", "stderr_lines": ["\u001b[1;35m[WARNING]: The specified \"api_version\" configuration for the galaxy server\u001b[0m", "\u001b[1;35m\"pulp_v2\" is not a public configuration, and may be removed at any time without\u001b[0m", "\u001b[1;35mwarning.\u001b[0m", "\u001b[1;35m[WARNING]: The specified collections path\u001b[0m", "\u001b[1;35m'/tmp/ansible.mk27mu05.test/publish/pulp_v2' is not part of the configured\u001b[0m", "\u001b[1;35mAnsible collections paths\u001b[0m", "\u001b[1;35m'/root/.ansible/collections:/usr/share/ansible/collections'. The installed\u001b[0m", "\u001b[1;35mcollection will not be picked up in an Ansible run, unless within a playbook-\u001b[0m", "\u001b[1;35madjacent collections directory.\u001b[0m", "\u001b[0;31mERROR! Error when getting the collection info for ansible_test.my_collection from pulp_v2 (http://galaxy-pulp/pulp_ansible/galaxy/primary/api/) (HTTP Code: 500, Message: Internal Server Error Code: Unknown)\u001b[0m"], "stdout": "\u001b[0;34mUsing /tmp/ansible.mk27mu05.test/galaxy/ansible.cfg as config file\u001b[0m\nStarting galaxy collection install process\nProcess install dependency map", "stdout_lines": ["\u001b[0;34mUsing /tmp/ansible.mk27mu05.test/galaxy/ansible.cfg as config file\u001b[0m", "Starting galaxy collection install process", "Process install dependency map"]}

The unescaped stderr:

[WARNING]: The specified "api_version" configuration for the galaxy server
"pulp_v2" is not a public configuration, and may be removed at any time without
warning.
[WARNING]: The specified collections path
'/tmp/ansible.mk27mu05.test/publish/pulp_v2' is not part of the configured
Ansible collections paths
'/root/.ansible/collections:/usr/share/ansible/collections'. The installed
collection will not be picked up in an Ansible run, unless within a playbook-
adjacent collections directory.
ERROR! Error when getting the collection info for ansible_test.my_collection from pulp_v2 (http://galaxy-pulp/pulp_ansible/galaxy/primary/api/) (HTTP Code: 500, Message: Internal Server Error Code: Unknown)

webknjaz · 2023-12-11T18:03:23Z

Sticking a sleep 60 doesn't fix it, so the problem with the new Galaxy NG is deeper.

cc @sivel FYI

sivel · 2023-12-11T18:07:59Z

Seems like the only way to know for sure what is failing, is to attach to the container logs for the galaxy container, and see the traceback causing the 500 ISE and then go from there.

webknjaz · 2023-12-11T18:12:41Z

Yeah, that's what I thought.

webknjaz · 2023-12-11T18:15:38Z

I attached and saw it streaming things like

pulp [a0feea53400f4988b47590376543557d]: django.request:ERROR: Internal Server Error: /api/galaxy/v3/plugin/ansible/content/primary/collections/index/
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/django/core/handlers/exception.py", line 55, in inner
    response = get_response(request)
  File "/usr/local/lib/python3.8/site-packages/django/core/handlers/base.py", line 197, in _get_response
    response = wrapped_callback(request, *callback_args, **callback_kwargs)
  File "/usr/local/lib/python3.8/site-packages/django/views/decorators/csrf.py", line 56, in wrapper_view
    return view_func(*args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/rest_framework/viewsets.py", line 125, in view
    return self.dispatch(request, *args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/rest_framework/views.py", line 509, in dispatch
    response = self.handle_exception(exc)
  File "/usr/local/lib/python3.8/site-packages/rest_framework/views.py", line 469, in handle_exception
    self.raise_uncaught_exception(exc)
  File "/usr/local/lib/python3.8/site-packages/rest_framework/views.py", line 480, in raise_uncaught_exception
    raise exc
  File "/usr/local/lib/python3.8/site-packages/rest_framework/views.py", line 497, in dispatch
    self.initial(request, *args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/rest_framework/views.py", line 415, in initial
    self.check_permissions(request)
  File "/usr/local/lib/python3.8/site-packages/rest_framework/views.py", line 332, in check_permissions
    if not permission.has_permission(request, self):
  File "/usr/local/lib/python3.8/site-packages/rest_access_policy/access_policy.py", line 69, in has_permission
    allowed = self._evaluate_statements(statements, request, view, action)
  File "/usr/local/lib/python3.8/site-packages/rest_access_policy/access_policy.py", line 113, in _evaluate_statements
    matched = self._get_statements_matching_conditions(
  File "/usr/local/lib/python3.8/site-packages/rest_access_policy/access_policy.py", line 262, in _get_statements_matching_conditions
    passed = self._check_condition(condition, request, view, action)
  File "/usr/local/lib/python3.8/site-packages/rest_access_policy/access_policy.py", line 288, in _check_condition
    result = method(request, view, action)
  File "/usr/local/lib/python3.8/site-packages/galaxy_ng/app/access_control/access_policy.py", line 253, in v3_can_view_repo_content
    distro = ansible_models.AnsibleDistribution.objects.get(base_path=path)
  File "/usr/local/lib/python3.8/site-packages/django/db/models/manager.py", line 87, in manager_method
    return getattr(self.get_queryset(), name)(*args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/django/db/models/query.py", line 637, in get
    raise self.model.DoesNotExist(
pulp_ansible.app.models.AnsibleDistribution.DoesNotExist: AnsibleDistribution matching query does not exist.
pulp [a0feea53400f4988b47590376543557d]: 127.0.0.1 - admin [11/Dec/2023:18:14:34 +0000] "GET /api/galaxy/v3/plugin/ansible/content/primary/collections/index/?limit=100&offset=0 HTTP/1.0" 500 145 "-" "ansible-httpget"
pulp [f01fb13dcfa242b4b291212793ad06b6]: django.request:ERROR: Internal Server Error: /api/galaxy/v3/plugin/ansible/content/secondary/collections/index/
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/django/core/handlers/exception.py", line 55, in inner
    response = get_response(request)
  File "/usr/local/lib/python3.8/site-packages/django/core/handlers/base.py", line 197, in _get_response
    response = wrapped_callback(request, *callback_args, **callback_kwargs)
  File "/usr/local/lib/python3.8/site-packages/django/views/decorators/csrf.py", line 56, in wrapper_view
    return view_func(*args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/rest_framework/viewsets.py", line 125, in view
    return self.dispatch(request, *args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/rest_framework/views.py", line 509, in dispatch
    response = self.handle_exception(exc)
  File "/usr/local/lib/python3.8/site-packages/rest_framework/views.py", line 469, in handle_exception
    self.raise_uncaught_exception(exc)
  File "/usr/local/lib/python3.8/site-packages/rest_framework/views.py", line 480, in raise_uncaught_exception
    raise exc
  File "/usr/local/lib/python3.8/site-packages/rest_framework/views.py", line 497, in dispatch
    self.initial(request, *args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/rest_framework/views.py", line 415, in initial
    self.check_permissions(request)
  File "/usr/local/lib/python3.8/site-packages/rest_framework/views.py", line 332, in check_permissions
    if not permission.has_permission(request, self):
  File "/usr/local/lib/python3.8/site-packages/rest_access_policy/access_policy.py", line 69, in has_permission
    allowed = self._evaluate_statements(statements, request, view, action)
  File "/usr/local/lib/python3.8/site-packages/rest_access_policy/access_policy.py", line 113, in _evaluate_statements
    matched = self._get_statements_matching_conditions(
  File "/usr/local/lib/python3.8/site-packages/rest_access_policy/access_policy.py", line 262, in _get_statements_matching_conditions
    passed = self._check_condition(condition, request, view, action)
  File "/usr/local/lib/python3.8/site-packages/rest_access_policy/access_policy.py", line 288, in _check_condition
    result = method(request, view, action)
  File "/usr/local/lib/python3.8/site-packages/galaxy_ng/app/access_control/access_policy.py", line 253, in v3_can_view_repo_content
    distro = ansible_models.AnsibleDistribution.objects.get(base_path=path)
  File "/usr/local/lib/python3.8/site-packages/django/db/models/manager.py", line 87, in manager_method
    return getattr(self.get_queryset(), name)(*args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/django/db/models/query.py", line 637, in get
    raise self.model.DoesNotExist(
pulp_ansible.app.models.AnsibleDistribution.DoesNotExist: AnsibleDistribution matching query does not exist.

even before that download test...

webknjaz · 2023-12-11T18:17:26Z

And this is the end of the log (I paused the test playbook after running that failing download command):

pulp [0c53ee324d834cfaa90d4d605c486e87]: 127.0.0.1 - admin [11/Dec/2023:18:14:50 +0000] "POST /api/galaxy/v3/auth/token/ HTTP/1.0" 200 52 "-" "ansible-httpget"
pulp [8e4c4fd15e4544329b7495a59bce908d]: 127.0.0.1 - admin [11/Dec/2023:18:15:34 +0000] "POST /pulp_ansible/galaxy/primary/api/v2/collections/ HTTP/1.0" 202 78 "-" "ansible-galaxy/2.17.0.dev0 (Linux; python:3.12.0)"
pulp [8e4c4fd15e4544329b7495a59bce908d]: pulpcore.tasking.tasks:INFO: Starting task 018c5a17-6f71-7930-a58b-e0f94b80e573
pulp [8e4c4fd15e4544329b7495a59bce908d]: pulp_ansible.app.tasks.collections:INFO: Processing collection from tmp/files/01/8c5a17-6f65-73e8-9649-33a56a764ef7
pulp [8e4c4fd15e4544329b7495a59bce908d]: pulp_ansible.app.tasks.utils:INFO: Reading MANIFEST.json from tmp/files/01/8c5a17-6f65-73e8-9649-33a56a764ef7
pulp [8e4c4fd15e4544329b7495a59bce908d]: pulp_ansible.app.tasks.utils:INFO: Reading FILES.json from tmp/files/01/8c5a17-6f65-73e8-9649-33a56a764ef7
/usr/lib64/python3.8/tarfile.py:2230: RuntimeWarning: The default behavior of tarfile extraction has been changed to disallow common exploits (including CVE-2007-4559). By default, absolute/parent paths are disallowed and some mode bits are cleared. See https://access.redhat.com/articles/7004769 for more details.
  warnings.warn(
pulp [6edadb4393614c8aa74faf59532b5b89]: 127.0.0.1 - admin [11/Dec/2023:18:15:35 +0000] "GET /pulp_ansible/galaxy/primary/api/v2/collection-imports/018c5a17-6f71-7930-a58b-e0f94b80e573/ HTTP/1.0" 200 649 "-" "ansible-galaxy/2.17.0.dev0 (Linux; python:3.12.0)"
pulp [8e4c4fd15e4544329b7495a59bce908d]: pulpcore.tasking.tasks:INFO: Task completed 018c5a17-6f71-7930-a58b-e0f94b80e573
pulp [565b91ac991b45ddbdeece69b4e8edaa]: 127.0.0.1 - admin [11/Dec/2023:18:15:37 +0000] "GET /pulp_ansible/galaxy/primary/api/v2/collection-imports/018c5a17-6f71-7930-a58b-e0f94b80e573/ HTTP/1.0" 200 676 "-" "ansible-galaxy/2.17.0.dev0 (Linux; python:3.12.0)"
pulp [be92b47f2f5c4616b9d34eb0a74694e7]: django.request:ERROR: Internal Server Error: /pulp_ansible/galaxy/primary/api/v2/collections/ansible_test/my_collection/
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/django/core/handlers/exception.py", line 55, in inner
    response = get_response(request)
  File "/usr/local/lib/python3.8/site-packages/django/core/handlers/base.py", line 197, in _get_response
    response = wrapped_callback(request, *callback_args, **callback_kwargs)
  File "/usr/local/lib/python3.8/site-packages/django/views/decorators/csrf.py", line 56, in wrapper_view
    return view_func(*args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/django/views/generic/base.py", line 104, in view
    return self.dispatch(request, *args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/rest_framework/views.py", line 509, in dispatch
    response = self.handle_exception(exc)
  File "/usr/local/lib/python3.8/site-packages/rest_framework/views.py", line 469, in handle_exception
    self.raise_uncaught_exception(exc)
  File "/usr/local/lib/python3.8/site-packages/rest_framework/views.py", line 480, in raise_uncaught_exception
    raise exc
  File "/usr/local/lib/python3.8/site-packages/rest_framework/views.py", line 506, in dispatch
    response = handler(request, *args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/pulp_ansible/app/galaxy/views.py", line 154, in get
    return response.Response(GalaxyCollectionSerializer(collection, context=context).data)
  File "/usr/local/lib/python3.8/site-packages/rest_framework/serializers.py", line 555, in data
    ret = super().data
  File "/usr/local/lib/python3.8/site-packages/rest_framework/serializers.py", line 253, in data
    self._data = self.to_representation(self.instance)
  File "/usr/local/lib/python3.8/site-packages/rest_framework/serializers.py", line 522, in to_representation
    ret[field.field_name] = field.to_representation(attribute)
  File "/usr/local/lib/python3.8/site-packages/rest_framework/fields.py", line 1838, in to_representation
    return method(value)
  File "/usr/local/lib/python3.8/site-packages/pulp_ansible/app/galaxy/serializers.py", line 116, in get_latest_version
    "version": rv.version,
AttributeError: 'NoneType' object has no attribute 'version'
pulp [be92b47f2f5c4616b9d34eb0a74694e7]: 127.0.0.1 - admin [11/Dec/2023:18:15:38 +0000] "GET /pulp_ansible/galaxy/primary/api/v2/collections/ansible_test/my_collection/ HTTP/1.0" 500 145 "-" "ansible-galaxy/2.17.0.dev0 (Linux; python:3.12.0)"

sivel · 2023-12-11T18:17:53Z

Some of those may be expected. See:

ansible/test/integration/targets/ansible-galaxy-collection/library/reset_pulp.py

Lines 142 to 149 in a9919dd

    
           def get_repo_collections(repository, module): 
        
               collections_uri = module.params['galaxy_ng_server'] + 'v3/plugin/ansible/content/' + repository + '/collections/index/' 
        
               # status code 500 isn't really expected, an unhandled exception is causing this instead of a 404 
        
               # See https://issues.redhat.com/browse/AAH-2329 
        
               info = invoke_api(module, collections_uri + '?limit=100&offset=0', status_codes=[200, 500]) 
        
               if not info: 
        
                   return [] 
        
               return [module.params['pulp_api'] + c['href'] for c in info['data']]

webknjaz · 2023-12-11T18:18:34Z

jeez

webknjaz · 2023-12-11T18:22:03Z

I think this is the relevant bit

  File "/usr/local/lib/python3.8/site-packages/pulp_ansible/app/galaxy/serializers.py", line 116, in get_latest_version
    "version": rv.version,
AttributeError: 'NoneType' object has no attribute 'version'

which suggests another bug in pulp_ansible again.

This patch adjusts the metadata processing to swallow 404 and handle other cases as a cache reset request. Ref: ansible#82396 (comment)

It updates the underlying `pulp_ansible` to v0.20.2, which fixes a race condition that is occasionally observable in our CI when provisioning the initial dummy collections for testing. Refs: * https://github.com/ansible/galaxy_ng/releases/tag/4.9.0 * ansible/galaxy_ng#1950 * ansible/galaxy#3267 * pulp/pulp_ansible#1571 * pulp/pulp_ansible#1572 * pulp/pulp_ansible#1595 * pulp/pulp_ansible#1623 * pulp/pulp_ansible#1624

Co-authored-by: Sloane Hertel <19572925+s-hertel@users.noreply.github.com>

This patch adjusts the metadata processing to swallow 404 and handle other cases as a cache reset request. Ref: ansible#82396 (comment)

s-hertel · 2025-03-17T12:11:48Z

+                if err.http_code == 404:
+                    # No collection found, return an empty list to keep things consistent with the various APIs
+                    return []
+                elif err.http_code != 500 or 'v3' in self.available_api_versions:
                    raise
-                # No collection found, return an empty list to keep things consistent with the various APIs
-                return []
+
+                # v2 appears to have a bug on some versions, where the metadata URL is an HTTP error 500
+                # since this URL is only used to handle the cache, proceed as if the cache needs to be refreshed.
+                modified_date = None
+                if '%s.%s' % (namespace, name) in modified_cache:
+                    del modified_cache['%s.%s' % (namespace, name)]
+                    self._set_cache()


@webknjaz Since v2 GalaxyAPI is deprecated and close to removal, I'm second-guessing this. Can we just remove the v2 tests early, instead of adding code to support the lack of fully functional v2 in 4.9.2?

Or, could we keep using an older galaxy ng for v2 coverage until 2.20 development starts and we can delete everything v2?

@s-hertel I think that coverage is a good reason to keep this. But also, I was looking forward to backporting this to reduce the CI flakiness. I feel like this might be okay to backport, but removals wouldn't be.

Should we merge+backport this and then delete whatever in follow-ups when 2.20 comes? The main value in this for me is the CI stability, which spans across several branches IIRC.

cc @mattclay would this be acceptable to backport for the sake of CI?

Since this is a bug fix for V2, it seems like a good idea to backport it. We should continue testing V2 for versions that support it. Once we branch for 2.20 we'll be able to rip out all the V2 code and tests.

If v2 was used by anything other than old galaxy and custom servers mimicking the v2 endpoint, I'd be more in favor of backporting this. This seems featureish for 3rd party galaxy servers, which should conform with https://old-galaxy.ansible.com v2 or GalaxyNG v3.

IMO we should not be adding more code for v2, unless it fixes issues with https://old-galaxy.ansible.com.

@s-hertel doesn't it fix issues with our own CI where it uses v2 in old branches? How else would we address that?

@webknjaz How will we know fixes on old branches are compatible with https://old-galaxy.ansible.com/ if we no longer test against a compatible v2? Specifying the api version is internal, so if someone's using a recent version of GalaxyNG, they'll be using v3 via the public API, not v2. We should keep backwards compat with old galaxy since that's how people use v2.

Or, could we keep using an older galaxy ng for v2 coverage until 2.20 development starts and we can delete everything v2?

Have you taken a look at which GalaxyNG version it actually breaks in and why? Can the image can be updated to a lesser extent to pull in some of the linked fixes and still work with v2?

Dunno, I haven't looked deeper. I was originally hoping for just an image update, but these changes grew on top, as you and sivel suggested.

mattclay

I think this looks good overall. Since we're changing code, not just tests, this should be turned into a bugfix and have a changelog fragment.

mattclay · 2025-03-18T15:49:39Z

+                if err.http_code == 404:
+                    # No collection found, return an empty list to keep things consistent with the various APIs
+                    return []
+                elif err.http_code != 500 or 'v3' in self.available_api_versions:
                    raise
-                # No collection found, return an empty list to keep things consistent with the various APIs
-                return []
+
+                # v2 appears to have a bug on some versions, where the metadata URL is an HTTP error 500
+                # since this URL is only used to handle the cache, proceed as if the cache needs to be refreshed.
+                modified_date = None
+                if '%s.%s' % (namespace, name) in modified_cache:
+                    del modified_cache['%s.%s' % (namespace, name)]
+                    self._set_cache()


Since this is a bug fix for V2, it seems like a good idea to backport it. We should continue testing V2 for versions that support it. Once we branch for 2.20 we'll be able to rip out all the V2 code and tests.

mattclay · 2025-03-20T05:39:47Z

@webknjaz Have you confirmed that the issue this is trying to fix is still occurring in the devel branch? #84763 included a work-around for one of the galaxy issues we were seeing while working on DT.

webknjaz · 2025-03-20T15:03:29Z

@mattclay no, I haven't. I guess I missed that PR. It looks like it might've fixed it. I'll keep an eye out for this test failures.
Are you going to backport that into older branches?

webknjaz · 2025-03-20T15:24:05Z

@mattclay I think this is an example of said issue on stable-2.16: https://dev.azure.com/ansible/ansible/_build/results?buildId=140568&view=logs&j=70eeb688-fc21-5491-4420-b60b604f62ec&t=d6f17d19-0bb3-5e63-8427-e2ae23c55414.

mattclay · 2025-03-20T15:52:21Z

@webknjaz I wasn't planning on backporting the entire PR, but it makes sense to backport the galaxy test fixes. Can you open backports for just those fixes?

webknjaz · 2025-04-01T13:36:17Z

Looks like devel still has problems with these tests: https://dev.azure.com/ansible/ansible/_build/results?buildId=141717&view=logs&j=02bf57c0-d919-5e2d-1c94-7b4ef705afdf&t=a9b57ab5-2f0b-5a77-0352-06fbabf684da&l=5867 (it's from @s-hertel's PR #84911). Should we still consider this PR, then?

webknjaz · 2025-04-10T14:12:44Z

stable-2.16: https://dev.azure.com/ansible/ansible/_build/results?buildId=142510&view=logs&j=70eeb688-fc21-5491-4420-b60b604f62ec&t=d6f17d19-0bb3-5e63-8427-e2ae23c55414&l=5231

webknjaz · 2025-04-15T13:58:48Z

stable-2.16: https://dev.azure.com/ansible/ansible/_build/results?buildId=142998&view=logs&j=114ad447-3cf5-5c3e-d6a4-0cd0b6d5c2c7&s=d65b03ed-7a7b-54a6-b1eb-60fdb469012e&t=04c2859f-ee16-5564-2ccc-f6dee4dfc978&l=5235

devel: https://dev.azure.com/ansible/ansible/_build/results?buildId=142999&view=logs&j=02bf57c0-d919-5e2d-1c94-7b4ef705afdf&t=a9b57ab5-2f0b-5a77-0352-06fbabf684da&l=5606 / https://dev.azure.com/ansible/ansible/_build/results?buildId=142999&view=logs&j=70eeb688-fc21-5491-4420-b60b604f62ec&t=d6f17d19-0bb3-5e63-8427-e2ae23c55414&l=5607

@mattclay this makes me think that #84763 didn't fully address the issue and merging DT made it more frequent (two jobs in the same CI run failing like this used to be rare).

mattclay · 2025-04-15T20:31:59Z

@webknjaz Thanks for monitoring this issue. Since the random sleep only improves the situation, rather than eliminating the it completely, it occurred to me that we could simply disable the parallel publishing in the test. If the parallel publishing is the source of this issue, then #85000 should resolve it.

webknjaz · 2025-04-16T11:31:16Z

@mattclay thanks! It doesn't seem to have helped: #85000 (comment). Pretty sure that it is failing on cleanup, not on publishing.

mattclay · 2025-04-16T15:57:20Z

Yeah, this looks like a different error though -- not related to the publishing problems we were seeing.

webknjaz · 2025-04-17T08:35:16Z

Yeah, this looks like a different error though -- not related to the publishing problems we were seeing.

@mattclay perhaps we were talking about different things, then. I checked a few log links in my previous comments above, and all of them exhibit the same race condition in the pulp reset task.

Here's one from today's nightly: https://dev.azure.com/ansible/ansible/_build/results?buildId=143192&view=logs&j=02bf57c0-d919-5e2d-1c94-7b4ef705afdf&t=a9b57ab5-2f0b-5a77-0352-06fbabf684da&l=5473

mattclay · 2025-04-17T21:31:38Z

@webknjaz I think part of the problem is that there are different errors for the initial run and the retry. To make things easier to discuss and troubleshoot, I'm disabling the retry on that integration test: #85013

mattclay · 2025-04-18T16:49:49Z

Now that the test isn't retrying, and isn't publishing in parallel, we're now seeing the publishing step time out instead:

https://dev.azure.com/ansible/ansible/_build/results?buildId=143289&view=logs&jobId=70eeb688-fc21-5491-4420-b60b604f62ec&j=70eeb688-fc21-5491-4420-b60b604f62ec&t=d6f17d19-0bb3-5e63-8427-e2ae23c55414

This timeout was occurring previously, but it was more difficult to notice because the test was being retried, and the retry failure was due to concurrency issues -- most likely because the previous publish was still running.

I've opened #85016 to increase the publish timeout, to see if that's sufficient to get the tests to pass consistently.

webknjaz · 2025-04-22T15:20:06Z

I've opened #85016 to increase the publish timeout, to see if that's sufficient to get the tests to pass consistently.

Looks like that did help. And you're right: I was confusing normal runs vs. reruns.

webknjaz · 2025-04-25T16:27:29Z

Here stable-2.18 hit that unique_is_highest bug earlier today:

{
  "rc": 1,
  "stderr": "\u001b[1;35m[WARNING]: The specified \"api_version\" configuration for the galaxy server\u001b[0m\n\u001b[1;35m\"pulp_v2\" is not a public configuration, and may be removed at any time without\u001b[0m\n\u001b[1;35mwarning.\u001b[0m\n\u001b[1;35m[WARNING]: Galaxy import warning message: No changelog found. Add a\u001b[0m\n\u001b[1;35mCHANGELOG.rst, CHANGELOG.md, or changelogs/changelog.yaml file.\u001b[0m\n\u001b[0;31mERROR! Galaxy import process failed: duplicate key value violates unique constraint \"unique_is_highest\"\u001b[0m\n\u001b[0;31mDETAIL:  Key (collection_id, is_highest)=(839aeb95-f5e3-4a2e-a853-1a73a8e4767c, t) already exists.\u001b[0m\n\u001b[0;31m (Code: UNKNOWN)\u001b[0m\n",
  "stdout": "Publishing collection artifact '/root/.ansible/tmp/ansible-moduletmp-1745565311.5552013-pjw5iaof/namespace1-name1-0.0.7/namespace1-name1-0.0.7.tar.gz' to galaxy_ng [http://galaxy-pulp/api/galaxy/content/primary/\nCollection](http://galaxy-pulp/api/galaxy/content/primary//nCollection) has been published to the Galaxy server galaxy_ng [http://galaxy-pulp/api/galaxy/content/primary/\nWaiting](http://galaxy-pulp/api/galaxy/content/primary//nWaiting) until Galaxy import task http://galaxy-pulp/api/galaxy/content/primary/v3/imports/collections/2b6dc0bb-d568-4d8f-bd9e-3d23598fa3ab/ has completed\n"
}

(https://dev.azure.com/ansible/ansible/_build/results?buildId=144050&view=logs&j=70eeb688-fc21-5491-4420-b60b604f62ec&t=d6f17d19-0bb3-5e63-8427-e2ae23c55414&l=1014)

webknjaz · 2025-04-28T15:39:18Z

stable-2.17, yesterday:
https://dev.azure.com/ansible/ansible/_build/results?buildId=144220&view=logs&j=70eeb688-fc21-5491-4420-b60b604f62ec&t=d6f17d19-0bb3-5e63-8427-e2ae23c55414&l=1014

webknjaz requested a review from sivel December 11, 2023 13:52

webknjaz self-assigned this Dec 11, 2023

ansibot added test This PR relates to tests. needs_triage Needs a first human triage before being processed. needs_revision This PR fails CI tests or a maintainer has requested a review/revision of the PR. labels Dec 11, 2023

webknjaz added the ci_verified Changes made in this PR are causing tests to fail. label Dec 11, 2023

bcoca removed the needs_triage Needs a first human triage before being processed. label Dec 12, 2023

ansibot added the stale_ci This PR has been tested by CI more than one week ago. Close and re-open this PR to get it retested. label Dec 19, 2023

s-hertel reviewed Jul 30, 2024

View reviewed changes

Comment thread test/lib/ansible_test/_internal/commands/integration/cloud/galaxy.py Outdated

ansibot added the stale_pr This PR has not been pushed to for more than one year. label Jan 28, 2025

webknjaz force-pushed the testing/integration-galaxy-ng-4.9.0 branch from 8a74e33 to ecf2426 Compare February 5, 2025 02:13

ansibot removed stale_pr This PR has not been pushed to for more than one year. ci_verified Changes made in this PR are causing tests to fail. stale_ci This PR has been tested by CI more than one week ago. Close and re-open this PR to get it retested. labels Feb 5, 2025

ansibot added the stale_ci This PR has been tested by CI more than one week ago. Close and re-open this PR to get it retested. label Feb 19, 2025

webknjaz force-pushed the testing/integration-galaxy-ng-4.9.0 branch from ecf2426 to e1b0878 Compare February 27, 2025 22:55

ansibot removed the stale_ci This PR has been tested by CI more than one week ago. Close and re-open this PR to get it retested. label Feb 27, 2025

webknjaz pushed a commit to webknjaz/ansible that referenced this pull request Mar 3, 2025

Work around broken v2 metadata endpoint in Pulp

9a9aba3

This patch adjusts the metadata processing to swallow 404 and handle other cases as a cache reset request. Ref: ansible#82396 (comment)

webknjaz and others added 3 commits March 3, 2025 17:05

🧪⇪ Bump galaxy_ng to v4.9.2

053d55d

Co-authored-by: Sloane Hertel <19572925+s-hertel@users.noreply.github.com>

Work around broken v2 metadata endpoint in Pulp

7470cf2

This patch adjusts the metadata processing to swallow 404 and handle other cases as a cache reset request. Ref: ansible#82396 (comment)

Update galaxy publish test for non-erroring Pulp

cd32765

webknjaz force-pushed the testing/integration-galaxy-ng-4.9.0 branch from 9a9aba3 to cd32765 Compare March 3, 2025 16:05

webknjaz changed the title ~~🧪⇪ Bump galaxy_ng to v4.9.0 in tests~~ 🧪⇪ Bump galaxy_ng to v4.9.2 in tests Mar 3, 2025

webknjaz requested a review from s-hertel March 3, 2025 16:05

webknjaz moved this from 📑 Ready 👌 to 🫸In review🫷 in 📅 Procrastinating in public 😵‍💫 Mar 3, 2025

ansibot removed the needs_revision This PR fails CI tests or a maintainer has requested a review/revision of the PR. label Mar 3, 2025

s-hertel reviewed Mar 17, 2025

View reviewed changes

ansibot added the stale_ci This PR has been tested by CI more than one week ago. Close and re-open this PR to get it retested. label Mar 17, 2025

webknjaz requested a review from mattclay March 18, 2025 15:39

mattclay reviewed Mar 18, 2025

View reviewed changes

webknjaz moved this from 🫸In review🫷 to 🚧 In progress 🚧 in 📅 Procrastinating in public 😵‍💫 Apr 15, 2025

ansibot added the stale_pr This PR has not been pushed to for more than one year. label Mar 7, 2026

ansibot added the needs_rebase https://docs.ansible.com/ansible/devel/dev_guide/developing_rebasing.html label Mar 25, 2026

Conversation

webknjaz commented Dec 11, 2023

SUMMARY

ISSUE TYPE

ADDITIONAL INFORMATION

Uh oh!

webknjaz commented Dec 11, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

webknjaz commented Dec 11, 2023

Uh oh!

sivel commented Dec 11, 2023

Uh oh!

webknjaz commented Dec 11, 2023

Uh oh!

webknjaz commented Dec 11, 2023

Uh oh!

webknjaz commented Dec 11, 2023

Uh oh!

sivel commented Dec 11, 2023

Uh oh!

webknjaz commented Dec 11, 2023

Uh oh!

webknjaz commented Dec 11, 2023

Uh oh!

Uh oh!

s-hertel Mar 17, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

webknjaz Mar 17, 2025

Choose a reason for hiding this comment

Uh oh!

mattclay Mar 18, 2025

Choose a reason for hiding this comment

Uh oh!

s-hertel Mar 18, 2025

Choose a reason for hiding this comment

Uh oh!

webknjaz Mar 18, 2025

Choose a reason for hiding this comment

Uh oh!

s-hertel Mar 18, 2025

Choose a reason for hiding this comment

Uh oh!

webknjaz Mar 18, 2025

Choose a reason for hiding this comment

Uh oh!

mattclay left a comment

Choose a reason for hiding this comment

Uh oh!

mattclay Mar 18, 2025

Choose a reason for hiding this comment

Uh oh!

mattclay commented Mar 20, 2025

Uh oh!

webknjaz commented Mar 20, 2025

Uh oh!

webknjaz commented Mar 20, 2025

Uh oh!

mattclay commented Mar 20, 2025

Uh oh!

webknjaz commented Apr 1, 2025

Uh oh!

webknjaz commented Apr 10, 2025

Uh oh!

webknjaz commented Apr 15, 2025

Uh oh!

mattclay commented Apr 15, 2025

Uh oh!

webknjaz commented Apr 16, 2025

Uh oh!

mattclay commented Apr 16, 2025

Uh oh!

webknjaz commented Apr 17, 2025

Uh oh!

mattclay commented Apr 17, 2025

Uh oh!

mattclay commented Apr 18, 2025

Uh oh!

webknjaz commented Dec 11, 2023 •

edited

Loading

s-hertel Mar 17, 2025 •

edited

Loading