Unable to install some galaxy collections #14495

MatthieuLeMee · 2023-10-02T11:53:26Z

Please confirm the following

I agree to follow this project's code of conduct.
I have checked the current issues for duplicates.
I understand that AWX is open source software provided for free and that I might not receive a timely response.
I am NOT reporting a (potential) security vulnerability. (These should be emailed to security@ansible.com instead.)

Bug Summary

I'm on minikube and AWX 21.8.0 for months without any problem. Since some hours, AWX cannot install some collections anymore from Ansible Galaxy platform.

After some investigation, it seems it has to do with a high number of publicated versions of collections. I did not change any configuration on AWX since months and i don't know why it stars to fail only today.

Some collections i tried (with versions count) :

community.general		131	ERROR
ansible.netcommon		164	ERROR
community.vmware		338	ERROR
community.grafana		19	INSTALL OK
community.sops			24	INSTALL OK
galaxyproject.general	1	INSTALL OK
community.sap_install	8	INSTALL OK
community.crypto		69	INSTALL OK
community.digitalocean	31	INSTALL OK
community.mysql			52	INSTALL OK
community.postgresql	40	INSTALL OK
community.docker		75	INSTALL OK

I also tried to purge some filesystem use in minikube with docker image prune -af reclaiming 40GB on 82GB total to avoid filesystem being full but it doesn't seem related. VM and minikube has not been restarted since 10 months.

In all my deployments, i'm using collections/requirements.yml file with :

---
  - name: community.general

AWX version

21.8.0

Select the relevant components

Installation method

minikube

Modifications

no

Ansible version

2.15.1

Operating system

CentOS Linux release 7.9.2009 (Core)

Web browser

No response

Steps to reproduce

Install AWX 21.8.0
Create a git repository with collections/requirements.yml file including community.general
Try project update from git sources

Expected results

Big collections installing fine.

Actual results

Error in project update:

[WARNING]: Skipping Galaxy server https://galaxy.ansible.com/api/. Got an
unexpected error when getting available versions of collection
community.general: '/api/v3/plugin/ansible/content/published/collections/index/
community/general/versions/'
ERROR! Unexpected Exception, this is probably a bug: '/api/v3/plugin/ansible/content/published/collections/index/community/general/versions/'

And full stacktrace is

ansible-galaxy [core 2.12.5.post0]
  config file = None
  configured module search path = ['/home/runner/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /usr/local/lib/python3.8/site-packages/ansible
  ansible collection location = /var/lib/awx/projects/.__awx_cache/_1690__awx_error_test_dev/stage/requirements_collections
  executable location = /usr/local/bin/ansible-galaxy
  python version = 3.8.13 (default, Jun 24 2022, 15:27:57) [GCC 8.5.0 20210514 (Red Hat 8.5.0-13)]
  jinja version = 2.11.3
  libyaml = True
No config file found; using defaults
Reading requirement file at '/var/lib/awx/projects/_1690__awx_error_test_dev/deployment/collections/requirements.yml'
Starting galaxy collection install process
Process install dependency map
Opened /home/runner/.ansible/galaxy_token
the full traceback was:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/resolvelib/resolvers.py", line 171, in _merge_into_criterion
    crit = self.state.criteria[name]
KeyError: 'community.general'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/bin/ansible-galaxy", line 128, in <module>
    exit_code = cli.run()
  File "/usr/local/lib/python3.8/site-packages/ansible/cli/galaxy.py", line 569, in run
    return context.CLIARGS['func']()
  File "/usr/local/lib/python3.8/site-packages/ansible/cli/galaxy.py", line 86, in method_wrapper
    return wrapped_method(*args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/ansible/cli/galaxy.py", line 1203, in execute_install
    self._execute_install_collection(
  File "/usr/local/lib/python3.8/site-packages/ansible/cli/galaxy.py", line 1230, in _execute_install_collection
    install_collections(
  File "/usr/local/lib/python3.8/site-packages/ansible/galaxy/collection/__init__.py", line 548, in install_collections
    dependency_map = _resolve_depenency_map(
  File "/usr/local/lib/python3.8/site-packages/ansible/galaxy/collection/__init__.py", line 1364, in _resolve_depenency_map
    return collection_dep_resolver.resolve(
  File "/usr/local/lib/python3.8/site-packages/resolvelib/resolvers.py", line 453, in resolve
    state = resolution.resolve(requirements, max_rounds=max_rounds)
  File "/usr/local/lib/python3.8/site-packages/resolvelib/resolvers.py", line 318, in resolve
    name, crit = self._merge_into_criterion(r, parent=None)
  File "/usr/local/lib/python3.8/site-packages/resolvelib/resolvers.py", line 173, in _merge_into_criterion
    crit = Criterion.from_requirement(self._p, requirement, parent)
  File "/usr/local/lib/python3.8/site-packages/resolvelib/resolvers.py", line 79, in from_requirement
    cands = build_iter_view(provider.find_matches([requirement]))
  File "/usr/local/lib/python3.8/site-packages/ansible/galaxy/dependency_resolution/providers.py", line 214, in find_matches
    coll_versions = self._api_proxy.get_collection_versions(first_req)
  File "/usr/local/lib/python3.8/site-packages/ansible/galaxy/collection/galaxy_api_proxy.py", line 101, in get_collection_versions
    return set(
  File "/usr/local/lib/python3.8/site-packages/ansible/galaxy/collection/galaxy_api_proxy.py", line 101, in <genexpr>
    return set(
  File "/usr/local/lib/python3.8/site-packages/ansible/galaxy/collection/galaxy_api_proxy.py", line 82, in _get_collection_versions
    raise last_error
  File "/usr/local/lib/python3.8/site-packages/ansible/galaxy/collection/galaxy_api_proxy.py", line 61, in _get_collection_versions
    versions = api.get_collection_versions(requirement.namespace, requirement.name)
  File "/usr/local/lib/python3.8/site-packages/ansible/galaxy/api.py", line 126, in wrapped
    return method(self, *args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/ansible/galaxy/api.py", line 869, in get_collection_versions
    data = self._call_galaxy(to_native(next_link, errors='surrogate_or_strict'),
  File "/usr/local/lib/python3.8/site-packages/ansible/module_utils/api.py", line 157, in run_function
    return call_retryable_function()
  File "/usr/local/lib/python3.8/site-packages/ansible/galaxy/api.py", line 396, in _call_galaxy
    path_cache = self._cache[cache_id][url_info.path]
KeyError: '/api/v3/plugin/ansible/content/published/collections/index/community/general/versions/'

Additional information

No response

The text was updated successfully, but these errors were encountered:

gbjgro3 · 2023-10-02T13:16:58Z

Same issue, was additionally effecting rebuilding an execution environment [ansible-build] with same error

JKetelaar · 2023-10-02T13:28:41Z

Same issue here...

al-lac · 2023-10-02T13:28:54Z

Yup, same issue as well. And we are running on 23.1.0.

We use the community collections a lot, so this breaks a lot of automations.

Frewx · 2023-10-02T14:06:51Z

Same issue on community.general with AWX-operator versions of both 22.6.0 and 21.10.1.
ERROR! Unexpected Exception, this is probably a bug: '/api/v3/plugin/ansible/content/published/collections/index/community/general/versions/'

lucasvivian · 2023-10-02T14:21:27Z

Same here, version 23.0.0. Update to 23.2.0 dont have any effect.

al-lac · 2023-10-02T14:31:47Z

What ansible version is AWX or the Default Execution Environment using?

Apparently there was a bug in ansible 2.13 that caused this issue, see the post on the ansible forum here: https://forum.ansible.com/t/unable-to-install-collection-from-new-galaxy/1238/4?u=alex

MatthieuLeMee · 2023-10-02T14:53:57Z

I suspect the problem to be in the awx-ee:latest as it's the only component which his upgraded automatically.

I'm currently running ansible 2.12.5

ansible-playbook [core 2.12.5.post0]
  config file = None
  configured module search path = ['/home/runner/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /usr/local/lib/python3.8/site-packages/ansible
  ansible collection location = /home/runner/.ansible/collections:/usr/share/ansible/collections
  executable location = /usr/local/bin/ansible-playbook
  python version = 3.8.13 (default, Jun 24 2022, 15:27:57) [GCC 8.5.0 20210514 (Red Hat 8.5.0-13)]
  jinja version = 2.11.3
  libyaml = True
No config file found; using defaults
host_list declined parsing /tmp/awx_75799_t1qyfgog/inventory/hosts as it did not pass its verify_file() method
script declined parsing /tmp/awx_75799_t1qyfgog/inventory/hosts as it did not pass its verify_file() method
auto declined parsing /tmp/awx_75799_t1qyfgog/inventory/hosts as it did not pass its verify_file() method
Parsed /tmp/awx_75799_t1qyfgog/inventory/hosts inventory source with ini plugin
Skipping callback 'awx_display', as we already have a stdout callback.
Skipping callback 'default', as we already have a stdout callback.
Skipping callback 'minimal', as we already have a stdout callback.
Skipping callback 'oneline', as we already have a stdout callback.

PLAYBOOK: project_update.yml ***************************************************
3 plays in project_update.yml

PLAY [Update source tree if necessary] *****************************************
META: ran handlers

TASK [delete project directory before update] **********************************
task path: /tmp/awx_75799_t1qyfgog/project/project_update.yml:29
<127.0.0.1> ESTABLISH LOCAL CONNECTION FOR USER: root
<127.0.0.1> EXEC /bin/sh -c 'echo ~root && sleep 0'
<127.0.0.1> EXEC /bin/sh -c '( umask 77 && mkdir -p "` echo /root/.ansible/tmp `"&& mkdir "` echo /root/.ansible/tmp/ansible-tmp-1696253911.173594-2555281-34008202159417 `" && echo ansible-tmp-1696253911.173594-2555281-34008202159417="` echo /root/.ansible/tmp/ansible-tmp-1696253911.173594-2555281-34008202159417 `" ) && sleep 0'
Using module file /usr/local/lib/python3.8/site-packages/ansible/modules/command.py
<127.0.0.1> PUT /home/runner/.ansible/tmp/ansible-local-2555276mi_um33f/tmpjv15sgc3 TO /root/.ansible/tmp/ansible-tmp-1696253911.173594-2555281-34008202159417/AnsiballZ_command.py
<127.0.0.1> EXEC /bin/sh -c 'chmod u+x /root/.ansible/tmp/ansible-tmp-1696253911.173594-2555281-34008202159417/ /root/.ansible/tmp/ansible-tmp-1696253911.173594-2555281-34008202159417/AnsiballZ_command.py && sleep 0'
<127.0.0.1> EXEC /bin/sh -c '/usr/bin/python3 /root/.ansible/tmp/ansible-tmp-1696253911.173594-2555281-34008202159417/AnsiballZ_command.py && sleep 0'
<127.0.0.1> EXEC /bin/sh -c 'rm -f -r /root/.ansible/tmp/ansible-tmp-1696253911.173594-2555281-34008202159417/ > /dev/null 2>&1 && sleep 0'
changed: [localhost] => {
    "changed": true,
    "cmd": [
        "find",
        "-delete"
    ],
    "delta": "0:00:00.010054",
    "end": "2023-10-02 13:38:31.527952",
    "invocation": {
        "module_args": {
            "_raw_params": "find -delete",
            "_uses_shell": false,
            "argv": null,
            "chdir": "/var/lib/awx/projects/_1690__awx_error_test_dev",
            "creates": null,
            "executable": null,
            "removes": null,
            "stdin": null,
            "stdin_add_newline": true,
            "strip_empty_ends": true,
            "warn": false
        }
    },
    "msg": "",
    "rc": 0,
    "start": "2023-10-02 13:38:31.517898",
    "stderr": "",
    "stderr_lines": [],
    "stdout": "",
    "stdout_lines": []
}

TASK [update project using git] ************************************************
task path: /tmp/awx_75799_t1qyfgog/project/project_update.yml:37
<127.0.0.1> ESTABLISH LOCAL CONNECTION FOR USER: root
<127.0.0.1> EXEC /bin/sh -c 'echo ~root && sleep 0'
<127.0.0.1> EXEC /bin/sh -c '( umask 77 && mkdir -p "` echo /root/.ansible/tmp `"&& mkdir "` echo /root/.ansible/tmp/ansible-tmp-1696253911.6062546-2555298-214660540045267 `" && echo ansible-tmp-1696253911.6062546-2555298-214660540045267="` echo /root/.ansible/tmp/ansible-tmp-1696253911.6062546-2555298-214660540045267 `" ) && sleep 0'
Using module file /usr/local/lib/python3.8/site-packages/ansible/modules/git.py
<127.0.0.1> PUT /home/runner/.ansible/tmp/ansible-local-2555276mi_um33f/tmphmopcz5l TO /root/.ansible/tmp/ansible-tmp-1696253911.6062546-2555298-214660540045267/AnsiballZ_git.py
<127.0.0.1> EXEC /bin/sh -c 'chmod u+x /root/.ansible/tmp/ansible-tmp-1696253911.6062546-2555298-214660540045267/ /root/.ansible/tmp/ansible-tmp-1696253911.6062546-2555298-214660540045267/AnsiballZ_git.py && sleep 0'
<127.0.0.1> EXEC /bin/sh -c '/usr/bin/python3 /root/.ansible/tmp/ansible-tmp-1696253911.6062546-2555298-214660540045267/AnsiballZ_git.py && sleep 0'
<127.0.0.1> EXEC /bin/sh -c 'rm -f -r /root/.ansible/tmp/ansible-tmp-1696253911.6062546-2555298-214660540045267/ > /dev/null 2>&1 && sleep 0'
changed: [localhost] => {
    "after": "ac79a6ed72f6c1ff6c6af652c50792e0627f040f",
    "before": null,
    "changed": true,
    "invocation": {
        "module_args": {
            "accept_hostkey": false,
            "accept_newhostkey": false,
            "archive": null,
            "archive_prefix": null,
            "bare": false,
            "clone": true,
            "depth": null,
            "dest": "/var/lib/awx/projects/_1690__awx_error_test_dev",
            "executable": null,
            "force": true,
            "gpg_whitelist": [],
            "key_file": null,
            "recursive": true,
            "reference": null,
            "refspec": "refs/heads/*:refs/remotes/origin/*",
            "remote": "origin",
            "repo": "https://$encrypted$:$encrypted$@gitlab.com/groupeima/socle/poc/awx-errortest.git",
            "separate_git_dir": null,
            "single_branch": false,
            "ssh_opts": null,
            "track_submodules": false,
            "umask": null,
            "update": true,
            "verify_commit": false,
            "version": "master"
        }
    }
}

TASK [Set the git repository version] ******************************************
task path: /tmp/awx_75799_t1qyfgog/project/project_update.yml:48
ok: [localhost] => {
    "ansible_facts": {
        "scm_version": "ac79a6ed72f6c1ff6c6af652c50792e0627f040f"
    },
    "changed": false
}

TASK [Repository Version] ******************************************************
task path: /tmp/awx_75799_t1qyfgog/project/project_update.yml:148
ok: [localhost] => {
    "msg": "Repository Version ac79a6ed72f6c1ff6c6af652c50792e0627f040f"
}
META: ran handlers
META: ran handlers

PLAY [Perform project signature/checksum verification] *************************
META: ran handlers
META: ran handlers
META: ran handlers

PLAY [Install content with ansible-galaxy command if necessary] ****************
META: ran handlers

TASK [debug] *******************************************************************
task path: /tmp/awx_75799_t1qyfgog/project/project_update.yml:194
skipping: [localhost] => {}

TASK [meta] ********************************************************************
task path: /tmp/awx_75799_t1qyfgog/project/project_update.yml:199
META: 
skipping: [localhost] => {
    "msg": "",
    "skip_reason": "end_play conditional evaluated to False, continuing play"
}

TASK [fetch galaxy roles from deployment/roles/requirements.yml] ***************
task path: /tmp/awx_75799_t1qyfgog/project/project_update.yml:206
<127.0.0.1> ESTABLISH LOCAL CONNECTION FOR USER: root
<127.0.0.1> EXEC /bin/sh -c 'echo ~root && sleep 0'
<127.0.0.1> EXEC /bin/sh -c '( umask 77 && mkdir -p "` echo /root/.ansible/tmp `"&& mkdir "` echo /root/.ansible/tmp/ansible-tmp-1696253915.5521088-2555354-140565239184359 `" && echo ansible-tmp-1696253915.5521088-2555354-140565239184359="` echo /root/.ansible/tmp/ansible-tmp-1696253915.5521088-2555354-140565239184359 `" ) && sleep 0'
Using module file /usr/local/lib/python3.8/site-packages/ansible/modules/command.py
<127.0.0.1> PUT /home/runner/.ansible/tmp/ansible-local-2555276mi_um33f/tmpnvtp16y_ TO /root/.ansible/tmp/ansible-tmp-1696253915.5521088-2555354-140565239184359/AnsiballZ_command.py
<127.0.0.1> EXEC /bin/sh -c 'chmod u+x /root/.ansible/tmp/ansible-tmp-1696253915.5521088-2555354-140565239184359/ /root/.ansible/tmp/ansible-tmp-1696253915.5521088-2555354-140565239184359/AnsiballZ_command.py && sleep 0'
<127.0.0.1> EXEC /bin/sh -c 'ANSIBLE_FORCE_COLOR=false GIT_SSH_COMMAND='"'"'ssh -o StrictHostKeyChecking=no'"'"' /usr/bin/python3 /root/.ansible/tmp/ansible-tmp-1696253915.5521088-2555354-140565239184359/AnsiballZ_command.py && sleep 0'
<127.0.0.1> EXEC /bin/sh -c 'rm -f -r /root/.ansible/tmp/ansible-tmp-1696253915.5521088-2555354-140565239184359/ > /dev/null 2>&1 && sleep 0'
changed: [localhost] => (item=/var/lib/awx/projects/_1690__awx_error_test_dev/deployment/roles/requirements.yml) => {
    "ansible_loop_var": "item",
    "changed": true,
    "cmd": [
        "ansible-galaxy",
        "role",
        "install",
        "-r",
        "/var/lib/awx/projects/_1690__awx_error_test_dev/deployment/roles/requirements.yml",
        "--roles-path",
        "/var/lib/awx/projects/.__awx_cache/_1690__awx_error_test_dev/stage/requirements_roles",
        "-vvv"
    ],
    "delta": "0:00:01.556536",
    "end": "2023-10-02 13:38:37.304106",
    "invocation": {
        "module_args": {
            "_raw_params": "ansible-galaxy role install -r /var/lib/awx/projects/_1690__awx_error_test_dev/deployment/roles/requirements.yml --roles-path /var/lib/awx/projects/.__awx_cache/_1690__awx_error_test_dev/stage/requirements_roles  -vvv\\n",
            "_uses_shell": false,
            "argv": null,
            "chdir": "/var/lib/awx/projects/_1690__awx_error_test_dev",
            "creates": null,
            "executable": null,
            "removes": null,
            "stdin": null,
            "stdin_add_newline": true,
            "strip_empty_ends": true,
            "warn": false
        }
    },
    "item": "/var/lib/awx/projects/_1690__awx_error_test_dev/deployment/roles/requirements.yml",
    "msg": "",
    "rc": 0,
    "start": "2023-10-02 13:38:35.747570",
    "stderr": "",
    "stderr_lines": [],
    "stdout": "ansible-galaxy [core 2.12.5.post0]\\n  config file = None\\n  configured module search path = ['/home/runner/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']\\n  ansible python module location = /usr/local/lib/python3.8/site-packages/ansible\\n  ansible collection location = /home/runner/.ansible/collections:/usr/share/ansible/collections\\n  executable location = /usr/local/bin/ansible-galaxy\\n  python version = 3.8.13 (default, Jun 24 2022, 15:27:57) [GCC 8.5.0 20210514 (Red Hat 8.5.0-13)]\\n  jinja version = 2.11.3\\n  libyaml = True\\nNo config file found; using defaults\\nReading requirement file at '/var/lib/awx/projects/_1690__awx_error_test_dev/deployment/roles/requirements.yml'\\nfound role {'name': '030.ansible_eksctl', 'version': '', 'scm': None} in yaml file\\nOpened /home/runner/.ansible/galaxy_token\\nStarting galaxy role install process\\nProcessing role 030.ansible_eksctl \\n- downloading role 'ansible_eksctl', owned by 030\\n- downloading role from https://github.com/030/ansible-eksctl/archive/1.0.6.tar.gz\\n- extracting 030.ansible_eksctl to /var/lib/awx/projects/.__awx_cache/_1690__awx_error_test_dev/stage/requirements_roles/030.ansible_eksctl\\n- 030.ansible_eksctl (1.0.6) was installed successfully",
    "stdout_lines": [
        "ansible-galaxy [core 2.12.5.post0]",
        "  config file = None",
        "  configured module search path = ['/home/runner/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']",
        "  ansible python module location = /usr/local/lib/python3.8/site-packages/ansible",
        "  ansible collection location = /home/runner/.ansible/collections:/usr/share/ansible/collections",
        "  executable location = /usr/local/bin/ansible-galaxy",
        "  python version = 3.8.13 (default, Jun 24 2022, 15:27:57) [GCC 8.5.0 20210514 (Red Hat 8.5.0-13)]",
        "  jinja version = 2.11.3",
        "  libyaml = True",
        "No config file found; using defaults",
        "Reading requirement file at '/var/lib/awx/projects/_1690__awx_error_test_dev/deployment/roles/requirements.yml'",
        "found role {'name': '030.ansible_eksctl', 'version': '', 'scm': None} in yaml file",
        "Opened /home/runner/.ansible/galaxy_token",
        "Starting galaxy role install process",
        "Processing role 030.ansible_eksctl ",
        "- downloading role 'ansible_eksctl', owned by 030",
        "- downloading role from https://github.com/030/ansible-eksctl/archive/1.0.6.tar.gz",
        "- extracting 030.ansible_eksctl to /var/lib/awx/projects/.__awx_cache/_1690__awx_error_test_dev/stage/requirements_roles/030.ansible_eksctl",
        "- 030.ansible_eksctl (1.0.6) was installed successfully"
    ]
}

TASK [fetch galaxy collections from deployment/collections/requirements.yml] ***
task path: /tmp/awx_75799_t1qyfgog/project/project_update.yml:222
<127.0.0.1> ESTABLISH LOCAL CONNECTION FOR USER: root
<127.0.0.1> EXEC /bin/sh -c 'echo ~root && sleep 0'
<127.0.0.1> EXEC /bin/sh -c '( umask 77 && mkdir -p "` echo /root/.ansible/tmp `"&& mkdir "` echo /root/.ansible/tmp/ansible-tmp-1696253917.3840697-2555373-75419267155584 `" && echo ansible-tmp-1696253917.3840697-2555373-75419267155584="` echo /root/.ansible/tmp/ansible-tmp-1696253917.3840697-2555373-75419267155584 `" ) && sleep 0'
Using module file /usr/local/lib/python3.8/site-packages/ansible/modules/command.py
<127.0.0.1> PUT /home/runner/.ansible/tmp/ansible-local-2555276mi_um33f/tmpu5xa_h0x TO /root/.ansible/tmp/ansible-tmp-1696253917.3840697-2555373-75419267155584/AnsiballZ_command.py
<127.0.0.1> EXEC /bin/sh -c 'chmod u+x /root/.ansible/tmp/ansible-tmp-1696253917.3840697-2555373-75419267155584/ /root/.ansible/tmp/ansible-tmp-1696253917.3840697-2555373-75419267155584/AnsiballZ_command.py && sleep 0'
<127.0.0.1> EXEC /bin/sh -c 'ANSIBLE_COLLECTIONS_PATHS=/var/lib/awx/projects/.__awx_cache/_1690__awx_error_test_dev/stage/requirements_collections ANSIBLE_LOCAL_TEMP=/var/lib/awx/projects/.__awx_cache/_1690__awx_error_test_dev/stage/tmp ANSIBLE_FORCE_COLOR=false GIT_SSH_COMMAND='"'"'ssh -o StrictHostKeyChecking=no'"'"' /usr/bin/python3 /root/.ansible/tmp/ansible-tmp-1696253917.3840697-2555373-75419267155584/AnsiballZ_command.py && sleep 0'
<127.0.0.1> EXEC /bin/sh -c 'rm -f -r /root/.ansible/tmp/ansible-tmp-1696253917.3840697-2555373-75419267155584/ > /dev/null 2>&1 && sleep 0'
failed: [localhost] (item=/var/lib/awx/projects/_1690__awx_error_test_dev/deployment/collections/requirements.yml) => {
    "ansible_loop_var": "item",
    "changed": false,
    "cmd": [
        "ansible-galaxy",
        "collection",
        "install",
        "-r",
        "/var/lib/awx/projects/_1690__awx_error_test_dev/deployment/collections/requirements.yml",
        "--collections-path",
        "/var/lib/awx/projects/.__awx_cache/_1690__awx_error_test_dev/stage/requirements_collections",
        "-vvv"
    ],
    "delta": "0:00:04.629590",
    "end": "2023-10-02 13:38:42.207146",
    "invocation": {
        "module_args": {
            "_raw_params": "ansible-galaxy collection install -r /var/lib/awx/projects/_1690__awx_error_test_dev/deployment/collections/requirements.yml --collections-path /var/lib/awx/projects/.__awx_cache/_1690__awx_error_test_dev/stage/requirements_collections  -vvv\\n",
            "_uses_shell": false,
            "argv": null,
            "chdir": "/var/lib/awx/projects/_1690__awx_error_test_dev",
            "creates": null,
            "executable": null,
            "removes": null,
            "stdin": null,
            "stdin_add_newline": true,
            "strip_empty_ends": true,
            "warn": false
        }
    },
    "item": "/var/lib/awx/projects/_1690__awx_error_test_dev/deployment/collections/requirements.yml",
    "msg": "non-zero return code",
    "rc": 1,
    "start": "2023-10-02 13:38:37.577556",
    "stderr": "[WARNING]: Skipping Galaxy server https://galaxy.ansible.com/api/. Got an\\nunexpected error when getting available versions of collection\\ncommunity.general: Unknown error when attempting to call Galaxy at\\n'https://galaxy.ansible.com/api/v3/collections/community/general/': <urlopen\\nerror [Errno 104] Connection reset by peer>\\nERROR! Unknown error when attempting to call Galaxy at 'https://galaxy.ansible.com/api/v3/collections/community/general/': <urlopen error [Errno 104] Connection reset by peer>",
    "stderr_lines": [
        "[WARNING]: Skipping Galaxy server https://galaxy.ansible.com/api/. Got an",
        "unexpected error when getting available versions of collection",
        "community.general: Unknown error when attempting to call Galaxy at",
        "'https://galaxy.ansible.com/api/v3/collections/community/general/': <urlopen",
        "error [Errno 104] Connection reset by peer>",
        "ERROR! Unknown error when attempting to call Galaxy at 'https://galaxy.ansible.com/api/v3/collections/community/general/': <urlopen error [Errno 104] Connection reset by peer>"
    ],
    "stdout": "ansible-galaxy [core 2.12.5.post0]\\n  config file = None\\n  configured module search path = ['/home/runner/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']\\n  ansible python module location = /usr/local/lib/python3.8/site-packages/ansible\\n  ansible collection location = /var/lib/awx/projects/.__awx_cache/_1690__awx_error_test_dev/stage/requirements_collections\\n  executable location = /usr/local/bin/ansible-galaxy\\n  python version = 3.8.13 (default, Jun 24 2022, 15:27:57) [GCC 8.5.0 20210514 (Red Hat 8.5.0-13)]\\n  jinja version = 2.11.3\\n  libyaml = True\\nNo config file found; using defaults\\nReading requirement file at '/var/lib/awx/projects/_1690__awx_error_test_dev/deployment/collections/requirements.yml'\\nStarting galaxy collection install process\\nProcess install dependency map\\nOpened /home/runner/.ansible/galaxy_token",
    "stdout_lines": [
        "ansible-galaxy [core 2.12.5.post0]",
        "  config file = None",
        "  configured module search path = ['/home/runner/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']",
        "  ansible python module location = /usr/local/lib/python3.8/site-packages/ansible",
        "  ansible collection location = /var/lib/awx/projects/.__awx_cache/_1690__awx_error_test_dev/stage/requirements_collections",
        "  executable location = /usr/local/bin/ansible-galaxy",
        "  python version = 3.8.13 (default, Jun 24 2022, 15:27:57) [GCC 8.5.0 20210514 (Red Hat 8.5.0-13)]",
        "  jinja version = 2.11.3",
        "  libyaml = True",
        "No config file found; using defaults",
        "Reading requirement file at '/var/lib/awx/projects/_1690__awx_error_test_dev/deployment/collections/requirements.yml'",
        "Starting galaxy collection install process",
        "Process install dependency map",
        "Opened /home/runner/.ansible/galaxy_token"
    ]
}

PLAY RECAP *********************************************************************
localhost                  : ok=5    changed=3    unreachable=0    failed=1    skipped=1    rescued=0    ignored=0

lucasvivian · 2023-10-02T14:59:29Z

co-worker have update from ansible 2.12 to 2.13 inside EE and seems to have fixed here.

al-lac · 2023-10-02T15:12:54Z

@MatthieuLeMee yeah i kind of suspected that, i saw that the latest image was updated a few minutes ago. But i don't have time right now to try it out. I hope there will be a new release / tag soon that updates ansible in there.

@lucasvivian great to hear!

al-lac · 2023-10-03T11:47:37Z

Updated the Control Plane Execution Environment to use the quay.io/ansible/awx-ee:latest image and now I can sync projects again.

Still waiting for a new tag though so i can pin it to a version that fits to the AWX release.

sudheer-palagiri · 2023-10-03T17:05:09Z

ansible --version

[DEPRECATION WARNING]: Ansible will require Python 3.8 or newer on the controller starting with Ansible 2.12. Current version: 3.6.8 (default, Jun 20 2023, 11:53:23) [GCC 4.8.5 20150623 (Red Hat 4.8.5-44)].
This feature will be removed from ansible-core in version 2.12. Deprecation warnings can be disabled by setting deprecation_warnings=False in ansible.cfg.
/usr/local/lib/python3.6/site-packages/ansible/parsing/vault/init.py:44: CryptographyDeprecationWarning: Python 3.6 is no longer supported by the Python core team. Therefore, support for it is deprecated incryptography. The next release of cryptography will remove support for Python 3.6.
from cryptography.exceptions import InvalidSignature
ansible [core 2.11.12]
config file = None
configured module search path = ['/root/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
ansible python module location = /usr/local/lib/python3.6/site-packages/ansible
ansible collection location = /root/.ansible/collections:/usr/share/ansible/collections
executable location = /usr/local/bin/ansible
python version = 3.6.8 (default, Jun 20 2023, 11:53:23) [GCC 4.8.5 20150623 (Red Hat 4.8.5-44)]
jinja version = 3.0.3
libyaml = True

/usr/local/bin/ansible-galaxy -v collection install community.general

[DEPRECATION WARNING]: Ansible will require Python 3.8 or newer on the controller starting with Ansible 2.12. Current version: 3.6.8 (default, Jun 20 2023, 11:53:23) [GCC 4.8.5 20150623 (Red Hat 4.8.5-44)].
This feature will be removed from ansible-core in version 2.12. Deprecation warnings can be disabled by setting deprecation_warnings=False in ansible.cfg.
/usr/local/lib/python3.6/site-packages/ansible/parsing/vault/init.py:44: CryptographyDeprecationWarning: Python 3.6 is no longer supported by the Python core team. Therefore, support for it is deprecated incryptography. The next release of cryptography will remove support for Python 3.6.
from cryptography.exceptions import InvalidSignature
[DEPRECATION WARNING]: Setting verbosity before the arg sub command is deprecated, set the verbosity after the sub command. This feature will be removed from ansible-core in version 2.13. Deprecation warnings
can be disabled by setting deprecation_warnings=False in ansible.cfg.
No config file found; using defaults
Starting galaxy collection install process
Process install dependency map
[WARNING]: Skipping Galaxy server https://galaxy.ansible.com/api/. Got an unexpected error when getting available versions of collection community.general:
'/api/v3/plugin/ansible/content/published/collections/index/community/general/versions/'
ERROR! Unexpected Exception, this is probably a bug: '/api/v3/plugin/ansible/content/published/collections/index/community/general/versions/'
to see the full traceback, use -vvv

jefbur · 2023-10-03T17:30:31Z

Same issues here on ansible 2.11.6:

ansible-playbooks$ ansible-galaxy collection install -r ./requirements.yml
Starting galaxy collection install process
Process install dependency map
[WARNING]: Skipping Galaxy server https://galaxy.ansible.com/api/. Got an unexpected error when getting available versions of collection ansible.netcommon:
'/api/v3/plugin/ansible/content/published/collections/index/ansible/netcommon/versions/'
ERROR! Unexpected Exception, this is probably a bug: '/api/v3/plugin/ansible/content/published/collections/index/ansible/netcommon/versions/'
to see the full traceback, use -vvv

Seems to be intermittent, bad server behind LoadBalancer?

aimers1975 · 2023-10-03T19:33:39Z

I know its been posted already but same issue:

ansible [core 2.11.12]
config file = /root/.ansible.cfg
configured module search path = ['/usr/lib/python3.10/site-packages/ansible_collections/ansible/posix/plugins/modules', '/usr/lib/python3.10/site-packages/ansible/modules']
ansible python module location = /usr/lib/python3.10/site-packages/ansible
ansible collection location = /usr/lib/python3.10/site-packages/ansible_collections:/root/.ansible/collections
executable location = /usr/bin/ansible
python version = 3.10.12 (main, Jun 30 2023, 17:58:40) [GCC]
jinja version = 3.1.2
libyaml = True

Frewx · 2023-10-03T19:53:39Z

Updated the Control Plane Execution Environment to use the quay.io/ansible/awx-ee:latest image and now I can sync projects again.

Still waiting for a new tag though so i can pin it to a version that fits to the AWX release.

I was using the image as awx-ee:latest, still the problem persists for me. I even tried to pull the image every time when a job is triggered, still no luck.

lachryrs · 2023-10-03T20:00:48Z

Updated the Control Plane Execution Environment to use the quay.io/ansible/awx-ee:latest image and now I can sync projects again.

Still waiting for a new tag though so i can pin it to a version that fits to the AWX release.

Worked for myself; ensure that you rebuild your kube's after changing your Control Plane EE to point at latest. Thanks.

Frewx · 2023-10-03T20:19:24Z

Okay after some investigating, in AWX operator version 2.5.0 (AWX version 22.6.0) there was a pod "awx-task-xxx" in a running state, this pod is in charge of the task execution. I checked this pods image an it says awx-ee:latest but I checked sha of the image and it was an older image. So I deleted this pod and it came back with the new awx-ee:latest image with sha256:19ade0cacf21. After the new awx-task pod creation, the problem is solved. (Because it used the new image).
Still though, I have an another AWX operator with version 1.1.2 (AWX version 21.10.1), which I've been using a long time. In this operator, awx-task pod is created whenever a job is triggered, so I thought that this is still the same, but apparently I was mistaken. The pod was created and was in a ready state, waiting for a job to trigger. And because of this I needed to delete the pod for new image to take place.
So to be clear of the operation steps that I took:

Delete the awx-task-xxx pod
Wait for it to be ready with the new image
Try a job to see if it's successful.

NOTE: Because of the operator awx-task-xxx pod, the solution @al-lac mentioned didn't work because it never pulled the newer image. (Even though I configured all the execution environments to always pull)

MatthieuLeMee · 2023-10-04T08:26:05Z

It has definately to do with the awx-ee version.
I customize this image and was based on an old awx-ee (containing ansible-core 2.12.5). I updated the control plane EE to use a newer awx-ee version (containing ansible-core 2.15) and it's fixed.

FYI, I'm not using the awx-ee:latest anymore after there has been wrong commits in recent weeks and blocked our production platform. By default, a local AWX installation is dependent on a latest tag which means its dependent on an online resource (quay.io) and will get the latest commits from the community which is wrong if you need stability.

Customizing the EE image does work and needs this deployement config :

...
apiVersion: awx.ansible.com/v1beta1
kind: AWX
metadata:
  name: awx
spec:
  replicas: 1

  image: pic.ima.eu/docker4ima/ima-awx
  image_version: $AWX_IMAGE_VERSION
  ee_images:
    - name: ima-awx-ee
      image: $AWX_EE_IMAGE_VERSION
  control_plane_ee_image: $AWX_EE_IMAGE_VERSION
  image_pull_secret: ima-registry-credential
  ee_pull_credentials_secret: awx-ee-pull-credentials
  ...

I will mark this issue as resolved.

vrelk-net · 2023-10-05T14:03:44Z

I just changed the "Pull" setting for the "Control Plane Execution Environment" from "Missing" to "Always", after which the issue went away.

MTWiley · 2023-10-05T17:38:15Z

https://forum.ansible.com/t/unable-to-install-collection-from-new-galaxy/1238/4?u=alex

At least for me ansible-version returns:

ansible [core 2.15.4] and I'm seeing the same issue.

mrtaxi · 2023-10-09T10:00:27Z

The Ansible team released a new Galaxy server.
You'd need to use Ansible >=2.13.9 to download content from it.

In case you are using an older version, there is a new url to use to get packages : https://old-galaxy.ansible.com/

Here is what to put in your ansible.cfg file :

[galaxy]
server = https://old-galaxy.ansible.com/

https://stackoverflow.com/a/77238083/19878558

kulajk · 2024-01-10T12:55:43Z

upgrading was helpful, it works now

ansible [core 2.12.10]
config file = /Users/kkulaj/repos/deployment/ansible.cfg
configured module search path = ['/Users/kkulaj/repos/deployment/library', '/Users/kkulaj/repos/deployment/plugins/modules']
ansible python module location = /Users/kkulaj/repos/deployment/.venv/lib/python3.8/site-packages/ansible
ansible collection location = /Users/kkulaj/repos/deployment/collections
executable location = /Users/kkulaj/repos/deployment/.venv/bin/ansible
python version = 3.8.18 (default, Jan 9 2024, 10:47:24) [Clang 15.0.0 (clang-1500.0.40.1)]
jinja version = 3.0.1
libyaml = True

According to [Unable to install some galaxy collections · Issue #14495 · ansible/awx · GitHub](ansible/awx#14495 (comment))

github-actions bot added component:awx_collection issues related to the collection for controlling AWX needs_triage type:bug community labels Oct 2, 2023

MatthieuLeMee closed this as completed Oct 4, 2023

ppmathis mentioned this issue Oct 4, 2023

Improved Jinja plugin caching breaks loading multiple custom filter plugins with same name ansible/ansible#81897

Closed

1 task

jaxesn mentioned this issue Oct 11, 2023

fixes cilium and image-builder builds aws/eks-anywhere-build-tooling#2533

Merged

This was referenced Oct 14, 2023

fix: make ansible-galaxy work again with community.general projectpotos/ansible-role-potos_basics#38

Merged

fix: make ansible-galaxy work again with community.general projectpotos/potos-iso-builder#50

Merged

mathias-muench pushed a commit to mathias-muench/ansible-builder that referenced this issue Mar 5, 2024

Workaround bug in galaxy

6f62f0b

According to [Unable to install some galaxy collections · Issue #14495 · ansible/awx · GitHub](ansible/awx#14495 (comment))

fermosan mentioned this issue May 10, 2024

Unable to install some galaxy collections (again) #15178

Closed

11 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Unable to install some galaxy collections #14495

Unable to install some galaxy collections #14495

MatthieuLeMee commented Oct 2, 2023 •

edited

gbjgro3 commented Oct 2, 2023

JKetelaar commented Oct 2, 2023

al-lac commented Oct 2, 2023

Frewx commented Oct 2, 2023

lucasvivian commented Oct 2, 2023 •

edited

al-lac commented Oct 2, 2023

MatthieuLeMee commented Oct 2, 2023 •

edited

lucasvivian commented Oct 2, 2023

al-lac commented Oct 2, 2023

al-lac commented Oct 3, 2023

sudheer-palagiri commented Oct 3, 2023

jefbur commented Oct 3, 2023 •

edited

aimers1975 commented Oct 3, 2023

Frewx commented Oct 3, 2023 •

edited

lachryrs commented Oct 3, 2023

Frewx commented Oct 3, 2023

MatthieuLeMee commented Oct 4, 2023

vrelk-net commented Oct 5, 2023

MTWiley commented Oct 5, 2023

mrtaxi commented Oct 9, 2023 •

edited

kulajk commented Jan 10, 2024 •

edited

Unable to install some galaxy collections #14495

Unable to install some galaxy collections #14495

Comments

MatthieuLeMee commented Oct 2, 2023 • edited

Please confirm the following

Bug Summary

AWX version

Select the relevant components

Installation method

Modifications

Ansible version

Operating system

Web browser

Steps to reproduce

Expected results

Actual results

Additional information

gbjgro3 commented Oct 2, 2023

JKetelaar commented Oct 2, 2023

al-lac commented Oct 2, 2023

Frewx commented Oct 2, 2023

lucasvivian commented Oct 2, 2023 • edited

al-lac commented Oct 2, 2023

MatthieuLeMee commented Oct 2, 2023 • edited

lucasvivian commented Oct 2, 2023

al-lac commented Oct 2, 2023

al-lac commented Oct 3, 2023

sudheer-palagiri commented Oct 3, 2023

ansible --version

/usr/local/bin/ansible-galaxy -v collection install community.general

jefbur commented Oct 3, 2023 • edited

aimers1975 commented Oct 3, 2023

Frewx commented Oct 3, 2023 • edited

lachryrs commented Oct 3, 2023

Frewx commented Oct 3, 2023

MatthieuLeMee commented Oct 4, 2023

vrelk-net commented Oct 5, 2023

MTWiley commented Oct 5, 2023

mrtaxi commented Oct 9, 2023 • edited

kulajk commented Jan 10, 2024 • edited

MatthieuLeMee commented Oct 2, 2023 •

edited

lucasvivian commented Oct 2, 2023 •

edited

MatthieuLeMee commented Oct 2, 2023 •

edited

jefbur commented Oct 3, 2023 •

edited

Frewx commented Oct 3, 2023 •

edited

mrtaxi commented Oct 9, 2023 •

edited

kulajk commented Jan 10, 2024 •

edited