--offline (or offline: true) does not do, what is said in the docs

[dbk-rabel]

Issue Type

• Bug report

Ansible and Ansible Lint details

ansible-lint --version
ansible-lint 6.8.7 using ansible 2.14.0

• ansible-lint installation method: pip

Desired Behavior

Behaviour of parameter --offline (oder equivalently option offline: true) is coherant with documentation:

--offline Disable installation of requirements.yml

Actual Behavior (Bug report only)

What --offline really does, are apparently these two things:

1. Do not connect to Github. (That's very useful btw.)
2. Pass --offline to 'ansible-galaxy collection install -r requirements.yml', which seems to have no effect at all.

1:42:12   WARNING  Retrying execution failure 250 of: ansible-galaxy collection install -v --offline -r requirements.yml -p /opt/app-root/src/.cache/ansible-compat/68c7a5/collections
11:42:12   ERROR    Using /<censored>/ansible.cfg as config file
11:42:12   to see the full traceback, use -vvv
11:42:12   
11:42:12   ERROR    ERROR! Unexpected Exception, this is probably a bug: [Errno 2] No such file or directory: b'https://<censored>.tar.gz
11:42:12   
11:42:12   Got 250 exit code while running: ansible-galaxy collection install -v --offline -r requirements.yml -p /opt/app-root/src/.cache/ansible-compat/68c7a5/collections

Proposed solution

1. Add the "do not connect to github" behaviour to the docs.
2. Fix --offline parameter, so that there is not ansible-galaxy collection install instead of just forwarding the --ofline parameter.

Thanks.
David

[MarkusTeufelberger]

I have a similar issue (also using ansible-core 2.14.0 and the newer ansible-lint 6.9.0), I have downloaded collections to their proper place already AND set the collections_paths in ansible-cfg correctly, there is absolutely no need for ansible-lint to run ansible-galaxy collection install, with an --offline flag or without. Just don't run galaxy at all if --offline is specified and don't mess with my collections path settings please. Now I need to waste traffic and time again for a while as well as deal with flaky availability of galaxy by turning offline mode off until this is fixed... :-(

[MarkusTeufelberger]

Likely related to #2683 btw

[bluikko]

Same here. I wonder why is ansible-galaxy called to download collections in the first place when all the collections are already installed in the venv?

$ ansible-galaxy collection list

# /usr/share/ansible/collections/ansible_collections
Collection        Version
----------------- -------
ansible.netcommon 4.1.0
ansible.posix     1.4.0
ansible.utils     2.7.0
community.crypto  2.9.0
community.docker  3.2.2
community.general 6.0.1
containers.podman 1.10.1

But still in ansible-lint tries to install:

INFO     Running ansible-galaxy collection install -v --offline -r collections/requirements.yml -p /x/.cache/ansible-compat/y/collections
...
ERROR! Failed to resolve the requested dependencies map. Could not satisfy the following requirements:
* ansible.netcommon:* (direct request)

As listed also in the above referenced issue.

[ssbarnea]

The --offline mode introduced by galaxy is of very different nature than the one used by ansible-lint, so we cannot really make use of it. I am working on applying the requires reverts and restore its old functionality.

Those using ansible-lint in offline mode will have to find a way to install the requirements themselves.

[rgl]

I'm afraid this is still not working here. I'm running ansible (and ansible-lint) in a container and installing the collections as:

ansible-galaxy collection install \
    -r requirements.yml \
    -p /usr/share/ansible/collections
ansible-galaxy role install \
    -r requirements.yml \
    -p /usr/share/ansible/roles

Then, using a wrapper, execute ansible-lint as:

./ansible-lint.sh --offline --parseable development.yml

But it still fails with:

+ cat /etc/os-release
PRETTY_NAME="Debian GNU/Linux 11 (bullseye)"
NAME="Debian GNU/Linux"
VERSION_ID="11"
VERSION="11 (bullseye)"
VERSION_CODENAME=bullseye
ID=debian
HOME_URL="https://www.debian.org/"
SUPPORT_URL="https://www.debian.org/support"
BUG_REPORT_URL="https://bugs.debian.org/"
+ ansible --version
ansible [core 2.14.0]
  config file = None
  configured module search path = ['/root/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /usr/local/lib/python3.9/dist-packages/ansible
  ansible collection location = /root/.ansible/collections:/usr/share/ansible/collections
  executable location = /usr/local/bin/ansible
  python version = 3.9.2 (default, Feb 28 2021, 17:03:44) [GCC 10.2.1 20210110] (/usr/bin/python3)
  jinja version = 3.1.2
  libyaml = True
+ python3 -m pip list
Package           Version
----------------- ---------
ansible-compat    2.2.6
ansible-core      2.14.0
ansible-lint      6.9.1
argcomplete       1.8.1
astroid           2.5.1
attrs             22.1.0
black             22.10.0
bracex            2.3.post1
click             8.1.3
commonmark        0.9.1
cryptography      3.3.2
filelock          3.8.0
isort             5.6.4
Jinja2            3.1.2
jsonschema        4.17.3
lazy-object-proxy 0.0.0
logilab-common    1.8.1
MarkupSafe        2.1.1
mccabe            0.6.1
mypy-extensions   0.4.3
netaddr           0.7.19
packaging         21.3
pathspec          0.10.2
pip               20.3.4
platformdirs      2.5.4
Pygments          2.13.0
pylint            2.7.2
pyOpenSSL         20.0.1
pyparsing         3.0.9
pyrsistent        0.19.2
PyYAML            6.0
resolvelib        0.8.1
rich              12.6.0
ruamel.yaml       0.17.21
ruamel.yaml.clib  0.2.7
setuptools        52.0.0
six               1.16.0
subprocess-tee    0.4.0
toml              0.10.1
tomli             2.0.1
typing-extensions 4.4.0
wcmatch           8.4.1
wheel             0.34.2
wrapt             1.12.1
yamllint          1.28.0
+ ansible-galaxy collection list

# /usr/share/ansible/collections/ansible_collections
Collection        Version
----------------- -------
ansible.posix     1.4.0  
ansible.utils     2.8.0  
community.docker  3.2.2  
community.general 6.0.1  
kubernetes.core   2.3.2  
+ exec docker run --rm --net=host -v /home/vagrant/Projects/my-ubuntu-ansible-playbooks:/project:ro -e GITHUB_ACTIONS -e GITHUB_WORKFLOW ansible-my-ubuntu-ansible-playbooks ansible-lint --offline --parseable development.yml
WARNING  Retrying execution failure 1 of: ansible-galaxy collection install -v --offline -r requirements.yml -p /root/.cache/ansible-compat/244210/collections
ERROR    Using /project/ansible.cfg as config file
Starting galaxy collection install process
Process install dependency map

ERROR    [WARNING]: The specified collections path '/root/.cache/ansible-
compat/244210/collections' is not part of the configured Ansible collections
paths '/root/.ansible/collections:/usr/share/ansible/collections'. The
installed collection will not be picked up in an Ansible run, unless within a
playbook-adjacent collections directory.
ERROR! Failed to resolve the requested dependencies map. Could not satisfy the following requirements:
* ansible.posix:1.4.0 (direct request)

Got 1 exit code while running: ansible-galaxy collection install -v --offline -r requirements.yml -p /root/.cache/ansible-compat/244210/collections

What am I missing?

[erikgb]

Why do we even have an "offline" mode? Why aren't the schemas delivered with the ansible-lint CLI? Requiring access to raw.githubusercontent.com by default seems very dirty IMO...

DEBUG    Checking for updated schemas...
DEBUG    Skipped updating schema that is part of the ansible-lint repository: https://raw.githubusercontent.com/ansible/ansible-lint/main/src/ansiblelint/schemas/ansible-lint-config.json
DEBUG    Refreshing ansible-navigator-config schema ...
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/urllib/request.py", line 1348, in do_open
    h.request(req.get_method(), req.selector, req.data, headers,
  File "/usr/local/lib/python3.10/http/client.py", line 1282, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "/usr/local/lib/python3.10/http/client.py", line 1328, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "/usr/local/lib/python3.10/http/client.py", line 1277, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/usr/local/lib/python3.10/http/client.py", line 1037, in _send_output
    self.send(msg)
  File "/usr/local/lib/python3.10/http/client.py", line 975, in send
    self.connect()
  File "/usr/local/lib/python3.10/http/client.py", line 1454, in connect
    self.sock = self._context.wrap_socket(self.sock,
  File "/usr/local/lib/python3.10/ssl.py", line 513, in wrap_socket
    return self.sslsocket_class._create(
  File "/usr/local/lib/python3.10/ssl.py", line 1071, in _create
    self.do_handshake()
  File "/usr/local/lib/python3.10/ssl.py", line 13[42](https://gitlab.statnett.no/iaut/crossplane-rke/-/jobs/4754225#L42), in do_handshake
    self._sslobj.do_handshake()
ConnectionResetError: [Errno 104] Connection reset by peer

[erikgb]

And when setting the --offline flag, the logs are just not making any sense (both ignored and not ignored): 🤔

WARNING  Offline mode ignored because `ansible-galaxy role install` command does not support it.
INFO     Running ansible-galaxy role install -vr requirements.yml --roles-path /root/.cache/ansible-compat/3d3a18/roles
INFO     Running ansible-galaxy collection install -v --offline -r requirements.yml -p /root/.cache/ansible-compat/3d3a18/collections

[bluikko]

(both ignored and not ignored)

I do not see an inconsistency in those messages. It is ignored for the role command as stated, not for the collection command.

But IMO the major problem here is this: ansible-lint does not detect already installed collections and always tries to install roles - regardless of --offline or not.

[apatard]

Why do we even have an "offline" mode? Why aren't the schemas delivered with the ansible-lint CLI? Requiring access to raw.githubusercontent.com by default seems very dirty IMO...

fwiw, there were some concerns raised about this before : #2703 (comment)

DEBUG    Checking for updated schemas...
DEBUG    Skipped updating schema that is part of the ansible-lint repository: https://raw.githubusercontent.com/ansible/ansible-lint/main/src/ansiblelint/schemas/ansible-lint-config.json
DEBUG    Refreshing ansible-navigator-config schema ...

Sounds weird as it not supposed to run refresh in offline mode. It probably deserves its own bug report imho

[ssbarnea]

I will also try to answer @erikgb questions:

• Our schemas do update quite often and we observed a notable number of bug reports caused by use of outdated schemas.
• We do include all the schemas with the linter but we also have a feature that can update these schemas at runtime, one that is disabled when offline mode is used.
• If network is down, the code is expected to use the cached/packaged schemas. You log looks like a genuine bug for an exception that we did not catch.

@apatard is right, do not use existing tickets to file new bugs. That is why I am going to close and lock this issue because the original bug was fixed in 6.9.1 and we should not spam others with reports from various umentioned versions.

PS. A hotfix for the offline refresh logic will be make today, I already have it in local testing.

[ssbarnea]

@apatard @erikgb @bluikko #2785 should address the reported issues with schema refresh.