Skip to content

delegate_to leads to SSL verification error #1611

Description

@sscheib
SUMMARY

Using delegate_to in a task with any theforeman.foreman.* module leads to SSL connection issues:

"msg": "Failed to connect to Foreman server: SSLError: HTTPSConnectionPool(host='host.example.com', port=443): Max retries exceeded with url: /api/status (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self signed certificate in certificate chain (_ssl.c:1129)')))"
}

Verification of the SSL connection using openssl s_client -connect -showcerts host.example.com </dev/null validates perfectly:

[..]
---
SSL handshake has read 8465 bytes and written 438 bytes
Verification: OK
---
New, TLSv1.3, Cipher is TLS_AES_256_GCM_SHA384
Server public key is 4096 bit
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
Early data was not sent
Verify return code: 0 (ok)
---
[..]
ISSUE TYPE
  • Bug Report
ANSIBLE VERSION
ansible [core 2.14.5]
  config file = /home/user/.ansible.cfg
  configured module search path = ['/home/user/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /usr/lib/python3.9/site-packages/ansible
  ansible collection location = /home/user/.ansible/collections:/usr/share/ansible/collections
  executable location = /usr/bin/ansible
  python version = 3.9.13 (main, Nov  9 2022, 13:16:24) [GCC 8.5.0 20210514 (Red Hat 8.5.0-15)] (/usr/bin/python3.9)
  jinja version = 3.1.2
  libyaml = True
COLLECTION VERSION
# /home/user/.ansible/collections/ansible_collections
Collection                  Version
--------------------------- -------
theforeman.foreman          3.10.0
KATELLO/FOREMAN VERSION
# rpm -q tfm-rubygem-katello foreman
package tfm-rubygem-katello is not installed
foreman-3.3.0.21-2.el8sat.noarch
STEPS TO REPRODUCE
---
- name: 'Configure OpenSCAP'
  hosts: 'all'
  gather_facts: false
  vars:
    scap_content_download_path: '/tmp'
    scap_content_files:
      - name: 'DISA STIG RHEL 8 (Version 1, Release 9)'
        url: 'https://dl.dod.cyber.mil/wp-content/uploads/stigs/zip/U_RHEL_8_V1R9_STIG_SCAP_1-2_Benchmark.zip'

      - name: 'DISA STIG RHEL 7 (Version 1, Release 11)'
        url: 'https://dl.dod.cyber.mil/wp-content/uploads/stigs/zip/U_RHEL_7_V3R11_STIG_SCAP_1-2_Benchmark.zip'

  tasks:
    - name: 'Download and unpack OpenSCAP content'
      ansible.builtin.unarchive:
        src: '{{ item.url }}'
        dest: '{{ scap_content_download_path }}'
        mode: 0600
        remote_src: true
      loop: '{{ scap_content_files }}'
      delegate_to: 'localhost'

    - name: 'Create OpenSCAP content'
      theforeman.foreman.scap_content:
        username: '{{ satellite_username }}'
        password: '{{ satellite_password }}'
        server_url: '{{ satellite_server_url }}'
        validate_certs: true
        scap_file: "{{ scap_content_download_path ~ '/' ~ item.url | basename | regex_replace('.zip$', '.xml') }}"
        state: 'present'
        title: '{{ item.name }}'
      delegate_to: 'localhost'
      loop: '{{ scap_content_files }}

As theforeman.foreman.scap_content needs scap_file, I need to delegate to localhost as the files only will be temporarily downloaded to /tmp and afterwards removed of the control node.
Note: I am not running in an execution environment, still using ansible-playbook

EXPECTED RESULTS

SCAP contents are created

ACTUAL RESULTS

SSL verification fails. This is only observed when using delegate_to. As seen above, the SSL connection validated successfully, though.

$ ansible-playbook -i inventory openscap.yml --vault-pass-file .vault.pass
PLAY [Configure OpenSCAP] ***************************************************************************************************************************************************************

TASK [Download and unpack OpenSCAP content] *********************************************************************************************************************************************
ok: [host.example.com -> localhost] => (item={'name': 'DISA STIG RHEL 8 (Version 1, Release 9)', 'url': 'https://dl.dod.cyber.mil/wp-content/uploads/stigs/zip/U_RHEL_8_V1R9_STIG_SCAP_1-2_Benchmark.zip'})
ok: [host.example.com -> localhost] => (item={'name': 'DISA STIG RHEL 7 (Version 1, Release 11)', 'url': 'https://dl.dod.cyber.mil/wp-content/uploads/stigs/zip/U_RHEL_7_V3R11_STIG_SCAP_1-2_Benchmark.zip'})

TASK [Create OpenSCAP content] **********************************************************************************************************************************************************
failed: [host.example.com -> localhost] (item={'name': 'DISA STIG RHEL 8 (Version 1, Release 9)', 'url': 'https://dl.dod.cyber.mil/wp-content/uploads/stigs/zip/U_RHEL_8_V1R9_STIG_SCAP_1-2_Benchmark.zip'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "DISA STIG RHEL 8 (Version 1, Release 9)", "url": "https://dl.dod.cyber.mil/wp-content/uploads/stigs/zip/U_RHEL_8_V1R9_STIG_SCAP_1-2_Benchmark.zip"}, "msg": "Failed to connect to Foreman server: SSLError: HTTPSConnectionPool(host='host.example.com', port=443): Max retries exceeded with url: /api/status (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self signed certificate in certificate chain (_ssl.c:1129)')))"}
failed: [host.example.com -> localhost] (item={'name': 'DISA STIG RHEL 7 (Version 1, Release 11)', 'url': 'https://dl.dod.cyber.mil/wp-content/uploads/stigs/zip/U_RHEL_7_V3R11_STIG_SCAP_1-2_Benchmark.zip'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "DISA STIG RHEL 7 (Version 1, Release 11)", "url": "https://dl.dod.cyber.mil/wp-content/uploads/stigs/zip/U_RHEL_7_V3R11_STIG_SCAP_1-2_Benchmark.zip"}, "msg": "Failed to connect to Foreman server: SSLError: HTTPSConnectionPool(host='host.example.com', port=443): Max retries exceeded with url: /api/status (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self signed certificate in certificate chain (_ssl.c:1129)')))"}

PLAY RECAP ******************************************************************************************************************************************************************************
host.example.com : ok=1    changed=0    unreachable=0    failed=1    skipped=0    rescued=0    ignored=0   

Activity

  1. evgeni commented on May 26, 2023

    @evgeni
    Member

    I'm using hosts: localhost vs delegate_to: localhost here, and that works. AFAIK those should be identical?

    As it's working for me, couple of questions

    1. What is localhost in this context?
    2. Which Python is used when you execute against localhost?
    3. If that's not the system Python, it might use the certs bundled in certifi which then ignores your system-wide installed certs that openssl uses?
  2. sscheib commented on May 26, 2023

    @sscheib
    ContributorAuthor

    Thanks for looking into this!
    I use 'all' for hosts as I provide variables via host_vars for the specific Satellite instances (configuring multiple)

    What is localhost in this context?

    The local controller node, which is a plain RHEL 8.7. Not running in an execution environment, so it is really referring to that machine.

    Which Python is used when you execute against localhost?

    It's the same as with hosts: 'all':

    "ansible_python_interpreter": "/usr/bin/python3.9"
    

    If that's not the system Python, it might use the certs bundled in certifi which then ignores your system-wide installed certs that openssl uses?

    It's the same in both cases

  3. sscheib commented on Feb 4, 2024

    @sscheib
    ContributorAuthor

    I cannot reproduce this issue anymore with the latest release. Either it was an issue on my end or it was unknowingly fixed with later version
    Anyway, closing it as complete.

  4. sscheib commented on Jun 6, 2024

    @sscheib
    ContributorAuthor

    I know, this is an old issue.

    For anybody stumbling on this issue, I finally managed to reproduce it.
    The issue is present in a Python virtual environment when custom certificates are used.

    This can be fixed with: https://stackoverflow.com/questions/34931378/certificate-verification-when-using-virtual-environments

  5. evgeni commented on Jun 7, 2024

    @evgeni
    Member

    Thanks for tracking this down!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Fields

    Red Hat Jira

    None yet

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions