Hosts added by `add_host` module don't respect `ansible_host_key_checking` variable

[philfry]

When adding a managed node to the current inventory using add_host the variable ansible_host_key_checking is not honoured. Let's say we have a vanilla machine ansible-test-100 and this playbook:

---
- hosts: localhost
  gather_facts: no
  tasks:
    - add_host:
        name: ansible-test-100
        ansible_host: 192.168.122.100
        ansible_ssh_private_key_file: /tmp/ansibletest-leases/100/id_ed25519

- hosts: ansible-test-100
  remote_user: ansible
  become: no
  tasks:
    - copy: content="Hello world" dest=/tmp/blah
    - file: path=/tmp/blah state=absent

As ssh doesn't know about the ssh host key yet, the playbook will prompt for a fingerprint confirmation:

$ ANSIBLE_STRATEGY=linear ansible-playbook foo.yml 

PLAY [localhost] ******************************************************************************************************

TASK [add_host] *******************************************************************************************************
changed: [localhost]

PLAY [ansible-test-100] ***********************************************************************************************
The authenticity of host '192.168.122.100 (192.168.122.100)' can't be established.
ED25519 key fingerprint is SHA256:JzKOGkgn5xX67/IAksbd/S/cof9Vzn1FD25a+pQfgXQ.
This key is not known by any other names.
Are you sure you want to continue connecting (yes/no/[fingerprint])?

To circumvent this, we can add ansible_host_key_checking: no to the add_host task:

    - add_host:
        name: ansible-test-100
        ansible_host: 192.168.122.100
        ansible_ssh_private_key_file: /tmp/ansibletest-leases/100/id_ed25519
        ansible_host_key_checking: no

Now the playbook runs fine:

$ ANSIBLE_STRATEGY=linear ansible-playbook foo.yml 

PLAY [localhost] ******************************************************************************************************

TASK [add_host] *******************************************************************************************************
changed: [localhost]

PLAY [ansible-test-100] ***********************************************************************************************

TASK [copy] ***********************************************************************************************************
--- before
+++ after: /tmp/blah
@@ -0,0 +1 @@
+Hello world
\ No newline at end of file

changed: [ansible-test-100]

TASK [file] ***********************************************************************************************************
--- before
+++ after
@@ -1,4 +1,4 @@
 {
     "path": "/tmp/blah",
-    "state": "file"
+    "state": "absent"
 }

changed: [ansible-test-100]

PLAY RECAP ************************************************************************************************************
ansible-test-100           : ok=2    changed=2    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0   
localhost                  : ok=1    changed=1    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0

Yet this doesn't work with mitogen:

$ ANSIBLE_STRATEGY=mitogen_linear ansible-playbook foo.yml 

PLAY [localhost] ******************************************************************************************************

TASK [add_host] *******************************************************************************************************
changed: [localhost]

PLAY [ansible-test-100] ***********************************************************************************************

TASK [copy] ***********************************************************************************************************
fatal: [ansible-test-100]: UNREACHABLE! => {"changed": false, "msg": "Host key checking is enabled, and SSH reported an unrecognized or mismatching host key.", "unreachable": true}

PLAY RECAP ************************************************************************************************************
ansible-test-100           : ok=0    changed=0    unreachable=1    failed=0    skipped=0    rescued=0    ignored=0   
localhost                  : ok=1    changed=1    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0

I suggest to add mitogen_ssh_host_key_checking which accepts accept, enforce or ignore to mimic this behaviour:

    - add_host:
        name: ansible-test-100
        ansible_host: 192.168.122.100
        ansible_ssh_private_key_file: /tmp/ansibletest-leases/100/id_ed25519
        ansible_host_key_checking: no
        mitogen_ssh_host_key_checking: ignore

=>

$ ANSIBLE_STRATEGY=mitogen_linear ansible-playbook foo.yml 

PLAY [localhost] ******************************************************************************************************

TASK [add_host] *******************************************************************************************************
changed: [localhost]

PLAY [ansible-test-100] ***********************************************************************************************

TASK [copy] ***********************************************************************************************************
--- before
+++ after: /tmp/blah
@@ -0,0 +1 @@
+Hello world
\ No newline at end of file

changed: [ansible-test-100]

TASK [file] ***********************************************************************************************************
--- before
+++ after
@@ -1,4 +1,4 @@
 {
     "path": "/tmp/blah",
-    "state": "file"
+    "state": "absent"
 }

changed: [ansible-test-100]

PLAY RECAP ************************************************************************************************************
ansible-test-100           : ok=2    changed=2    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0   
localhost                  : ok=1    changed=1    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0

I'll open a draft pr for this, it'd be nice if you could have a look. Or maybe I'm just reinventing the wheel and such a feature already exists somehow 😄

[moreati]

If vanilla respects ansible_host_key_checking and Ansible+Mitogen doesn't, then I think we should fix that rather than add another variable. Did you have other reason(s) for adding mitogen_ssh_host_key_checking?

[philfry]

The reason I used a dedicated variable was because the skeleton for check_host_keys accepting 'accept', 'enforce' and 'ignore' was already there. Vanilla ansible only takes a bool as argument (which translate to 'enforce' or 'ignore' in mitogen) but I did not want to break any possible features regarding 'accept'. Also it was easier for me to test 😇

But I'm totally fine with hooking to ansible_host_key_checking and ansible_ssh_host_key_checking.

[philfry]

I changed the pr to use ansible_host_key_checking and ansible_ssh_host_key_checking. For now it's an extra commit but I'd squash them if you like the general idea of this pr.

[moreati]

I confirm the issue when add_hosts is passed ansible_host_key_checking.
It doesn't occur using env var ANSIBLE_HOST_KEY_CHECKING=no instead.

There are other differences I was unaware of. These may also qualify as bugs.
Fixing/amending them is probably outside scope of this issue.

Related differences

• Ansible prompts user to accept an unknown host key, Mitogen does not
• Ansible adds an accepted key to known_hosts, Mitogen+Ansible does not

Notes to self

• In ansible-core v2.16.6 parakmiko_ssh & ssh are the only connection plugins that mention ansible_host_key_checking.
• ansible_ssh_host_key_checking is accepted by paramiko_ssh, ansible_paramiko_ssh_host_key_checking has higher priority.
• Used rev 4996ec2 (ansible_mitogen: Fix "filedescriptor out of range in select()" in WorkerProcess #1065, very close to current master).

Reproduction

# issue1066_repro.yml 
- hosts: localhost
  gather_facts: false
  tasks:
    - known_hosts:  {name: "192.168.1.112", state: absent}
    - add_host:     {name: ansible-test-100, ansible_host: "192.168.1.112"}

- hosts: ansible-test-100
  gather_facts: false
  tasks:
    - ping:

# issue1066_repro2.yml
- hosts: localhost
  gather_facts: false
  tasks:
    - known_hosts:  {name: "192.168.1.112", state: absent}
    - add_host:     {name: ansible-test-100, ansible_host: "192.168.1.112", ansible_host_key_checking: false}

- hosts: ansible-test-100
  gather_facts: false
  tasks:
    - ping:

Playbook runs

Vanilla Ansible, add_hosts

$ ANSIBLE_STRATEGY=linear ansible-playbook issue1066_repro.yml            
[WARNING]: No inventory was parsed, only implicit localhost is available
[WARNING]: provided hosts list is empty, only localhost is available. Note that the implicit localhost does not match 'all'

PLAY [localhost] *************************

TASK [known_hosts] ***********************
ok: [localhost]

TASK [add_host] **************************
changed: [localhost]

PLAY [ansible-test-100] ******************

TASK [ping] ******************************
The authenticity of host '192.168.1.112 (192.168.1.112)' can't be established.
ED25519 key fingerprint is SHA256:AXEqIPmNmX02Nqs4H0fsIV286vmXvYhZXfPTO373c10.
This key is not known by any other names.
Are you sure you want to continue connecting (yes/no/[fingerprint])? ^C [ERROR]: User interrupted execution

Vanilla Ansible, add_hosts, ANSIBLE_HOST_KEY_CHECKING=no

Result: Playbook completes, key is added to known_hosts

$ ANSIBLE_STRATEGY=linear ANSIBLE_HOST_KEY_CHECKING=no ansible-playbook issue1066_repro.yml
[WARNING]: No inventory was parsed, only implicit localhost is available
[WARNING]: provided hosts list is empty, only localhost is available. Note that the implicit localhost does not match 'all'

PLAY [localhost] *************************

TASK [known_hosts] ***********************
ok: [localhost]

TASK [add_host] **************************
changed: [localhost]

PLAY [ansible-test-100] ******************

TASK [ping] ******************************
ok: [ansible-test-100]

PLAY RECAP *******************************
ansible-test-100           : ok=1    changed=0    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0   
localhost                  : ok=2    changed=1    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0   

Mitogen + Ansible, add_hosts

Result: No prompt, playbook fails

$ ANSIBLE_STRATEGY=mitogen_linear ansible-playbook issue1066_repro.yml                     
[WARNING]: No inventory was parsed, only implicit localhost is available
[WARNING]: provided hosts list is empty, only localhost is available. Note that the implicit localhost does not match 'all'

PLAY [localhost] *************************

TASK [known_hosts] ***********************
changed: [localhost]

TASK [add_host] **************************
changed: [localhost]

PLAY [ansible-test-100] ******************

TASK [ping] ******************************
[WARNING]: Unhandled error in Python interpreter discovery for host ansible-test-100: Host key checking is enabled, and SSH reported an unrecognized or mismatching host key.
fatal: [ansible-test-100]: UNREACHABLE! => {"changed": false, "msg": "Host key checking is enabled, and SSH reported an unrecognized or mismatching host key.", "unreachable": true}

PLAY RECAP *******************************
ansible-test-100           : ok=0    changed=0    unreachable=1    failed=0    skipped=0    rescued=0    ignored=0   
localhost                  : ok=2    changed=2    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0   

Mitogen + Ansible, add_hosts, ANSIBLE_HOST_KEY_CHECKING=no

Result: Playbook succeeds, key is not added to known hosts

$ ANSIBLE_STRATEGY=mitogen_linear ANSIBLE_HOST_KEY_CHECKING=no ansible-playbook issue1066_repro.yml
[WARNING]: No inventory was parsed, only implicit localhost is available
[WARNING]: provided hosts list is empty, only localhost is available. Note that the implicit localhost does not match 'all'

PLAY [localhost] *************************

TASK [known_hosts] ***********************
ok: [localhost]

TASK [add_host] **************************
changed: [localhost]

PLAY [ansible-test-100] ******************

TASK [ping] ******************************
ok: [ansible-test-100]

PLAY RECAP *******************************
ansible-test-100           : ok=1    changed=0    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0   
localhost                  : ok=2    changed=1    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0   

Mitogen + Ansible, add_hosts with ansible_host_key_checking=false

Result: Playbook fails, no prompt

$ ANSIBLE_STRATEGY=mitogen_linear ansible-playbook issue1066_repro2.yml 
[WARNING]: No inventory was parsed, only implicit localhost is available
[WARNING]: provided hosts list is empty, only localhost is available. Note that the implicit localhost does not match 'all'

PLAY [localhost] *************************************************

TASK [known_hosts] ***********************************************
ok: [localhost]

TASK [add_host] **************************************************
changed: [localhost]

PLAY [ansible-test-100] ******************************************

TASK [ping] ******************************************************
[WARNING]: Unhandled error in Python interpreter discovery for host ansible-test-100: Host key checking is enabled, and SSH reported an unrecognized or mismatching host key.
fatal: [ansible-test-100]: UNREACHABLE! => {"changed": false, "msg": "Host key checking is enabled, and SSH reported an unrecognized or mismatching host key.", "unreachable": true}

PLAY RECAP *******************************************************
ansible-test-100           : ok=0    changed=0    unreachable=1    failed=0    skipped=0    rescued=0    ignored=0   
localhost                  : ok=2    changed=1    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0   

Characterisation

# inv.ini 
mitogen     ansible_host=192.168.1.112

# issue1066_ping.yml 
- hosts: mitogen
  gather_facts: false
  tasks:
    - ping:

Playbook runs

Vanilla Ansible, host from inventory.

Result: prompts on an unknown key.

$ ansible -i inv.ini -o localhost -mknown_hosts -a"name={{ hostvars.mitogen.ansible_host }} state=absent"
localhost | SUCCESS => {"changed": false,"gid": 20,"group": "staff","hash_host": false,"key": null,"mode": "0600","name": "192.168.1.112","owner": "alex","path": "/Users/alex/.ssh/known_hosts","size": 13196,"state": "file","uid": 501}
$ ANSIBLE_STRATEGY=linear ansible-playbook -i inv.ini issue1066_ping.yml 

PLAY [mitogen] *********************

TASK [ping] ************************
The authenticity of host '192.168.1.112 (192.168.1.112)' can't be established.
ED25519 key fingerprint is SHA256:AXEqIPmNmX02Nqs4H0fsIV286vmXvYhZXfPTO373c10.
This key is not known by any other names.
Are you sure you want to continue connecting (yes/no/[fingerprint])? ^C [ERROR]: User interrupted execution

Vanilla Ansible, host from inventory, ANSIBLE_HOST_KEY_CHECKING=no

Resut: Completes, key is added to known_hosts

$ ANSIBLE_STRATEGY=linear ANSIBLE_HOST_KEY_CHECKING=no ansible-playbook -i inv.ini issue1066_ping.yml

PLAY [mitogen] *********************

TASK [ping] ************************
ok: [mitogen]

PLAY RECAP *************************
mitogen                    : ok=1    changed=0    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0   

Mitogen + Ansible, host form inventory

Result: doesn't prompt, playbook fails

$ ansible -i inv.ini -o localhost -mknown_hosts -a"name={{ hostvars.mitogen.ansible_host }} state=absent"
localhost | CHANGED => {"changed": true,"gid": 20,"group": "staff","hash_host": false,"key": null,"mode": "0600","name": "192.168.1.112","owner": "alex","path": "/Users/alex/.ssh/known_hosts","size": 13196,"state": "file","uid": 501}
$ ANSIBLE_STRATEGY=mitogen_linear ansible-playbook -i ~/ansible_inventory.yml issue1066_ping.yml                             

PLAY [mitogen] *********************

TASK [ping] ************************
[WARNING]: Unhandled error in Python interpreter discovery for host mitogen: Host key checking is enabled, and SSH reported an unrecognized or mismatching host key.
fatal: [mitogen]: UNREACHABLE! => {"changed": false, "msg": "Host key checking is enabled, and SSH reported an unrecognized or mismatching host key.", "unreachable": true}

PLAY RECAP *************************
mitogen                    : ok=0    changed=0    unreachable=1    failed=0    skipped=0    rescued=0    ignored=0   

Mitogen + Ansible, host form inventory, ANSIBLE_HOST_KEY_CHECKING=no

Result: Playbook completes, key is not added to known hosts

$ ANSIBLE_STRATEGY=mitogen_linear ANSIBLE_HOST_KEY_CHECKING=no ansible-playbook -i ~/ansible_inventory.yml issue1066_ping.yml

PLAY [mitogen] *********************

TASK [ping] ************************
ok: [mitogen]

PLAY RECAP *************************
mitogen                    : ok=1    changed=0    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0   

[moreati]

Notes to self, these aren't things to address within this issue/PR

• Ansible SSH option is "host_key_checking"

• ansible-core 2.16.6 ansible.plugins.connection.ssh.Connection._build_command() adds -o StrictHostKeyChecking=no if the option evaluates False (default True)

• OpenSSH 9.6 StrictHostKeyChecking accepts any of true, false, yes, no, ask, off, accept-new https://github.com/openssh/openssh-portable/blob/V_9_6_P1/readconf.c#L913-L922

• Mitogen <= 0.3.7 maps Ansible's host_key_checking -> kwarg check_host_keys

  host_key_checking	check_host_keys
  False	ignore
  True	enforce

  check_host_keys accepts any of accept, enforce, ignore

  • enforce -> StrictHostKeyChecking yes
  • accept -> StrictHostKeyChecking ask and Mitogen answers
    any host key prompts with yes.
  • ignore -> StrictHostKeyChecking no, UserKnownHostsFile /dev/null, GlobalKnownHostsFile /dev/null

• ansible_mitogen.transport_config contains a mishmash of naming conventions.

  • no prefix (e.g. Spec.port, Spec.become_exe)
  • prefix (e.g. Spec.mitogen_kind)
  • prefix & plugin name (e.g. Spec.ansible_ssh_timeout, Spec.ansible_doas_exe, Spec.mitogen_lxc_path)

• ansible_mitogen.connection.convert_bool() possibly duplicates ansible.module_utils.parsing.convert_bool.boolean(), and/or it should be somewhere more generic.