[WIP] Handle disappearing hosts

[bcoca]

fixes #55669

ISSUE TYPE

• Bugfix Pull Request

COMPONENT NAME

core

[nitzmahone]

I built a reproducer for this with an inventory plugin (intending to turn it into a test for this PR), but as-is doesn't fix the issue (piqued my interest as I've often wondered how we'd handle disappearing hosts)...

[bcoca]

@nitzmahone what is the reproducer? original issue was mostly a confluence of race conditions and i was having a hard time making it happen.

[nitzmahone]

vanish.py

from ansible.plugins.inventory import BaseInventoryPlugin

runcount = 0


class InventoryModule(BaseInventoryPlugin):

    NAME = 'vanish'

    def verify_file(self, path):
        if not path.endswith('.vanish.yml'):
            return False
        return super(InventoryModule, self).verify_file(path)

    def parse(self, inventory, loader, path, cache=True):
        global runcount
        runcount += 1
        group = 'vanishgroup'
        inventory.add_group(group)
        inventory.add_host('h1', group=group)
        print('runcount is %d' % runcount)
        if runcount <= 1:
            inventory.add_host('disappearing_host', group=group)
        inventory.set_variable(group, 'ansible_connection', 'local')

disappearing_host.yml

- hosts: vanishgroup
  gather_facts: no
  tasks:
  - shell: echo yo
    notify: dude
  - meta: refresh_inventory
  - ping:
  handlers:
  - name: dude
    debug: msg=howdy

disappear.vanish.yml

plugin: vanish

ansible-playbook -i disappear.vanish.yml disappearing_host.yml

reproduces it 100% for me

[nitzmahone]

I suspect there are probably a number of other potential hazards for this, but handler exec is a big one. It seems to work as expected without a handler and using the default callback.

[bcoca]

this looks like diff case than the one i was handling: free strat + one of disappearing hosts being slow to return previous result when inventory (run_once) was refreshed.

so we probably have multiple cases that cause errors when a host disappears mid play.

[nitzmahone]

Yeah, I thought about strategy: free too, though I didn't see anything mentioning that in the original issue... Reproducing that case under free should be pretty easy with this- just do a shell: sleep {{ bla }} and have the disappearing host's value of bla be longer than the others.

[nitzmahone]

I'm also wondering if we should leave "removed" hosts in place and just have an active flag on them that play_iterator et al would just ignore when asking for the next host... Would probably solve a lot of these kinds of problems, since other things like task completion and stats that need to look up hosts would still find them. Otherwise this is probably going to be a long series of race condition whack-a-mole...

[bcoca]

discussed a lot in irc with original author so ticket does not have good reflection of full issue, still unsure that is the case since i could not easily reproduce.

as for ignoring the host, that is kind of what this does, sets state to 'nothing to do' and puts in removed_keys() so it is skipped even if iterator has it in preset objects

need to check why handlers still run, but also need to discuss --force-handlers in this case as well as meta: clear_host_errors

[bcoca]

new state 'removed' for host so iterator skips, but still reports on it (remove from hostvars?)

[webknjaz]

/azp run

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).