Execution of ansible-runner hangs with no output either to stdout or stderr.

[iv1111]

Hello,

I've been using a standalone ansible-runner application in the context of ManageIQ. Recently, I've been having a problem with the usage of ssh_key (as described here int the documentation).

After placing my unencrypted private SSH key into that file and running my playbook with the ansible-runner I've noticed that the runner hangs indefinitely and I can see no output at all (even in debug mode) under /tmp/ansible-runnerX/artifacts/result/{stdout,stderr} directory. I have tried to reproduce the steps that the runner does by looking at /tmp/ansible-runnerX/artifacts/result/command.

It seems that this is the step where it hangs: ssh-add /tmp/ansible-runnerX/artifacts/result/ssh_key_data. It looks like the ssh_key_data is a named pipe that you can't read anything from. I've tried opening it with other applications like vim with the same result (it hangs).

A brief search showed me some similar problems observed in AWX:

• FIFO pipe code will hang indefinitely preventing any logging output awx#11453
• Unable to use SSH keys as machine credentials - Job hangs with very little output awx#11613
• Jobs using ssh credentials get stuck when loading passphrase awx#11051
• Jobs with ID > 9999 are stuck in pending state, but nothing is happening awx#10489

I don't understand why this problem occurs now as I've tested both encrypted and unencrypted private keys with ansible-runner in the past and it worked well.

Context:

• GNU/Linux RHEL 8.4
• ansible-runner: 2.1.1
• Python 3.9.7

[TheOnion7]

As I mentioned in ansible/awx/#11613 , in my case , the culprit was falcon-sensor. Hope it will help you out.

[iv1111]

@TheOnion7 You're absolutely right. I confirm that the problem is the falcon-sensor. It seems to interfere with whatever is supposed to populate the ssh_key_data pipe. Thanks for your help!

Question to the ansible-runner team. Do you plan a workaround for this issue? It unfortunately makes using private key access with the runner unusable (if falcon-sensor is enabled).

[matburt]

I'm open to suggestions. The use of named pipes here is to specifically prevent sensitive data from landing on the filesystem. It seems like the real problem is that falcon-sensor breaks normal usage of named pipes? The only remediation would be to not use named pipes which has its own set of caveats and problems.

[iv1111]

Yeah, if we look at it from another angle then the falcon-sensor blocks the execution of a non-malicious application, which it shouldn't.

However, once this occurs the ansible-runner just sits there writing neither to stdin nor to stderr. I think it would be more dev/user friendly if the stderr would at least indicate that it's not possible to read from the ssh_key_data or that it timed out doing so.

Also, adding the corresp. information about the falcon-sensor as a side note here under this section would be beneficial too.

[matburt]

Knowing we are blocked on an a pipe that will never return data seems like we would need to solve the halting problem first. I'm open for a PR that adds information to the docs, though.

[iv1111]

Knowing we are blocked on an a pipe that will never return data seems like we would need to solve the halting problem first.

Determining that an algorithm terminates is certainly undecidable but what I meant is perhaps a timeout can be added that'd trigger if no data comes from the pipe for a certain amount of time and would terminate the execution writing a message to stderr.

If that's not possible/worth the effort, then maybe at least in a --debug mode one could be more verbose and write a message to stdout before starting to wait for the pipe, e.g. "initiating the reading from ssh_key_data named pipe" (or similar). This way it would give an indication to the user what the last operation was before it got stuck.