plays with many hosts fail with `too many open files` error on macOS

[peshay]

• Which version of Ansible are you running?

ansible 2.7.7
  ansible python module location = /usr/local/lib/python3.7/site-packages/ansible
  executable location = /usr/local/bin/ansible
  python version = 3.7.2 (default, Feb 19 2019, 14:40:12) [Clang 10.0.0 (clang-1000.11.45.5)]

• Is your version of Ansible patched in any way?
  No
• Are you running with any custom modules, or module_utils loaded?
  No
• Have you tried the latest master version from Git?
  Yes
• Do you have some idea of what the underlying problem may be?
  https://mitogen.rtfd.io/en/stable/ansible.html#common-problems has
  instructions to help figure out the likely cause and how to gather relevant
  logs.
  I have no problem when I do it to one host. With -f 1 it worked till host 48 and then all hosts failed with

fatal: [some-host]: UNREACHABLE! => {"changed": false, "msg": "Child start failed: [Errno 24] Too many open files. Command was: ssh -o \"LogLevel ERROR\" -l root -o \"Compression yes\" -o \"ServerAliveInterval 15\" -o \"ServerAliveCountMax 3\" -o \"BatchMode yes\" -o \"StrictHostKeyChecking yes\" -C -o ControlMaster=auto -o ControlPersist=60s some-host /usr/bin/python -c \"'import codecs,os,sys;_=codecs.decode;exec(_(_(\\\"eNqFkU1Lw0AQhs/Nr8htdunSbmr9CiwoPYgHEYKYgxbJx8YuprvLJmmsv95pIjapBw8L8zDvzDu8G7FYmGpmlZWEeo61A1KFj1AY90Fo6E2wzhu7IJwFnNMjR2xIDrtBz1lpKkmiIbghxENoEdCw2qN9mdTouvWF8CFPXKs0+InOu6b8lFlTJ2kpu/a8qdw8VXpu9/XGaMA7JyeyqegGd9JVyuiX8Gzd2Uq9Uw4ZbqO7Zw5rMR7rNYglGTfYGKdAtqo271KHyaa5wRcul8uLKwrUww2tU7UkAYOH+6dHzvmrBnTOTI4BU28l3sgh4txYqTFYcCnQmZNJToLzy2tOGXwpi5sKK466mEGbwiH1wv4YrLq6T/JE3f6n/ntlML7y948W9BvKaq7Q\\\".encode(),\\\"base64\\\"),\\\"zip\\\"))'\"", "unreachable": true}

• Mention your host and target OS and versions
  host: macOS 10.14.2
  targets: mostly CentOS Linux release 7.6.1810 (Core)
• Mention your host and target Python versions
  host: 3.7.2
  targets: 2.7.5

[dw]

This is vaguely related to #470. How many hosts are you targeting?

It's not mentioned in the docs anywhere, and I simply haven't had a chance to add a (simple) error message for it yet. Quite a lot of file descriptors are currently required

Increase your ulimit -n to at least 32 + 8 + (2 * number of hosts), and preferably double that again.

[dw]

This happens for two reasons, the first already mentioned: due to some messy internal handling that is very subtle to fix without breaking stuff, we currently use 2 file descriptors for every connection. It's "simple" to fix this, but anything relating to changing file descriptor lifecycle must be done very carefully :)

The second reason is because unlike with normal Ansible, there is a single process that really has connections to every machine, and file descriptor limits are per-process, not per-user. In normal Ansible, those descriptors are usually spread out across many child tasks, but for that reason, Ansible cannot keep the connections it creates open across the lifetime of a single task

[peshay]

Yes, thanks! Setting the ulimit -n higher helped.

[dw]

Going to keep this open for now as a reminder to add a useful error hint! Thanks for reporting this

[dw]

I've started adding hints for the most easy to reproduce cases, but there is no central place where all possible file limit errors can be trapped -- they happen all over the Python standard library, for example.

I've also added a change to automatically increase the Ansible run's soft file limit to the configured hard limit. At least on distributions like Ubuntu, the default soft limit is 1024 while the default hard limit is closer to 1 million. That should drastically cut down on the number of users who ever bump into this.

Thanks again for reporting!