ansible-test: Set max number of open files in containers to 10240

[Andersson007]

SUMMARY

I'm developing a collection for one service from scratch.

I wrote integration tests.

When trying to run the service with ansible-test with --docker, I got the following error from the service I want to test against:

ERROR: failed to start server: failed to create engines: hard open file descriptor limit of 1024 is under the 
minimum required 1956

I tried to run ulimit -n 9000 inside the container with and without --docker-privileged -> in both the cases i got "error setting limit (Operation not permitted)".

With this patch, the following command works now and the service starts successfully:

ansible-test integration test_service_query --docker ubuntu2004 --docker-ulimit "nofile=90000:90000" -vvv > ~/test.log

I'm also OK with if ulimits will be increased in docker files for the tests containers
However, this solution gives more flexibility and covers more possible cases

UPDATE
During discussion we decided to hardcode 10240 as max number of open files to not provide the additional option.

ISSUE TYPE

• Feature Pull Request

COMPONENT NAME

ansible-test

[mattclay]

Instead of requiring users to pass another option to make tests work, I think we should pick a ulimit and have ansible-test set it automatically. The question then becomes, what should the ulimit be?

[Andersson007]

Instead of requiring users to pass another option to make tests work, I think we should pick a ulimit and have ansible-test set it automatically. The question then becomes, what should the ulimit be?

XFS supports 2 ** 64 files in the system:) I don't think many services check and require higher ulimit than the standard number. We could increase it, say, twice, i.e. to 2048 for now (that's also fine for my service). Then, we can increase it more later if someone else needs it.

@mattclay if it sounds sensible, i could adjust the number in this PR.

[mattclay]

@Andersson007 Yeah, a hard-coded ulimit of 2048 seems like a reasonable fix for now, since this is the first time I'm aware that this issue has come up.

[Andersson007]

/azp run

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).

[Andersson007]

the tests are green now

[mattclay]

Taking a closer look at this, I'm not sure this is what we need.

The default hard limit comes from the Docker daemon, which at least on my system is much higher than 2K. Attempting to raise that limit with --ulimit fails, so this change would only lower it.

The soft limit of 1K comes from ansible-test itself. Changing it with --ulimit will only affect what the container uses, not what ansible-test uses inside the container.

@Andersson007 Can you run ansible-test shell --docker -vv without this change and see what ansible-test reports for RLIMIT_NOFILE inside the container (the second one reported)?

[Andersson007]

I see the following:

RLIMIT_NOFILE: (1024, 1048576)    # this is the first one
...
RLIMIT_NOFILE: (1024, 1024)   # this is at the end
...

I also ran

root@3950af2f216f:~/ansible$ ulimit -n
1024

[Andersson007]

$ docker --version
Docker version 19.03.13, build 4484c46

Fedora release 33. Docker was installed via dnf, settings are default

[mattclay]

@Andersson007 OK, that's what I expected. Try changing the limit to 2048 here:

ansible/test/lib/ansible_test/_internal/constants.py

Line 7 in 1c34492

SOFT_RLIMIT_NOFILE = 1024

Let me know if that resolves the issue you're seeing.

[Andersson007]

@mattclay now the log contains
RLIMIT_NOFILE: (1024, 1048576)
RLIMIT_NOFILE: (1024, 1024)
ANSIBLE_PYTHON_MODULE_RLIMIT_NOFILE=2048

And my tests fail again with the original error:

ERROR: failed to start server: failed to create engines: hard open file descriptor limit of 1024 is un
der the minimum required 1956

So it didn't help

[mattclay]

@Andersson007 My main concern with this now is that it will lower the limit for existing users if their limit is higher than 2K. Unfortunately I couldn't find any way to query the docker default without actually running a container. We could just try 2K and see if anyone reports issues, or perhaps pick a higher value to be safe. Since we're still setting the soft limit within ansible-test, a higher hard limit for docker shouldn't cause problems unless it exceeds the system's limit. Thoughts?

[Andersson007]

@mattclay it's hard to imagine if anyone really need more than even 2048 files open for purposes of Ansible tests but who knows. The most important question is if the higher limit can cause security issues for testing infrastructure? If not, we could increase it to 4096 or even 10240 and see.

[mattclay]

@Andersson007 Lets go with 10K and see if anyone has issues with it.

[Andersson007]

@mattclay done, PTAL

[Andersson007]

@mattclay while the collection is still in development and I think it'll take some time, right now this change is not really needed.
I can test locally running ansible-test from the branch.
I'll close this PR and when the collection is ready for submission, I'll re-open it. I think it would be better.
Thank you!

[Andersson007]

@Andersson007 Lets go with 10K and see if anyone has issues with it.

@mattclay it's been done and i've just transferred the new collection to ansible-collections.
Can we merge this to allow the collection to be tested against devel?