Disable backup handler in tests #1399
Conversation
|
Allright one of the test runs failed, so it's clearly not the only culprit... EDIT: the file limit was somewhat low on semaphore, since I added the limit configuration it doesn't seem to repro |
We've been witnessing random test suites freezes (since ages). We've observed that when these freezes happen, there are usually a lot of "too many open files" errors raised by the OS. The backup handler is a likely culprit as the IntegrationSpec is running multiple nodes and exchanging HTLCs at a fast rate. At least it won't hurt disabling it in tests, and will speed up the test suite.
| # configure file limits | ||
| - ulimit -S -n 1024000 | ||
| - echo "fs.file-max = 1024000" | sudo tee -a /etc/sysctl.conf |
There was a problem hiding this comment.
Now that we have fixed the test issues in #1400 I wonder if this is really necessary. If it is, why do the test pass? Shouldn't the "too many open files" error completely break the tests?
There was a problem hiding this comment.
Shouldn't the "too many open files" error completely break the tests?
It looks like it does, but it makes our test suite hang so we don't cleanly witness it (because we kill the build after 15 minutes to avoid wasting resources).
I wonder if this is really necessary
I can't be entirely sure, but since I added that I haven't repro-ed the hang on semaphore (I restarted the test suite several times), so it looks like it happens less often (hopefully it doesn't happen anymore). At least it can't hurt to raise this limit (the ulimit is already at 1024000 by default in semaphore, but it wasn't configured in sysctl.conf).
There was a problem hiding this comment.
Ok, so this fixes the intermittent hanging while #1400 introduced then fixed a permanent hanging.
We've been witnessing random test suites freezes (since ages).
We've observed that when these freezes happen, there are usually a lot of "too many open files" errors raised by the OS.
The backup handler is a likely culprit as the IntegrationSpec is running multiple nodes and exchanging HTLCs at a fast rate.
At least it won't hurt disabling it in tests, and will speed up the test suite.