Ellis stops replying to requests

[graemerobertson]

I spin up a Clearwater Docker deployment and run our live test suite against (see the live-test-docker CI job). The live tests fail reliably. Specifically, about midway through the run, an Ellis request times out. The remaining tests continue to run as normal once the request times out.

I've no idea what's going on here really. There's nothing obvious in the nginx logs or ellis logs. There had been no code changes in any of our repos when this first started happening. I thought we might be using a newer version of another package that ellis depends on, and that was bugged, but I spun up a normal AIO node (no Docker) and didn't reproduce this problem.

Setting to medium priority because it damages our docker story.

[graemerobertson]

Update on this. We've just spun up a clearwater-docker deployment and run the live tests successfully 13 times in a row (having previously been failing most times). Now we've just failed again, this time with every single Ellis request seemingly timing out.

No idea.

[graemerobertson]

I'm seeing this very infrequently again now.

Still no idea.