`OSError: [Errno 9] Bad file descriptor` heisen-failure

[sed-i]

Occasionally my integration tests fail early on with:

OSError: [Errno 9] Bad file descriptor

yet a subsequent test passes smoothly without any changes to anything.

This is a randomly recurring issue observed continuously in the past few months.

heisen-failure.log

[cderici]

This seems to be related to #580 . We worked on a bunch of issues related to asyncio in there, and we saw that some of the problems were caused by our tests themselves. So I'd start the checking from your test called test_build_and_deploy_with_alternative_images.

For example, one of the things that I'd look at, is if the test is manually opening connections then I'd check if it is closing all of them before exiting. Another thing to check would be to see if it's sharing any connections opened somewhere else, if so, then it should be closed somewhere eventually. Another thing to check would be to make sure we're not directly accessing the ws object in the test (e.g. connection.ws.close()), we found that it causes a bunch of other problems (because asyncio tasks/events are closely tied to the connection itself). Our tests are currently passing consistently after we fixed these sorts of points in our tests.

We also found a bunch of spots internally in pylibjuju where we weren't treating the connection object properly, so those seems to be fixed in #580, but it's of course possible that we have missed something in there and that's the cause of your problem.

So to diagnose for both cases, I'd need to see exactly how the test_build_and_deploy_with_alternative_images (or any other failing test) is using pylibjuju.

[cderici]

Is it this one https://github.com/canonical/prometheus-operator/blob/ea351276712fd54c45635181eb5296e39274bff1/tests/integration/test_charm.py#L21?

[sed-i]

Is it this one https://github.com/canonical/prometheus-operator/blob/ea351276712fd54c45635181eb5296e39274bff1/tests/integration/test_charm.py#L21?

Yes, exactly.

[sed-i]

Got a tip from @rbarry82:

Juju is closing the web socket, then asyncio tries to set a property on it. fnctl will not write to a closed FD
[possibly] a systemd tmpfile timer is removing the LXD socket
it’s also easy enough to just systemctl disable systemd-tmpfiles-clean.timer

[SimonRichardson]

websockets.exceptions.ConnectionClosed: code = 0 (unknown), reason = websocket closed

If you look in the logs, it looks like the connection is also being close, but we're not getting a close control frame to tell us why.

This could actually be the cause of the failure, but it requires Juju to correctly send closed control frames correctly to then see if the problem goes away. See #604

[github-actions]

This issue is marked as incomplete because it has been open 30 days with no activity. Please remove incomplete label or comment or this will be closed in 5 days.

[github-actions]

This issue was closed because it has been stalled for 5 days with no activity.