[wsrelay] Give connection tasks time to clean up

[relrod]

SUMMARY

When we close/cancel a connection to a web node, give the task time to clean up after itself and cleanly exit. Otherwise, the Python GC might clean up the task too early and this leads to ugly log messages like this: "Task was destroyed but it is pending!"

ISSUE TYPE

• Bug, Docs Fix or other nominal change

COMPONENT NAME

• API

[fosterseth]

were you able to reproduce the scenario that logged the "Task was destroyed.." message?

[relrod]

I wasn't able to reproduce it, but I am fairly confident that this would cause that in some conditions. Even if there's something else causing that, this is still something that I think needs to be fixed.

[relrod]

https://bugs.python.org/issue44665 - This is kind of a reference, if you need a citation 😄

And it specifically mentions the error we're seeing. The issue itself is about adding asyncio documentation to prevent doing exactly what the current code is doing (killing off the strong reference to the task before it is completed).

This seems to be a well-known phenomenon in python asyncio, you have to keep a strong reference around to tasks you create with create_task() or bad things can happen.

[AlanCoding]

Needs a rebase.

I also kind of liked the callback method in the python bug you linked task.add_done_callback(lambda t: running_tasks.remove(t)), since it would allow instance ejection from the tracking dictionary. But this interest is almost entirely academic, as there's no good functional reason for such a change.

You should consider my comment about running these concurrently, but either way there is probably little risk because removing a node is an infrequent task and this is unlikely to choke because the main loop gets stalled waiting for the cancellations.

[AlanCoding]

I see the error mentioned in issues #14057 and #14003 and both mention a fairly ancient resolution of the k8s log size issue. So I think we're good.

[AlanCoding]

Looks great, very async