=htc #1245 fix pool idle-timeout race condition

[jrudolph]

Previously, a shutdown request didn't check if there are outstanding
requests so that outstanding responses were just lost.

This is still missing tests, but I haven't been able to reproduce it reliably. The reason is that there's only a short time frame where you can hit the race, i.e. while the scheduled gateway.shutdown is dispatched through PoolMasterActor to PoolInterfaceActor a new request must come in.

[akka-ci]

Test PASSed.

[ktoso]

Fix LGTM, thanks for the explanation - perhaps add it above the shuttingDown too so it's easier to remember once we revisit this code some day why it was needed without reading the impl.

I see why adding a good reproducer will be tricky... Starting a server that's very slot in delivering the responses perhaps could do it?

[jrudolph]

I see why adding a good reproducer will be tricky... Starting a server that's very slot in delivering the responses perhaps could do it?

No, actually that's not where the race condition is. It's between PoolMasterActor and PoolInterfaceActor. PIA schedules the idle-timeout and when it triggers it sends a Shutdown message to the PMA. This will then send a Shutdown message to the PIA. The other process is an incoming request that will be dispatched from the PMA to the PIA and which will reset the idle-timeout. If the request comes in right after the idle-timeout triggered the problem happened.

[jrudolph]

Added another comment as you suggested.

[akka-ci]

Test PASSed.

[jrudolph]

I have an idea for a test, let's try that.

[ktoso]

Thanks for explaining the race in detail, that's indeed very tricky to reproduce :O Looking forward to see how you get around to it :)

[jrudolph]

Haha, the test revealed another race condition 🙄

[ktoso]

The pool is a present that keeps on giving... 🙄

[jrudolph]

The test turned out to end up quite simple. It failed for me reliably with the timings given but that might be different on another machine.

[akka-ci]

Test PASSed.

[jrudolph]

Fixes #1245