Handle race condition between cancel requests and server reuse

[JelteF]

Cancel requests happen out of band with regular queries. Because
PgBouncer reuses server connections across clients its possible to get
race conditions here if you're not careful.

Without this patch there were two possibilities for a race condition
where a cancel request for client A could cancel a query that was sent
by client B. These race conditions happened when:

1. A cancel request on client A was forwarded to the server only after
   the server was reused by another client.
2. A cancel request for client A was already forwarded to the server,
   but not yet handled by the server and then the server was reused for
   client B.

The first race conditions was especially likely if connection
establishment takes a while, for example because of TLS roundtrips.
Because a new connection needs to be established for every forwarded
cancellation request. While establishing this connection the query
running on the server might finish by itself already, and then the
server could be reused by another already waiting client.

To resolve these race conditions the following two fixes are implemented:

1. We now only forward the cancel request if the server connection over
   which the query was sent hasn't become idle or disconnected in the
   meantime. This is done by doubly-linking cancel requests and the
   original server connection that was used for the query.
2. Once a cancel request is forwarded to the server, we do not allow reuse of
   the server until the cancel request completes. This required tracking
   the response of the server for cancel requests too, instead of
   closing the connection right after forwarding the cancel.

Fixes #544

In passing this also adds more comprehensive comments to the different
StatLists in the PgPool struct. Because understanding how the state
machine worked was the part of this PR that took me the longest.

[x4m]

I'm not really familiar with the PgBouncer's code, but in Odyssey we generate cancelation key for each client. So client cannot cancel someone else's server connection. Does this resolve the issue?

Anyway the behavior of cancels is somewhat problematic. Sometimes server's CPU is starving due to large amount of fork()s. Clients start to cancel their queries. But cancels produce more fork()s! And ISTM this PR addressess the issue, doesn't it?

[JelteF]

@x4m PgBouncer also generates unique cancellation tokens per client. So, that alone won't protect you against the same issue in Odyssey.

To clarify the issue: These unique cancellation tokens, when received by the connection pooler, are handled by by sending a cancellation token to the actual Postgres server. This cancellation token should be the one that matches the postgres connection on which the query of the client is currently being sent. The problem is that while sending this cancellation token to Postgres, the query on that connection might finish, and the postgres connection might be reused for a query from another client. If that's possible in Odyssey then you almost certainly have the same bug.

One easy thing to try is run the test I wrote for this PR against Odyssey and see what happens.

Anyway the behavior of cancels is somewhat problematic. Sometimes server's CPU is starving due to large amount of fork()s. Clients start to cancel their queries. But cancels produce more fork()s! And ISTM this PR addressess the issue, doesn't it?

I'm not sure why you think this PR addresses the issue. It's definitely not the purpose of the PR, but it might reduce cancellation load as a side-effect.

[sumedhpathak]

The other comment we discussed was to add this behind a feature flag, so we could toggle this off in case there is an issue.

[sumedhpathak]

LGTM!