Observability

[nikita-volkov]

Adds a stream of events describing the significant state changes in the pool for monitoring purposes.

[nikita-volkov]

@robx Hey Rob! Can you please give a feedback on this?

[steve-chavez]

Many thanks for this @nikita-volkov! Just started experimenting on PostgREST/postgrest#3229

Do you think hasql-pool could also expose metrics like "total waiters"? At one point (when hasql-pool used resource-pool) we had that one and other metrics, see PostgREST/postgrest#2129.

[steve-chavez]

When could this happen? When the pool size is big and there are lots of connections?

[nikita-volkov]

I'm talking about the user supplied action here. It should be quick since it will block the pool processes.

[steve-chavez]

Q: Why does the UUID need to be generated internally? (just wondering if it can be left out for simplifying the code)

[nikita-volkov]

It's for identifying the connection. Lets the user isolate events on a particular connection

[nikita-volkov]

Many thanks for this @nikita-volkov! Just started experimenting on PostgREST/postgrest#3229

Great! Thanks! Looking forward for feedback!

Do you think hasql-pool could also expose metrics like "total waiters"? At one point (when hasql-pool used resource-pool) we had that one and other metrics, see PostgREST/postgrest#2129.

Could you clarify on what you imply by "total waiters"? Unfortunately I couldn't derive that from the link.

[steve-chavez]

Could you clarify on what you imply by "total waiters"? Unfortunately I couldn't derive that from the link.

So the motivation is to be able to prevent poolAcquisitionTimeout, on the postgREST side this results in an API error (ref).

For this, I understand we could use the following metrics:

1. pending_requests (same idea as total waiters) https://opentelemetry.io/docs/specs/semconv/database/database-metrics/#metric-dbclientconnectionspending_requests

   • If this keeps increasing it would be an indicative that a pool acquisition timeout will happen. So an admin can adjust the pool size or tune the queries to prevent it.
   • I've also seen this named as "concurrentWaiters: Number of waiting threads" (ref)

2. pool wait time https://opentelemetry.io/docs/specs/semconv/database/database-metrics/#metric-dbclientconnectionswait_time

   • Same idea as above, if it keeps increasing then a pool timeout will happen.

[nikita-volkov]

Thanks for the references! I'll see what can be done.

[nikita-volkov]

I've taken a look. Seems like both of those metrics are already computable without the features involved in this PR.

executeObservedSession :: Pool -> Monitor -> Session -> IO ()
executeObservedSession pool monitor session = do
  Monitor.incGauge monitor "db.client.connections.pending_requests"
  startTime <- getCurrentTime
  Pool.use pool do
    liftIO $ do
      endTime <- getCurrentTime
      Monitor.observeHistogram monitor "db.client.connections.wait_time" (diffUTCTime endTime startTime)
      Monitor.decGauge monitor "db.client.connections.pending_requests"
    session

@steve-chavez Is that correct?

[steve-chavez]

@nikita-volkov You're right! Thanks for the snippet!

[steve-chavez]

Monitor.observeHistogram monitor "db.client.connections.wait_time" (diffUTCTime endTime startTime)

Curious, where does observeHistogram come from? I found decGauge on hoogle but not observeHistogram.

[nikita-volkov]

Ah. It was just pseudocode. I was implying something like this.

[nikita-volkov]

Thanks for cooperation guys! It's released.