Thread pool configuration

[ppllaxxa]

Hello All!

I am searching an advice. Short description of our archetecture:
We have one controller as entry point, which is receiving notifications which should be processed.
This controller parses params and sends message to Akka aktor (notification feed).
Notification aktor is making a decision based on message which child aktor should be created and sending message to corresponding child aktor which is created for each message.
And each child aktor makes a calls to some other classes which performs all corresponding work for each message.
This classes designed in such way that performs all the fork in Future with appropriate execution context (so we configured separate contexts and not using default playframework dispatcher for such work).
Once work is done child aktor sends message back to notification aktor, notification aktor sends poison pill to child.

Most of work requires DB connection, so based on this article https://github.com/brettwooldridge/HikariCP/wiki/About-Pool-Sizing
we configured our connection pool accordingly and our separate execution contexts also have the same pool size, i.e. application.conf:

db {
  default {
    hikaricp {
      maximumPoolSize = 20
    }
  }
}

contexts {
  work1 {
    executor = "thread-pool-executor"
    throughput = 1
    thread-pool-executor {
      fixed-pool-size = 5
    }
  }
  work2 {
    executor = "thread-pool-executor"
    throughput = 1
    thread-pool-executor {
      fixed-pool-size = 5
    }
  }
  work3 {
    executor = "thread-pool-executor"
    throughput = 1
    thread-pool-executor {
      fixed-pool-size = 5
    }
  }
  work4 {
    executor = "thread-pool-executor"
    throughput = 1
    thread-pool-executor {
      fixed-pool-size = 5
    }
  }
}

Maybe my understooding of this article (https://github.com/brettwooldridge/HikariCP/wiki/About-Pool-Sizing) wrong and it describes the maximum connections which DB server could process simultaneously.

My concern - I supposed that with correct configuration of thread and connection pool our application should consume most of resources of server during high load but actual results are different:
During high load of application we could see in our logs that number of messages are grown in queue (we are logging it in our notification aktor) but consumption of server resources are low.

Maybe there is a way to get info about each execution context in realtime (number of threads in work/number of idle threads)?

[mkurz]

Where exactly do you use the work[1-4] thread pools? Like do you use it only when accessing the database? I think just one thread pool for accessing the db should be enough with a fixed size of 20 threads (same size like db pool) and only use that pool when accessing the db, but for nothing else.
But it's a bit hard to tell from this information what's going on.

[ppllaxxa]

Got it, thank you. What about size of connection pool? Should it be calculated based on app server resources or db server resources?

Is there any way to check current status of execution context - threads in work/idle threads?

[mkurz]

The connection pool size and also the thread pool used for db operations usually should be calculated based on the the db server resources, I think that's what the HikariCP docs also say (I read them a long time ago).

I think you can just use any monitoring tools, like VisualVM or jconsole to get better insights inthe thread behaviour.