pool: Improve scalability of nearline storage subsystem

Motivation:

The nearline storage subsystem has a thread pool for various tasks. Some
of these tasks are blocking, e.g. callouts to the name space. Since the
thread pool is unlimited, a high inflow of new requests can cause the
thread pool to grow rapidly and even exceed thread limitations. In that
case the pool dies.

Modification:

Make the thread pool size configurable and add a conservative default.

Result:

Introduced the new pool.limits.nearline-threads property to limit the
number of threads used by the nearline storage subsystem. The default is
30 threads.

Target: trunk
Request: 2.15
Request: 2.14
Request: 2.13
Require-notes: yes
Require-book: no
Acked-by: Albert Rossi <arossi@fnal.gov>
Patch: https://rb.dcache.org/r/9151/

modules/dcache/src/main/resources/org/dcache/pool/classic/pool.xml

@@ -172,8 +172,9 @@
           <bean class="org.dcache.util.CDCListeningExecutorServiceDecorator"
                 destroy-method="shutdown">
               <constructor-arg>
-                  <bean class="java.util.concurrent.Executors"
-                        factory-method="newCachedThreadPool"/>
+                  <bean class="org.dcache.util.BoundedCachedExecutor">
+                      <constructor-arg value="${pool.limits.nearline-threads}"/>
+                  </bean>
               </constructor-arg>
           </bean>
       </property>

skel/share/defaults/pool.properties

@@ -100,6 +100,10 @@ pool.check-health-command=
 # Worker thread pool size. Used by migration module and for pool to pool transfers.
 pool.limits.worker-threads=5
 
+# Nearline storage thread pool size. Used for blocking nearline storage operations,
+# e.g. name space operations or callouts into installed nearline storage providers.
+pool.limits.nearline-threads=30
+
 # Pool cell name. Currently this has to be the same as the pool name.
 pool.cell.name=${pool.name}
 

skel/share/services/pool.batch

@@ -20,6 +20,7 @@ check -strong pool.authn.ocsp-mode
 check pool.authn.ciphers
 
 check -strong pool.limits.worker-threads
+check -strong pool.limits.nearline-threads
 check -strong pool.enable.repository-check
 check -strong pool.enable.remove-precious-files-on-delete
 check -strong pool.plugins.meta