Tasks submitted to shutting down fixed executors are sometimes neither executed nor rejected

[DaveCTurner]

AFAICT it seems that fixed executors (with bounded queues) sometimes just drop tasks on the floor after shutdown, rather than rejecting them as one might expect. For a demonstration test case, see testFixedBoundedRejectOnShutdown added (albeit muted) in #104581 - this test fails for me whenever it submits a force-executed task.

[elasticsearchmachine]

Pinging @elastic/es-core-infra (Team:Core/Infra)

[DaveCTurner]

Ah I think I see the problem, we're just blindly putting the task into the queue even if nobody is consuming it any more:

elasticsearch/server/src/main/java/org/elasticsearch/common/util/concurrent/EsAbortPolicy.java

Line 25 in aeb2b77

((SizeBlockingQueue<Runnable>) queue).forcePut(r);

We should be doing something similar to how the ForceQueuePolicy handles it:

elasticsearch/server/src/main/java/org/elasticsearch/common/util/concurrent/EsExecutors.java

Lines 384 to 389 in aeb2b77

	put(executor, task);
	// we need to check again the executor state as it might have been concurrently shut down; in this case
	// the executor's workers are shutting down and might have already picked up the task for execution.
	if (executor.isShutdown() && executor.remove(task)) {
	reject(executor, task);
	}