Address concurrency issues in DequeRecycler close

[jaymode]

This change addresses some concurrency issues that can occur when
closing a DequeRecycler. The first issue that is addressed is a
ConcurrentModificationException that can occur when using a locked
DequeRecycler that is backed by an ArrayDeque. The ArrayDeque is not
thread safe and requires external synchronization. In most cases the
locked DequeRecycler handles this correctly but closing the
DequeRecycler is not protected by the lock, which can lead to the
ConcurrentModificationException if other threads are calling the obtain
method. Additionally, the DequeRecycler close method used an iterator
to go over all entries in the Deque and then later cleared them. The
close method now uses pollFirst in a loop to empty the Deque and still
execute the destroy method.

Finally, the ConcurrentDequeRecycler had an overzealous assertion in
the close method that the size of the Deque is equivalent to the
externally tracked size. This assertion is not always going to be true
due to the nature of the implementation. There is no lock guarding both
the deque and the size value, so there is always a chance that the two
could be wrong depending on ongoing requests. This assertion has been
removed and a comment has been added that mentions there can be some
discrepancies between the actual size of the deque and the externally
tracked size.

These issues may have always been present, but I believe the changes in
#39317 has made their presence known.

Closes #41683

[elasticmachine]

Pinging @elastic/es-core-infra

[jpountz]

Thanks @jaymode for looking into it. I think #39317 introduced problems indeed by closing the recycler before making sure that there are no more ongoing tasks in the threadpool. I tried to think about how we could make sure that close is only called after all tasks have run but I'm now wondering whether it would be simpler to just remove close() from Recycler and PageCacheRecycler, and removing the call to PageCacheRecycler#close from Node#close? The only things that these close() calls are releasing is memory, which would be reclaimed anyway by the garbage collector since there is no point in keeping references to a closed node.

[jaymode]

@jpountz that makes sense to me. I'll close this PR and open another to remove the close method