Backport PR #15233 to 8.9: Fix DeadLetterQueueWriter unable to finalize segment error

[github-actions]

Backport PR #15233 to 8.9 branch, original message:

---

Release notes

Fixed DeadLetterQueueWriter unable to finalize segment - java.nio.file.NoSuchFileException

What does this PR do?

For more details about the error, please check the #15227.

This PR moves the Files.size(...) call into the try catch block, that way, when the oldest segment is deleted by the DeadLetterQueueReader, no NoSuchFileException will be thrown up, and the writter will gracefully update the oldest segment on the next updateOldestSegmentReference invocation (scheduled flush, entry write, delete expired, etc).

It also adds the volatile keyword to the shared mutable variables, making sure that all the changes will be instantly visible among all the running threads (scheduler & writer).

Why is it important/What is the impact to the user?

Checklist

• My code follows the style guidelines of this project
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• I have made corresponding change to the default configuration files (and/or docker env variables)
• I have added tests that prove my fix is effective or that my feature works

How to test this PR locally

• Set up the example pipeline posted here: [DeadLetterQueueWriter] Unable to finalize segment - java.nio.file.NoSuchFileException #15227
• Change the DeadLetterQueueWriter#createFlushScheduler to run more oftetn, it's not mandatory but will make the problem happen much faster:

- flushScheduler.scheduleAtFixedRate(this::scheduledFlushCheck, 1L, 1L, TimeUnit.SECONDS);
+ flushScheduler.scheduleAtFixedRate(this::scheduledFlushCheck, 1L, 1L, TimeUnit.MILLISECONDS);

• Running with the main branch code, it should eventually throw a few java.nio.file.NoSuchFileException.
• Running with this PR changes, no errors should be raised.

Related issues

• Closes [DeadLetterQueueWriter] Unable to finalize segment - java.nio.file.NoSuchFileException #15227
• Closes Intermittent logstash pod pipeline stops with [DeadLetterQueueWriter] unable to finalize segment #15078

[jsvd]

LGTM