Fix the file channel and the test suite on Windows

[ppkarwasz]

On Windows, the file channel cannot restart over an existing checkpoint directory: mapped and open files stay OS-locked, deletions fail with The process cannot access the file because it is being used by another process, and 20+ flume-file-channel tests fail (first observed on the Windows job of #470).

File channel (the main change)

• Resource leaks fixed on every lifecycle path.

  • The BadCheckpointException recovery in Log.replay discarded the old queue and backing store without closing them,
  • a failed FileChannel.start() never released its log,
  • and several constructor error paths leaked file handles or mappings (including the V2 to V3 checkpoint upgrade).

  All of these now close their resources, and EventQueueBackingStoreFile.close() is idempotent with fail-fast guards against use after close.

• The checkpoint MappedByteBuffer is reclaimed by the GC. close() drops every reference to the buffer, so the garbage collector can reclaim it and hence safely unmap the file.
  The few operations that can race the collector pay a small, Windows-only delay: the corrupt-checkpoint recovery retries deletions with GC cycles in between, and tests wait for deletability the same way (TestUtils.awaitDelete). On a first-attempt success, which is the norm everywhere and always the case on Linux, there is no extra cost.

• The queueset MapDB is no longer memory-mapped. MapDB 0.9.x unmaps its buffers by reflecting on sun.nio.ch.DirectBuffer, which fails silently on Java 17, so db.close() left the files locked. The index only lives during replay, so plain file access is fine.

Windows fixes outside the file channel

Enabling windows-latest in CI flushed out one failing module per run:

• flume-ng-node and flume-ng-environment-variable-config-filter: the EnvironmentVariables JUnit rule needs --add-opens java.base/java.lang=ALL-UNNAMED on Windows, where it also patches ProcessEnvironment.theCaseInsensitiveEnvironment.
• flume-tools: TestFileChannelIntegrityTool retries its teardown deletions, so a still-mapped checkpoint no longer leaks files into the next test.
• flume-ng-core: TestLogFile closes its writer before deleting the file, and TestLog fills the disk well below the minimum-space limit, so background churn on a shared runner cannot push the usable space back above it.
• Tests that are inherently POSIX are skipped on Windows: TestExternalProcessConfigFilter (shell scripts), TestFileChannelErrorMetrics (deletes in-use files), and the Taildir tests (unix:ino file attribute). The syslog TLS test fix is copied from Build with the logging-parent reusable workflows #470.

Both ubuntu-latest and windows-latest CI jobs are green. Whichever of this PR and #470 merges second drops the build.yml change (replaced there by build.yaml) and deduplicates the copied commits.

🤖 Generated with Claude Code

[Copilot]

Pull request overview

This PR targets Windows-specific file-handle and memory-mapping retention issues in the Flume file channel, improving restart reliability over existing checkpoint directories and stabilizing Windows CI by tightening resource lifecycle management and test cleanup.

Changes:

• Disable MapDB mmap usage for the transient queueSet index and add explicit MappedByteBuffer unmapping to release Windows file locks promptly.
• Fix several file-channel lifecycle leaks (replay recovery, failed start/stop paths, constructor failure paths) and harden tests to close resources and avoid Windows-incompatible test patterns.
• Update docs/tests and CI to reflect Windows limitations (taildir inode tracking) and validate on windows-latest.

Reviewed changes

Copilot reviewed 23 out of 23 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
src/site/antora/modules/ROOT/pages/FlumeUserGuide.adoc	Clarifies taildir source requires unix file attributes; explicitly notes it won’t work on Windows.
flume-parent/pom.xml	Adds Awaitility version + dependency management for new test synchronization.
flume-ng-sources/flume-taildir-source/src/test/java/org/apache/flume/source/taildir/TestTaildirSource.java	Skips taildir tests when the default FS lacks the unix attribute view.
flume-ng-sources/flume-taildir-source/src/test/java/org/apache/flume/source/taildir/TestTaildirEventReader.java	Same as above for the event reader tests.
flume-ng-sources/flume-syslog-source/src/test/java/org/apache/flume/source/syslog/TestSyslogTcpSource.java	Uses Awaitility + safer socket lifecycle to reduce Windows TLS/RST-related flakiness.
flume-ng-sources/flume-syslog-source/pom.xml	Adds Awaitility test dependency for syslog source tests.
flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestFlumeEventQueue.java	Ensures per-test isolation/cleanup for queue backing store directories and closes leaked resources.
flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestFileChannelRestart.java	Closes streams needed for Windows checkpoint-file deletion during recovery.
flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestFileChannelErrorMetrics.java	Skips deletion-based I/O error injection tests on Windows where open-file deletion is unsupported.
flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestFileChannelBase.java	Always stops the channel to release partially-opened resources after failed starts.
flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestEventQueueBackingStoreFactory.java	Ensures queues are closed to release files before subsequent steps/tests.
flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestCheckpointRebuilder.java	Closes queues after replay to release checkpoint files before channel replay.
flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestCheckpoint.java	Adds cleanup and explicit replay completion/close to avoid queueSet DB locking.
flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/encryption/TestFileChannelEncryption.java	Makes exception-message assertions platform-tolerant.
flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/MappedFiles.java	Introduces explicit unmapping via Unsafe.invokeCleaner to release Windows file locks.
flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Log.java	Closes discarded queues/backing stores during bad-checkpoint recovery; guards queue usage in close.
flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FlumeEventQueue.java	Disables MapDB mmap mode and adds constructor failure cleanup to avoid handle leaks.
flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FileChannel.java	Ensures stop()/close() releases resources even if start() failed partway.
flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/EventQueueBackingStoreFileV3.java	Ensures constructor failures release superclass checkpoint mapping to avoid Windows locks.
flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/EventQueueBackingStoreFileV2.java	Ensures runtime constructor failures close/unmap resources.
flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/EventQueueBackingStoreFile.java	Refactors mapping creation with safe cleanup on failure; adds close idempotence and closed guards.
flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/EventQueueBackingStoreFactory.java	Ensures V2 backing store is closed before V3 remaps the same checkpoint during upgrade.
.github/workflows/build.yml	Adds windows-latest to the workflow matrix to validate on Windows.

Suppressed comments (1)

flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestFlumeEventQueue.java:69

• reset() creates a new temp directory but never cleans up the previous one before overwriting baseDir, so the directory created in the supplier constructor leaks. Also, inflightTakes/inflightPuts are pointing at the opposite filenames (inflightputs vs inflighttakes), which is confusing and inconsistent with the rest of the file-channel tests.

        void reset() {
            baseDir = Files.createTempDir();
            checkpoint = new File(baseDir, "checkpoint");
            inflightTakes = new File(baseDir, "inflightputs");
            inflightPuts = new File(baseDir, "inflighttakes");

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

Pull request overview

Copilot reviewed 31 out of 31 changed files in this pull request and generated no new comments.

Suppressed comments (2)

flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestCheckpoint.java:70

• Only queueOut2 is closed. queueIn and queueOut keep their inflight RandomAccessFiles open, which can prevent cleanup from deleting inflightTakes/inflightPuts on Windows and also leaks file descriptors on all platforms. Close the earlier queues as well once you are done with them.

        queueOut2.close();

flume-ng-sources/flume-syslog-source/src/test/java/org/apache/flume/source/syslog/TestSyslogTcpSource.java:283

• The Awaitility wait has no explicit upper bound. If the event is never accepted (e.g., regression/flaky CI), this can stall the test suite for a long time. Add an explicit atMost timeout to keep failures bounded and easier to diagnose.

            await().until(() -> source.getSourceCounter().getEventAcceptedCount() >= 1);