[FLINK-26957][runtime] Removes flush in FileSystemJobResultStore.createDirtyResultInternal

[XComp]

The writeValue calls close by default internally. Calling flush afterwards
could cause errors. It's also not really necessary. OutputStream.flush does
not guarantee persistence according to its JavaDoc. In contrast, calling
close does guarantee it.

What is the purpose of the change

Some FileSystem implementations fail when calling flush on a closed OutputStream (e.g. HDFS).
To cover this behavior in the tests, we added this check to the LocalDataOutputStream.

Brief change log

• Removes the flush call: The data is persisted as part of the OutputStream.close call as stated in the JavaDoc
• Added asserts to LocalDataOutputStream to cover this behavior in tests.

Verifying this change

• Adds unit tests to LocalFileSystemTest to check for the assertions
• Run CI to see whether other test failures pop up

Does this pull request potentially affect one of the following parts:

• Dependencies (does it add or upgrade a dependency): no
• The public API, i.e., is any changed class annotated with @Public(Evolving): no
• The serializers: no
• The runtime per-record code paths (performance sensitive): no
• Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: no
• The S3 file system connector: no

Documentation

• Does this pull request introduce a new feature? no
• If yes, how is the feature documented? not applicable

[flinkbot]

CI report:

• ac81de7 Azure: SUCCESS

Bot commands

The @flinkbot bot supports the following commands:

• @flinkbot run azure re-run the last Azure build

[XComp]

@flinkbot run azure

[XComp]

I forgot to change the unit tests expected error from IllegalStateException to IOException which caused a failure. -.- ...that's why the force-push

[XComp]

I added a test case for closing the OutputStream twice in the most-recent force-push.

[dmvk]

Hi @XComp, this already looks really good. Great job! <3 I've added few comments and questions, please take a look.

[XComp]

CI looks good - I created a 1.15 backport already to make CI pass (hoping that there are no additional changes required)

[dmvk]

LGTM, great job! 👍 Can you please fix the commit history before merging?

[XComp]

I'm gonna merge. I only dropped on commit.