[FLINK-21164][rest] Delete temporary jars

[zentol]

Modifies the Jar(Run|Plan)Handler to call PackagedProgram#deleteExtractedLibraries once we are done with the job submission / plan generation.

This is not covered by tests; I don't see a good way to do that since from the outside it is neither obvious which files are supposed to (not) exist nor where the temporary files would be located (by virtue of PackagedProgram not respecting the io.tmp.dirs option).

[flinkbot]

Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community
to review your pull request. We will use this comment to track the progress of the review.

Automated Checks

Last check on commit c833d99 (Fri May 28 08:15:03 UTC 2021)

Warnings:

• No documentation files were touched! Remember to keep the Flink docs up to date!

_{Mention the bot in a comment to re-run the automated checks.}

Review Progress

• ❓ 1. The [description] looks good.
• ❓ 2. There is [consensus] that the contribution should go into to Flink.
• ❓ 3. Needs [attention] from.
• ❓ 4. The change fits into the overall [architecture].
• ❓ 5. Overall code [quality] is good.

Please see the Pull Request Review Guide for a full explanation of the review process.

The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required

Bot commands

The @flinkbot bot supports the following commands:

• @flinkbot approve description to approve one or more aspects (aspects: description, consensus, architecture and quality)
• @flinkbot approve all to approve all aspects
• @flinkbot approve-until architecture to approve everything until architecture
• @flinkbot attention @username1 [@username2 ..] to require somebody's attention
• @flinkbot disapprove architecture to remove an approval you gave earlier

[flinkbot]

CI report:

• ce3c983 UNKNOWN
• c833d99 Azure: SUCCESS

Bot commands

The @flinkbot bot supports the following commands:

• @flinkbot run travis re-run the last Travis build
• @flinkbot run azure re-run the last Azure build

[tillrohrmann]

Thanks for creating this PR @zentol. The change itself looks good. I had a couple of comments concerning other places where the PackagedProgram is being used.

[tillrohrmann]

Isn't JarListHandler also susceptible to the problem?

[tillrohrmann]

I am also wondering whether the ApplicationClusterEntryPoint really cleans up a PackagedProgram after using. Maybe this is something to investigate as a follow up.

[zentol]

Yes, the JarListHandler has the same problem. I did not know it actually creates a PackagedProgram.

As for the ApplicationClusterEntryPoint, the files should be cleaned up when the entry point shuts down because we call File#deleteOnExit in PackagedProgram#createTempFile.

[zentol]

Now whether they are actually deleted depends a bit on when the JVM executes the delete calls. If the ClassLoader is still around at that time, then the deletion may in fact fail.

Explicitly closing the ClassLoader in the PackagedProgram (which unfortunately leaks into the JobGraph...) is a separate issue (see FLINK-9844).

[tillrohrmann]

Alrighty. Thanks for the update.

[zentol]

@tillrohrmann I updated the PR. PackagedProgram now implements AutoClosable, and all usages of PackagedProgram#deleteExtractedLibraries have been migrated to use close() instead.
In FLINK-9844 we can then extend this method to also close the ClassLoader.

I will also file a follow-up ticket to explicitly call close() in the entry points.

[tillrohrmann]

Thanks for the update @zentol. LGTM. +1 for merging.