Kill processes in container before copying output files

[PhilippWendler]

Currently, the following happens at the end of a run in container mode:

• Main process of tool is stopped (because it terminates or because it is killed due to a limit violation).
• Wall time and energy are measured.
• Output files are copied.
• Container is killed (this also kills all subprocesses).
• Limits are cancelled.
• Subprocesses are killed via cgroups (has no effect).
• CPU time is read.

This means that if subprocesses change output files after the main process has terminated, the output files can be in an inconsistent state. Furthermore, subprocesses can continue accumulating CPU time while output files are copied.

We should probably kill all subprocesses and potentially do even more cleanup before copying output files. The problem is that the best way to kill the subprocesses is to kill the whole container, but we still need the container for copying output files (cf. 4bba456 and 78a801a). Using cgroups to kill the subprocesses is more inefficient and can be really problematic if the freezer cgroup is not available and a fork bomb runs in the container.

[PhilippWendler]

Another solution would be to open a file descriptor for accessing the directory in the container before starting the tool. As long as we keep this file descriptor open, we should be able to access the container's directories, so we can kill the container (and all subprocesses within it) immediately. This would probably be the cleanest solution, but requires rewriting containerexecutor._transfer_output_files with directory fds and works only on Python >= 3.3.

[PhilippWendler]

This is a lot of implementation effort unless we can make use of glob.iglob with directory fd's, which will land in Python 3.10 (cf. python/cpython#16075). So let's wait.