Bugfix/prevent concurrency with cached venv

[jscheffl]

This PR reverts PR #35252 as the fix made in this PR just fixed one side effect.

Unfortunately when initially the PR #33355 was created I did not see that all temporary scripts and in/out serialized values are written into the tmp folder of the venv and thus pollute the venv. As the vench in the cached case might be used by multiple task slots in parallel, temporary files in a cached venc really generate a concurrency issue - more than just a termination log.

This PR:

• Revers the efforts of @potiuk in Handle left-over termination logs from previous runs with same cache #35252 - Sorry as termination log is now not kept but in a separate tmp folder I needed to remove the newly contributed pytest :-(
• Instead each execution of external python/venv is not generating a dedicated temp folder for call data
• The generation and use of the tmp path is pushed into the execution to reduce redundant code.

If there is a general demand to keep the termination log after a failed run, we need to find a mechanism on top of the tempfolder so that it is not deleted as cleanup.

Note, the DIFF is much smaller if you ignore the whitespace.

[potiuk]

Yeah. Looks good. Let me also rebase my #35160 - the new improved test harness runs all the Venv tests using caching so it has a chance to detect similar problems.

UPDATE: https://github.com/apache/airflow/actions/runs/6686368999 is the run of the new test harness rebased on top of this one :).

[potiuk]

• Revers the efforts of @potiuk in Handle left-over termination logs from previous runs with same cache #35252 - Sorry as termination log is now not kept but in a separate tmp folder I needed to remove the newly contributed pytest :-(

Yep. Not needed any more, agree.

[potiuk]

BTW. @jens-scheffler-bosch - first test suites are getting green in #35160, so I confirm it fixes the original problem too.

[utkarsharma2]

@jens-scheffler-bosch Just curious, what other mechanish we have in place to debug the failed run?

[jscheffl]

@jens-scheffler-bosch Just curious, what other mechanish we have in place to debug the failed run?

Same here. I assume we have a general contratiction / conflict between "ensure that all mess is cleaned up so that no temp traces are left behind so that worker is not running out of disk space after some time" and "I want to debug, keep all the logs please". I think the best is first hand to clean up so that worker can run stable over a long time, otherwise if we miss details of traces we need to add (e.g. by cat to stdout) to put all problems into airflow task stdout so that the (whateverr logging facility of the worker) archives the logs.
What I see is that as long as an error code is not a trigger of s skip of the result, the termination log already today is added as text to the AirflowException which is raised if error code is returned. So I assume in current code it is not too bad.

[potiuk]

hat other mechanish we have in place to debug the failed run?

Yeah - the log is the way to pass it to the place it will be visible (airflow log). And it's not much different from what we have for any other operator - we basically only see the logs + exception thrown.