fix(ssh): validate job dir cleanup against configured remote_base_dir - #70802
fix(ssh): validate job dir cleanup against configured remote_base_dir#70802mmaxjr wants to merge 1 commit into
Conversation
SSHRemoteJobOperator._validate_base_dir() lets a user set a custom remote_base_dir, and RemoteJobPaths correctly builds job_dir under it. But at cleanup time, _validate_job_dir() always checked job_dir against the hardcoded default base directory instead of the one actually configured, so cleanup raised "Invalid job directory ... Expected path under '/tmp/airflow-ssh-jobs'" for any job that used a custom remote_base_dir - execution and monitoring worked fine, only cleanup broke. _validate_job_dir(), build_posix_cleanup_command(), and build_windows_cleanup_command() now take an optional base_dir, and the operator passes its own self.remote_base_dir through at cleanup time. Defaults to the previous hardcoded behavior when base_dir is omitted, so other callers are unaffected. Fixes apache#69813 Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
|
Congratulations on your first Pull Request and welcome to the Apache Airflow community! If you have any issues or are unsure about any anything please check our Contributors' Guide
|
|
Thanks — this is a close call, and sorry to close it. Three PRs independently fixed this bug; I am going with #69885, open since 14 July. Yours is good work and better in two respects: the docstring actually explains why a caller with a custom base dir must pass it, and you linked the originating issue #69813. #69885 has the edge on coverage though — it pins the rejection case on Windows as well as POSIX, whereas this covers the negative path for POSIX only. On a validation guard the refusals are the part most worth locking down. Also worth knowing: #70091 has since rewritten these same command builders on I would rather not lose your docstring — if you would like to carry it and the issue link over as a follow-up once #69885 lands, that would genuinely improve the result. Drafted-by: Claude Code (Opus 5); reviewed by @potiuk before posting |
Fixes #69813
Problem
SSHRemoteJobOperatorlets users set a customremote_base_dir, and it'scorrectly validated for safety (
_validate_base_dir) and used to buildjob_dirviaRemoteJobPaths. Execution and monitoring both work finewith it.
But
_cleanup_remote_jobcallsbuild_posix_cleanup_command/build_windows_cleanup_command, which always validatedjob_diragainstthe hardcoded default base directory (
_validate_job_dir), not the oneactually configured. So cleanup for any job using a custom
remote_base_diralways raised: