SSHRemoteJobOperator: inconsistent "remote_base_dir" validation

[AchimGaedkeLynker]

Under which category would you file this issue?

Providers

Apache Airflow version

3.3

What happened and how to reproduce it?

(On a system I can't report from) I use SSHRemoteJobOperator with remote_base_dir="/tmp-data/airflow-ssh-jobs".

The operator starts and tracks the remote job till the end - as it has before the update of the provider.

But instead of finishing cleanly (as the remote command did after an hour...), the operator fails while cleaning up the job.
It raises a Value Error with the description:

Invalid job directory: '/tmp-data/airflow-ssh-jobs/af_ ....'. Expected Path under '/tmp/airflow-ssh-jobs' for safety.

Removing the remote_base_dir argument avoids this failure.

What you think should happen instead?

Either the remote_base_dir should be rejected when instantiating the operator or be accepted at the very end.

Looking at the code,

airflow/providers/ssh/src/airflow/providers/ssh/operators/ssh_remote_job.py

Line 167 in 7f60646

def _validate_base_dir(path: str) -> None:

seems to have different validation criteria than

airflow/providers/ssh/src/airflow/providers/ssh/utils/remote_job.py

Line 33 in 7f60646

def _validate_job_dir(job_dir: str, remote_os: Literal["posix", "windows"]) -> None:

is very restrictive as it only allows paths beginning with the default path.

This is the validation used inside the cleanup

airflow/providers/ssh/src/airflow/providers/ssh/operators/ssh_remote_job.py

Line 455 in 7f60646

cleanup_cmd = build_posix_cleanup_command(job_dir)

... i.e fails at the very end.

Operating System

Ubuntu 24.04

Deployment

Virtualenv installation

Apache Airflow Provider(s)

ssh

Versions of Apache Airflow Providers

python 3.12
apache-airflow-providers-ssh==5.0.3

Official Helm Chart version

Not Applicable

Kubernetes Version

No response

Helm Chart configuration

No response

Docker Image customizations

No response

Anything else?

Just occurred when updating from airflow 3.2.x to 3.3.0, incl apache-airflow-providers-ssh 5.0.3 ->5.0.2

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct