Env vars missing - root cause of 101 issue

[NewtonChutney]

Hello @danielbeach,
I think there are separate temporary environments for each task run.. I've found with bash operator, that each run is in a different temporary folder.. Maybe python operator is similar?

UPDATE:
I noticed you were using a particular temporary directory to store, so this shouldn't be an issue

And when you run on a set of Kubernetes workers, shouldn't the download, chmod, and execution be in the same task on the DAG, so they get executed inside the same worker?

[NewtonChutney]

The issue is caused by the lack of env variables

Had you tried check_output.. it would have been obvious.. 😅

[veloxl]

I was also going to suggest using subprocess.check_output rather than subprocess.check_call here:

subprocess.check_call(f"{temp_dir} {uri}", shell=True)

More info on stackoverflow.

The TL;DR is check_call() starts the external process and returns immediately without waiting for it to exit. check_output() waits for the background python processes to complete and then exits. As @NewtonChutney mentioned, check_output() should also provide the output from the child processes which should make debugging easier.

Also, you might be able to pass env variables to the subprocess command like this:

subprocess.check_output(f"{temp_dir} {uri}", shell=True, env={"KEY": "VALUE"})

I haven't tested it but might be worth a look.

[NewtonChutney]

Yep, we have a similar setup, to inject env variables for bashOperator..
Currently, we're using it to run pytests inside Airflow's worker environment..