GoogleDisplayVideo360CreateSDFDownloadTaskOperator does not push necessary xcom #14073
Description
Apache Airflow version: 1.10.12
Kubernetes version (if you are using kubernetes) (use kubectl version):
Environment: Google Cloud Composer 1.13.3
- Cloud provider or hardware configuration:
- Google Cloud Composer
What happened:
The SDF operator workflow does not work correctly. When you create a new SDF operation using GoogleDisplayVideo360CreateSDFDownloadTaskOperator it successfully creates the new task. The next step in the process is to use the GoogleDisplayVideo360SDFtoGCSOperator to download the SDF file that was created. This second task relies on the "name" of the SDF operation that was created. The second task tries to use xcom to pull the name. However, the first task does not push the name to the xcom. Thus, the GoogleDisplayVideo360SDFtoGCSOperator always fails because it cannot download an operation with name 'None'.
The error is: ERROR - Parameter "name" value "None" does not match the pattern "^sdfdownloadtasks/operations/[^/]+$"
What you expected to happen: The GoogleDisplayVideo360SDFtoGCSOperator should be able to pull the name from the xcom and successfully download the file.
How to reproduce it:
Run any workflow trying to create and download an SDF, like this example dag. The GoogleDisplayVideo360SDFtoGCSOperator will fail every time unless you hardcode the name of a valid SDF operation.
Anything else we need to know:
I have written a working patch for the issue and will submit it shortly.