Repository navigation
Jinja templating for messages in PubSubPublishOperator #22303
|
Hi, I would like to use Jinja templating for the messages sent by the I use the following DAG as a minimal example: from airflow import DAG
from datetime import datetime
from airflow.contrib.operators.kubernetes_pod_operator import KubernetesPodOperator
from airflow.contrib.operators.pubsub_operator import PubSubPublishOperator
from base64 import b64encode as b64e
import json
with DAG(
dag_id = 'pubsub-message-jinja',
schedule_interval = "* * * * *",
default_args = {
'owner': 'admin',
'start_date': datetime.strptime('2022-01-01', '%Y-%m-%d'),
},
catchup = False,
) as dag:
message_raw = { "data": "{{ ds }}" }
message_encoded = b64e(json.dumps(message_raw).encode("utf-8"))
pubsub_publisher = PubSubPublishOperator(
gcp_conn_id="dummy_gcp_conn_id",
task_id="pubsub-notification",
project="dummy_gcp_project_id",
topic="dummy_gcp_topic_name",
messages=[
{'data': message_encoded },
]
)
pubsub_publisherThe message is successfully sent to the topic. However the Jinja templating did not occur and the message received is According to the PubSubPublishOperator docs, the field Is there something I am missing ? Regards, |
Replies: 4 comments 4 replies
|
The issue that you are executing top-level code on a templated string so If you need enforce the encoding on the messages, you can try defining a
Another user wrote their own |
|
@josh-fell Thank you for your helpful answer! I got the example working with a |
|
I thought allowing string-type message data would make the operator easier to use, so I submitted a PR. |
|
Below is the my code i am unable to publish message in composer 2.0 from airflow import DAG #from airflow.contrib.operators.dataproc_operator import DataProcSparkOperator default_args = { DAG_ID='%s_PUBSUB_TEST'%REGISTRY dag = DAG( #success_pubsub_msg=[{"data":"{{ task_instance.xcom_pull('get_pubsub_success_mesg', key='message')}}"}] get_pubsub_success_mesg = PythonOperator( PubSubPublishSuccess = PubSubPublishMessageOperator(task_id='publish-messages_success', get_pubsub_success_mesg >> PubSubPublishSuccess |
The issue that you are executing top-level code on a templated string so
b64e(json.dumps(message_raw).encode("utf-8"))evaluates the literal string "{{ ds }}" rather than the rendered value that you are expecting. Templated strings are not rendered/evaluated until the task begins to execute.If you need enforce the encoding on the messages, you can try defining a
user_defined_macrosoruser_defined_filtersto use with the Jinja templating.