How can i get parameter from PapermillOperator into my notebook (.ipynb file)

[marcelo225]

Hi, everyone!

I've got the following DAG code:

notebook_input_path = os.environ['AIRFLOW_HOME'] + '/dags/notebooks/'
notebook_output_path = os.environ['AIRFLOW_HOME'] + '/dags/outputs/'

# My current DAG
with DAG(
    'my_dag',
    default_args={
        'owner': 'something',
        'start_date': days_ago(1),
        'depends_on_past': False,
    },
    concurrency=4,
    max_active_runs=1,
    schedule_interval=None,
) as dag:

test = PapermillOperator(
      task_id="etl_test",
      input_nb=notebook_input_path + "test/notebook.ipynb",
      output_nb=notebook_output_path + "notebook_output-{{ execution_date }}.ipynb",
      parameters={"id": "{{ dag_run.conf['id'] }}"},
      dag=dag,
 )

# When I call the DAG
$ airflow dags trigger --conf '{"id": 123}' my_dag

Question: How can I get the id parameter from PapermillOperator into my .ipynb file?

Thanks!

[potiuk]

I've never used the operator, but looking at the code, parameters are passed to "papermils' parameters and they are just passed as parameters https://github.com/nteract/papermill#executing-a-notebook and you can just "use" them in your notebook as variables.

But this is just by looking at the code, so there might be other things in place.

[marcelo225]

Thanks, @potiuk .
I've got another question though.

About the above code at the line parameters at PapermillOperator, I can't pass a parameter, using dag_run.conf['id']
I can't find out anything about how to pass parameters like this. It's at Airflow Documentation, but It doesn't show up more details about this.
Link: https://airflow.apache.org/docs/apache-airflow/stable/dag-run.html#passing-parameters-when-triggering-dags

I found this providers documenttion about PapermillOperator, but it doesn't explain how can pass parameters by dag_run.conf as well.

Could you explain how to do it?
Thanks!

[potiuk]

I think you should look for JINJA in our docs. The docs are as clear as I can explain them here. And look through examples in our code.