Airflow Python Client 2.6.0rc2 fails with read-only dag_id attribute on post_dag_run

[potiuk]

When running get_tasks API calls, the Python cllient (when used with Airflow 2.6.0) fails with:

Exception when calling DAGRunAPI->post_dag_run: `dag_id` is a read-only attribute. Use `from_openapi_data` to instantiate class with read only attributes.

Apparently because it expects not Nuill execution timeout,.

How to reproduce:

1. breeze start-airflow --db-reset --use-airflow-version 2.6.0 --load-example-dags --load-default-connections

2. In the terminal lnstall 2.6.0rc1 of python-client: pip install apache-airflow-client==2.6.0.rc2

3. Ctrl-C webserver

4. Change configuration in ~/airflow/airflow.cfg (with vim for example)

• set [webserver] -> expose_config = True
• enable basic authentication by adding ,airflow.api.auth.backend.basic_auth to [api] -> auth_backends

5. run airflow webserver again

6. Copy the following scripts to the container (for example to files folder to be able to copy it in the host) and name it test_python_client.py:

import airflow_client.client
from pprint import pprint
from airflow_client.client.api import config_api, dag_api, dag_run_api
from airflow_client.client.model.dag_run import DAGRun

configuration = airflow_client.client.Configuration(
    host="http://localhost:8080/api/v1",
    username='admin',
    password='admin'
)

dag_id = "example_bash_operator"

# Enter a context with an instance of the API client
with airflow_client.client.ApiClient(configuration) as api_client:
    # Get current configuration
    conf_api_instance = config_api.ConfigApi(api_client)
    try:
        api_response = conf_api_instance.get_config()
        pprint(api_response)
    except airflow_client.client.OpenApiException as e:
        print("Exception when calling ConfigApi->get_config: %s\n" % e)


    # Get dag list
    dag_api_instance = dag_api.DAGApi(api_client)
    try:
        api_response = dag_api_instance.get_dags()
        pprint(api_response)
    except airflow_client.client.OpenApiException as e:
        print("Exception when calling DagAPI->get_dags: %s\n" % e)

    print("Caling get tasks")
    # Get tasks for a DAG (TODO: issue#20)
    try:
        api_response = dag_api_instance.get_tasks(dag_id)
        pprint(api_response)
    except airflow_client.client.exceptions.OpenApiException as e:
        print("Exception when calling DagAPI->get_tasks: %s\n" % e)


    print("Caling post dag run")
    # Trigger a dag run (TODO: issue#21)
    dag_run_api_instance = dag_run_api.DAGRunApi(api_client)
    try:
        # Create a DAGRun object
        dag_run = DAGRun(
            dag_run_id='some_test_run',
            dag_id=dag_id,
            external_trigger=True,
        )
        api_response = dag_run_api_instance.post_dag_run(dag_id, dag_run)
        pprint(api_response)
    except airflow_client.client.exceptions.OpenApiException as e:
        print("Exception when calling DAGRunAPI->post_dag_run: %s\n" % e)

7. Run the script

Result: post_dag_run will fail with this error without even attempting to call the API.

Caling post dag run
Exception when calling DAGRunAPI->post_dag_run: `dag_id` is a read-only attribute. Use `from_openapi_data` to instantiate class with read only attributes.

[potiuk]

Related to #74 (both are exposed by the same test script).

[pierrejeambrun]

I don't think this is a mistake. DagRun Schema has dump_only dag_id (which seems normal). The dag_id required for triggering a Dag Run is a path parameter, this is working on my end:

    # DagRun
    dag_run_api_instance = dag_run_api.DAGRunApi(api_client)
    dag_id = "dynamic_task_group"
    try:
        api_response = dag_run_api_instance.post_dag_run(
            dag_id,
            dag_run_api.DAGRun(
                dag_run_id="run_from_python_api_client",
            ),
        )
        pprint(api_response)
    except airflow_client.client.OpenApiException as e:
        print("Exception when calling DagRunApi->post_dag_run: %s\n" % e)

Note: Same problem for external_trigger, dump_only. Already defaults to True in the model and the applicative code, so REST API call will end up creating dagrun with extrenal_trigger to True, you cannot set this to False for instance while triggering from the REST API. Which makes sense, those are external triggers.

[potiuk]

I don't think this is a mistake. DagRun Schema has dump_only dag_id (which seems normal).

I am fine with that.

But I think that raises question - should we be adding an explanation about it in release notes ?

the problem with this is that it is not some artifficial example :).

It did work before (I used that script in the past to test it, and well it worked (I think the problem is that the new OpenAPI is more picky when it comes to validating stuff and it allows to do stuff like this :) (which I understand it should not be done):

        dag_run = DAGRun(
            dag_run_id='some_test_run',
            dag_id=dag_id,
            external_trigger=True,
        )

And the thing is - the example comes from our repo:

https://github.com/apache/airflow-client-python/blob/main/dev/test_python_client.py

And this actually was added as a response to another issue #21 where dag triggering did not work at all :).

So we are facing high likelihood we are going to break workflows of the people whose problem we solved before and even gave them some examples on how to do stuff :)

Actually I am quite fine that we say now "you are doing it wrong" - I think that would be cleanest solution, but likely maybe we should explain it in release notes how to use DAG RUN triggering now at the very least ?

[potiuk]

What do others think?

[potiuk]

cc: @msumit @ephraimbuddy @mik-laj ^^ -> I think we will need your opinion on that one too :).. just removing dag_id is not a huge breaking change, so I would be fine if we just explain it, personallly and explain this was really a buggy behaviour.

[potiuk]

PR how it could look like here: #78

It also refreshes the infrastructure/examples a bit.

[ephraimbuddy]

Nice catch @pierrejeambrun.

Since it worked before, mentioning it in #78 makes sense to me