Creating a Subclassed instance now errors with got an unexpected keyword argument

[ashb]

We just noticed this in Airflow CI on upgrading to 25.2.0.

The error is

ERROR    airflow.models.dagbag.DagBag:dagbag.py:394 Failed to import: /opt/airflow/airflow/example_dags/example_dynamic_task_mapping.py
Traceback (most recent call last):
  File "/opt/airflow/airflow/models/dagbag.py", line 384, in parse
    loader.exec_module(new_module)
  File "<frozen importlib._bootstrap_external>", line 999, in exec_module
  File "<frozen importlib._bootstrap>", line 488, in _call_with_frames_removed
  File "/opt/airflow/airflow/example_dags/example_dynamic_task_mapping.py", line 27, in <module>
    with DAG(dag_id="example_dynamic_task_mapping", schedule=None, start_date=datetime(2022, 3, 4)) as dag:
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
TypeError: DAG.__init__() got an unexpected keyword argument 'dag_id'

(Yes, I know your view on subclassing. I don't disagree either)

Minimal repro case:

import attrs

def _all_after_dag_id_to_kw_only(cls, fields: list[attrs.Attribute]):
    i = iter(fields)
    f = next(i)
    if f.name != "dag_id":
        raise RuntimeError("dag_id was not the first field")
    yield f

    for f in i:
        yield f.evolve(kw_only=True)


@attrs.define(field_transformer=_all_after_dag_id_to_kw_only)
class Base:
    dag_id: str
    has_default: str = "default"


@attrs.define()
class Subclass(Base):
    other: bool


x = Subclass(dag_id="foo", other=False)

print(repr(x))
print(attrs.__version__)

[kaxil]

#1401 might have caused it

[ashb]

Might do, yes, but the minimal example is not failing, so it's some quirk of what we are doing in Airflow

[ashb]

cc @jamesmurphy-mc

[hynek]

pls provide a repro and I'll see if I can do anything before leaving for my vacation tomorrow afternoon

[ashb]

I updated the example to include help(Subclass.__init__);

On 25.1.0 we see this:

__init__(self, dag_id: str, other: bool, *, has_default: str = 'default') -> None
    Method generated by attrs for class Subclass.

On 25.2.0 we see this:

__init__(self, other: bool) -> None
    Method generated by attrs for class Subclass.

[ashb]

@hynek No rush at all!, we are happy to pin to !latest for now

[tirkarthi]

I have got a simple reproducer. Let me try that.

[hynek]

cc @sscherfke for good measure

[ashb]

@tirkarthi I've got a reproducer now. Top post has been updated.

[tirkarthi]

@ashb Thanks, I got the same one but didn't to subclass.

import attrs

def _all_after_dag_id_to_kw_only(cls, fields: list[attrs.Attribute]):
    i = iter(fields)
    f = next(i)
    if f.name != "dag_id":
        raise RuntimeError("dag_id was not the first field")
    yield f

    for f in i:
        yield f.evolve(kw_only=True)


@attrs.define(repr=False, field_transformer=_all_after_dag_id_to_kw_only, slots=False)
class DAG:
    dag_id: str = attrs.field(kw_only=False, validator=attrs.validators.instance_of(str))


dag = DAG(dag_id="test")
print(dag)

(.venv) ➜  airflow git:(main) ✗ python /tmp/attrs_repro.py
<__main__.DAG object at 0x7f69c5305bd0>
(.venv) ➜  airflow git:(main) ✗ pip install attrs==25.2.0
Collecting attrs==25.2.0
  Downloading attrs-25.2.0-py3-none-any.whl.metadata (11 kB)
Downloading attrs-25.2.0-py3-none-any.whl (64 kB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 64.0/64.0 kB 1.5 MB/s eta 0:00:00
Installing collected packages: attrs
  Attempting uninstall: attrs
    Found existing installation: attrs 25.1.0
    Uninstalling attrs-25.1.0:
      Successfully uninstalled attrs-25.1.0
ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
ydb 3.18.15 requires protobuf<5.0.0,>=3.13.0, but you have protobuf 5.29.3 which is incompatible.
Successfully installed attrs-25.2.0

[notice] A new release of pip is available: 24.0 -> 25.0.1
[notice] To update, run: pip install --upgrade pip
(.venv) ➜  airflow git:(main) ✗ python /tmp/attrs_repro.py 
Traceback (most recent call last):
  File "/tmp/attrs_repro.py", line 19, in <module>
    dag = DAG(dag_id="test")
          ^^^^^^^^^^^^^^^^^^
TypeError: DAG.__init__() got an unexpected keyword argument 'dag_id'

[ashb]

Walking up the stack to where the field transformer is called, I notice that base_attrs is an empty list, even though collect_mro is true.
Debugging continues!

[tirkarthi]

print(inspect.signature(DAG))

On 24.2.0 : (dag_id: str) -> None
On 25.2.0 : () -> None

[ashb]

Oh, this might be part of the problem, Base.attrs_attrs is empty when a field transformer is used:

import attrs

print(f"{attrs.__version__=}")
def _all_after_dag_id_to_kw_only(cls, fields: list[attrs.Attribute]):
    i = iter(fields)
    f = next(i)
    if f.name != "dag_id":
        raise RuntimeError("dag_id was not the first field")
    yield f

    for f in i:
        yield f.evolve(kw_only=True)


@attrs.define(field_transformer=_all_after_dag_id_to_kw_only)
class Base:
    dag_id: str
    has_default: str = "default"

print(attrs.fields(Base))

Prints:

attrs.__version__='25.2.0'
()

[ashb]

Ahhh, got it.

The issue is that the field transformer is defined as a generator, so it's exhausted once looking for the a.init and a.kw_only args, then it's empty in the uses elsewhere in _transform_attr

[hynek]

Ahhh, got it.

🎉

The issue is that the field transformer is defined as a generator, so it's exhausted once looking for the a.init and a.kw_only args, then it's empty in the uses elsewhere in _transform_attr

Without being able to look at the code: sounds like we should be able to convert it to a tuple on first use? Quick PRs with tests welcome. ;)

[ashb]

Yup, that was exactly my plan, inspect.isgenerator() + tuple.

[hynek]

25.3.0 is live and I am gone