Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Fix Airflow serialization for namedtuple #37168

Merged
merged 10 commits into from
Feb 18, 2024
Merged

Fix Airflow serialization for namedtuple #37168

merged 10 commits into from
Feb 18, 2024

Commits on Feb 17, 2024

  1. feat: Implement namedtuple serialization 

    namedtuple is serialized like 'builtins.tuple'
    @Joffreybvn
    Joffreybvn committed Feb 17, 2024
    Configuration menu
    Copy the full SHA
    bf3f057 View commit details
    Browse the repository at this point in the history

  2. fix: static-checks

    @Joffreybvn
    Joffreybvn committed Feb 17, 2024
    Configuration menu
    Copy the full SHA
    842b6f3 View commit details
    Browse the repository at this point in the history

  3. fix: check namedtuple based on attributes 

    In the future, if another object very similar to tuple exists, we can also check for `_make` and `_replace` attributes.
    @Joffreybvn
    Joffreybvn committed Feb 17, 2024
    Configuration menu
    Copy the full SHA
    5bae577 View commit details
    Browse the repository at this point in the history

  4. fix: refactor duplicates serialization block 

    The classname of the namedtuple has to be `"builtins.tuple"`, instead of the qualname (the qualname will refer to the name given to the namedtuple when created).

I assume that `_serializers[qn].serialize(o)` won't always return a `classname` == qualname. Thus, creating a new `classname` variable set to "builtins.tuple", to ensure the namedtuple is considered as tuple.
    @Joffreybvn
    Joffreybvn committed Feb 17, 2024
    Configuration menu
    Copy the full SHA
    034fcbd View commit details
    Browse the repository at this point in the history

  5. fix: put custom serializer block after builtin serializers

    @Joffreybvn
    Joffreybvn committed Feb 17, 2024
    Configuration menu
    Copy the full SHA
    0eb7d3c View commit details
    Browse the repository at this point in the history

  6. Update serde.py

    @bolkedebruin @Joffreybvn
    bolkedebruin authored and Joffreybvn committed Feb 17, 2024
    Configuration menu
    Copy the full SHA
    9015e3a View commit details
    Browse the repository at this point in the history

  7. fix: remove classname

    @Joffreybvn
    Joffreybvn committed Feb 17, 2024
    Configuration menu
    Copy the full SHA
    e7be465 View commit details
    Browse the repository at this point in the history

  8. Revert "fix: remove classname" 

    This reverts commit e7be465.
    @Joffreybvn
    Joffreybvn committed Feb 17, 2024
    Configuration menu
    Copy the full SHA
    84894ba View commit details
    Browse the repository at this point in the history

  9. Revert "Update serde.py" 

    This reverts commit 9015e3a.
    @Joffreybvn
    Joffreybvn committed Feb 17, 2024
    Configuration menu
    Copy the full SHA
    8cb701f View commit details
    Browse the repository at this point in the history

  10. fix: Add comment on classname overriding 

    Note: we do need to specify the classname, which should be "builtins.tuple":

The serialize method (in airflow/serialization/serializers/builtin.py) does qualname() on the namedtuple, which returns "airflow.providers.databricks.hooks.Row" (the namedtuple dynamically created in the hook). If this is used as classname, it will fail to deserialize: there won't be any deserializer for it.

If we hardcode it to "builtins.tuple", it gets deserialized correctly.
    @Joffreybvn
    Joffreybvn committed Feb 17, 2024
    Configuration menu
    Copy the full SHA
    807845a View commit details
    Browse the repository at this point in the history