Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Soft delete datasets that are no longer referenced in DAG schedules or task outlets #27828

Merged
merged 26 commits into from
Nov 25, 2022
Merged

Soft delete datasets that are no longer referenced in DAG schedules or task outlets #27828

merged 26 commits into from
Nov 25, 2022

Commits on Nov 23, 2022

  1. Soft delete datasets that are no longer referenced anywhere

    @blag
    blag committed Nov 23, 2022
    Configuration menu
    Copy the full SHA
    fb8d644 View commit details
    Browse the repository at this point in the history

  2. Move the orphaning logic into the scheduler and adjust config option … 

    …name accordingly
    @blag
    blag committed Nov 23, 2022
    Configuration menu
    Copy the full SHA
    67139d6 View commit details
    Browse the repository at this point in the history

Commits on Nov 25, 2022

  1. Rename config option scheduler.cleanup_interval -> scheduler.parsing_… 

    …cleanup_interval
    @blag
    blag committed Nov 25, 2022
    Configuration menu
    Copy the full SHA
    99cbdf3 View commit details
    Browse the repository at this point in the history

  2. Include default column value in migration

    @blag
    blag committed Nov 25, 2022
    Configuration menu
    Copy the full SHA
    7efb532 View commit details
    Browse the repository at this point in the history

  3. deprecate old interval config; move code to scheduler_job

    @jedcunningham
    jedcunningham committed Nov 25, 2022
    Configuration menu
    Copy the full SHA
    29f3d0a View commit details
    Browse the repository at this point in the history

  4. First pass at a test

    @jedcunningham
    jedcunningham committed Nov 25, 2022
    Configuration menu
    Copy the full SHA
    1950f15 View commit details
    Browse the repository at this point in the history

  5. Fix migration

    @jedcunningham
    jedcunningham committed Nov 25, 2022
    Configuration menu
    Copy the full SHA
    594e899 View commit details
    Browse the repository at this point in the history

  6. Apply suggestions from code review 

    Co-authored-by: Jed Cunningham <66968678+jedcunningham@users.noreply.github.com>
    @blag @jedcunningham
    blag and jedcunningham committed Nov 25, 2022
    Configuration menu
    Copy the full SHA
    2cb2d99 View commit details
    Browse the repository at this point in the history

  7. Don't batch migrations if we don't need to

    @blag
    blag committed Nov 25, 2022
    Configuration menu
    Copy the full SHA
    652f745 View commit details
    Browse the repository at this point in the history

  8. Revert "Don't batch migrations if we don't need to" - gotta batch mig… 

    …rations for SQLite

This reverts commit 652f745.
    @blag
    blag committed Nov 25, 2022
    Configuration menu
    Copy the full SHA
    e47c498 View commit details
    Browse the repository at this point in the history

  9. Tweak migrations

    @blag
    blag committed Nov 25, 2022
    Configuration menu
    Copy the full SHA
    a46086a View commit details
    Browse the repository at this point in the history

  10. Use sqlalchemy.sql.True_() to support all DB backends

    @blag
    blag committed Nov 25, 2022
    Configuration menu
    Copy the full SHA
    8d6d9c7 View commit details
    Browse the repository at this point in the history

  11. Various cleanups

    @blag
    blag committed Nov 25, 2022
    Configuration menu
    Copy the full SHA
    e88d909 View commit details
    Browse the repository at this point in the history

  12. Add test for un-orphaning datasets once they are referenced again

    @blag
    blag committed Nov 25, 2022
    Configuration menu
    Copy the full SHA
    a123537 View commit details
    Browse the repository at this point in the history

  13. Use sqlalchemy.sql.expression.true() 

    Co-authored-by: Ephraim Anierobi <splendidzigy24@gmail.com>
    @blag @ephraimbuddy
    blag and ephraimbuddy committed Nov 25, 2022
    Configuration menu
    Copy the full SHA
    e244273 View commit details
    Browse the repository at this point in the history

  14. Fix orphaning datasets on MSSQL

    @blag
    blag committed Nov 25, 2022
    Configuration menu
    Copy the full SHA
    a3930b3 View commit details
    Browse the repository at this point in the history

  15. Comment the un-orphan process and use sqla.sql.expression.false()

    @blag
    blag committed Nov 25, 2022
    Configuration menu
    Copy the full SHA
    559ee24 View commit details
    Browse the repository at this point in the history

  16. Merge remote-tracking branch 'apache/main' into soft-delete-orphaned-… 

    …datasets
    @blag
    blag committed Nov 25, 2022
    Configuration menu
    Copy the full SHA
    83e865b View commit details
    Browse the repository at this point in the history

  17. Add newsfragment about renamed config option

    @blag
    blag committed Nov 25, 2022
    Configuration menu
    Copy the full SHA
    9eae8db View commit details
    Browse the repository at this point in the history

  18. add mssql_drop_default flag

    @ephraimbuddy
    ephraimbuddy committed Nov 25, 2022
    Configuration menu
    Copy the full SHA
    3a9864a View commit details
    Browse the repository at this point in the history

  19. Use server_default in the ORM as well

    @blag
    blag committed Nov 25, 2022
    Configuration menu
    Copy the full SHA
    a6b3714 View commit details
    Browse the repository at this point in the history

  20. Defensively clear datasets before and after DAG tests

    @blag
    blag committed Nov 25, 2022
    Configuration menu
    Copy the full SHA
    6112e19 View commit details
    Browse the repository at this point in the history

  21. Reconcile migration with ORM model

    @blag
    blag committed Nov 25, 2022
    Configuration menu
    Copy the full SHA
    2197254 View commit details
    Browse the repository at this point in the history

  22. Remove now erroneous comment

    @blag
    blag committed Nov 25, 2022
    Configuration menu
    Copy the full SHA
    bc7a35c View commit details
    Browse the repository at this point in the history

  23. Change to use server_default='0'

    @ephraimbuddy
    ephraimbuddy committed Nov 25, 2022
    Configuration menu
    Copy the full SHA
    a5cdb47 View commit details
    Browse the repository at this point in the history

  24. Update airflow/configuration.py

    @jedcunningham
    jedcunningham committed Nov 25, 2022
    Configuration menu
    Copy the full SHA
    286ed01 View commit details
    Browse the repository at this point in the history