Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

New "refresh" mode and "dev_mode" #1063

Merged
merged 51 commits into from
Jun 3, 2024
Merged

New "refresh" mode and "dev_mode" #1063

merged 51 commits into from
Jun 3, 2024

Commits on May 29, 2024

  1. Rename full_refresh -> dev_mode, add deprecation warning

    @steinitzu
    steinitzu committed May 29, 2024
    Configuration menu
    Copy the full SHA
    dba576b View commit details
    Browse the repository at this point in the history

  2. Replace some full_refresh usage in code and docs

    @steinitzu
    steinitzu committed May 29, 2024
    Configuration menu
    Copy the full SHA
    432d1d7 View commit details
    Browse the repository at this point in the history

  3. Replace full_refresh usage in tests

    @steinitzu
    steinitzu committed May 29, 2024
    Configuration menu
    Copy the full SHA
    c7e86ef View commit details
    Browse the repository at this point in the history

  4. Init experimental refresh = full with drop command

    @steinitzu
    steinitzu committed May 29, 2024
    Configuration menu
    Copy the full SHA
    8c13683 View commit details
    Browse the repository at this point in the history

  5. Refresh modes with dropped_tables file

    @steinitzu
    steinitzu committed May 29, 2024
    Configuration menu
    Copy the full SHA
    613257e View commit details
    Browse the repository at this point in the history

  6. Init separate local/write drop command

    @steinitzu
    steinitzu committed May 29, 2024
    Configuration menu
    Copy the full SHA
    2d46305 View commit details
    Browse the repository at this point in the history

  7. Use load_package_state instead of drop tables file

    @steinitzu
    steinitzu committed May 29, 2024
    Configuration menu
    Copy the full SHA
    9b63cff View commit details
    Browse the repository at this point in the history

  8. Use drop schema in init_client (TODO: error)

    @steinitzu
    steinitzu committed May 29, 2024
    Configuration menu
    Copy the full SHA
    3b6d809 View commit details
    Browse the repository at this point in the history

  9. Separate cli drop command instructions/execute

    @steinitzu
    steinitzu committed May 29, 2024
    Configuration menu
    Copy the full SHA
    4334553 View commit details
    Browse the repository at this point in the history

  10. drop tables in init_client

    @steinitzu
    steinitzu committed May 29, 2024
    Configuration menu
    Copy the full SHA
    b6a21aa View commit details
    Browse the repository at this point in the history

  11. dropped_tables field on load package state

    @steinitzu
    steinitzu committed May 29, 2024
    Configuration menu
    Copy the full SHA
    e9543e0 View commit details
    Browse the repository at this point in the history

  12. Fix import

    @steinitzu
    steinitzu committed May 29, 2024
    Configuration menu
    Copy the full SHA
    7f95d9e View commit details
    Browse the repository at this point in the history

  13. test/fix truncate mode

    @steinitzu
    steinitzu committed May 29, 2024
    Configuration menu
    Copy the full SHA
    3e0af96 View commit details
    Browse the repository at this point in the history

  14. Save truncated tables in load package state

    @steinitzu
    steinitzu committed May 29, 2024
    Configuration menu
    Copy the full SHA
    e9528b1 View commit details
    Browse the repository at this point in the history

  15. Remove load package state copying

    @steinitzu
    steinitzu committed May 29, 2024
    Configuration menu
    Copy the full SHA
    10fe6f3 View commit details
    Browse the repository at this point in the history

  16. cleanup

    @steinitzu
    steinitzu committed May 29, 2024
    Configuration menu
    Copy the full SHA
    44cc1cc View commit details
    Browse the repository at this point in the history

  17. Drop cmd use package state, refactoring

    @steinitzu
    steinitzu committed May 29, 2024
    Configuration menu
    Copy the full SHA
    54de570 View commit details
    Browse the repository at this point in the history

  18. Don't drop tables without data

    @steinitzu
    steinitzu committed May 29, 2024
    Configuration menu
    Copy the full SHA
    6aae866 View commit details
    Browse the repository at this point in the history

  19. Validate literals in configspec

    @steinitzu
    steinitzu committed May 29, 2024
    Configuration menu
    Copy the full SHA
    29b6c96 View commit details
    Browse the repository at this point in the history

  20. Match stored schema by version+version_hash 

    solves detecting when dropped tables need to be recreated
    @steinitzu
    steinitzu committed May 29, 2024
    Configuration menu
    Copy the full SHA
    8a2276b View commit details
    Browse the repository at this point in the history

  21. Cleanup

    @steinitzu
    steinitzu committed May 29, 2024
    Configuration menu
    Copy the full SHA
    2546021 View commit details
    Browse the repository at this point in the history

  22. Fix dlt version test

    @steinitzu
    steinitzu committed May 29, 2024
    Configuration menu
    Copy the full SHA
    c749eae View commit details
    Browse the repository at this point in the history

  23. Cleanup

    @steinitzu
    steinitzu committed May 29, 2024
    Configuration menu
    Copy the full SHA
    7802e85 View commit details
    Browse the repository at this point in the history

  24. Remove dropped_tables_filename

    @steinitzu
    steinitzu committed May 29, 2024
    Configuration menu
    Copy the full SHA
    ce41fb0 View commit details
    Browse the repository at this point in the history

  25. Fix snippet

    @steinitzu
    steinitzu committed May 29, 2024
    Configuration menu
    Copy the full SHA
    0f9358d View commit details
    Browse the repository at this point in the history

  26. Pipeline refresh docs

    @steinitzu
    steinitzu committed May 29, 2024
    Configuration menu
    Copy the full SHA
    7a161a8 View commit details
    Browse the repository at this point in the history

  27. refresh argument docstring

    @steinitzu
    steinitzu committed May 29, 2024
    Configuration menu
    Copy the full SHA
    113d577 View commit details
    Browse the repository at this point in the history

  28. Restore and update commented drop cmd tests

    @steinitzu
    steinitzu committed May 29, 2024
    Configuration menu
    Copy the full SHA
    73aab0e View commit details
    Browse the repository at this point in the history

  29. Cleanup refresh tests and test whether table dropped vs truncated

    @steinitzu
    steinitzu committed May 29, 2024
    Configuration menu
    Copy the full SHA
    f18509e View commit details
    Browse the repository at this point in the history

  30. Test existing schema hash

    @steinitzu
    steinitzu committed May 29, 2024
    Configuration menu
    Copy the full SHA
    b662381 View commit details
    Browse the repository at this point in the history

  31. Revert "Match stored schema by version+version_hash" 

    This reverts commit 689b3ca.
    @steinitzu
    steinitzu committed May 29, 2024
    Configuration menu
    Copy the full SHA
    ed43445 View commit details
    Browse the repository at this point in the history

  32. Use replace_schema flag

    @steinitzu
    steinitzu committed May 29, 2024
    Configuration menu
    Copy the full SHA
    fbfb97c View commit details
    Browse the repository at this point in the history

  33. Change drop_tables replace_schema to delete_schema

    @steinitzu
    steinitzu committed May 29, 2024
    Configuration menu
    Copy the full SHA
    2ed7d39 View commit details
    Browse the repository at this point in the history

  34. Refresh drop only selected sources

    @steinitzu
    steinitzu committed May 29, 2024
    Configuration menu
    Copy the full SHA
    b8d98f6 View commit details
    Browse the repository at this point in the history

  35. Rename refresh modes, update docs

    @steinitzu
    steinitzu committed May 29, 2024
    Configuration menu
    Copy the full SHA
    1a28168 View commit details
    Browse the repository at this point in the history

  36. pipeline.run/extract refresh argument

    @steinitzu
    steinitzu committed May 29, 2024
    Configuration menu
    Copy the full SHA
    360e2c4 View commit details
    Browse the repository at this point in the history

  37. Don't modify schema when refresh='drop_data'

    @steinitzu
    steinitzu committed May 29, 2024
    Configuration menu
    Copy the full SHA
    6f5a853 View commit details
    Browse the repository at this point in the history

  38. Move refresh tests to load, add filesystem truncate test

    @steinitzu
    steinitzu committed May 29, 2024
    Configuration menu
    Copy the full SHA
    a122834 View commit details
    Browse the repository at this point in the history

  39. Fix duck import

    @steinitzu
    steinitzu committed May 29, 2024
    Configuration menu
    Copy the full SHA
    bbf56a0 View commit details
    Browse the repository at this point in the history

  40. Remove generated doc sections

    @steinitzu
    steinitzu committed May 29, 2024
    Configuration menu
    Copy the full SHA
    d9bfe90 View commit details
    Browse the repository at this point in the history

  41. Default full_refresh=None

    @steinitzu
    steinitzu committed May 29, 2024
    Configuration menu
    Copy the full SHA
    0abffa5 View commit details
    Browse the repository at this point in the history

  42. Cleanup unused imports

    @steinitzu
    steinitzu committed May 29, 2024
    Configuration menu
    Copy the full SHA
    f096596 View commit details
    Browse the repository at this point in the history

  43. Close caution blocks

    @steinitzu
    steinitzu committed May 29, 2024
    Configuration menu
    Copy the full SHA
    be7e0d0 View commit details
    Browse the repository at this point in the history

  44. Update config field docstring

    @steinitzu
    steinitzu committed May 29, 2024
    Configuration menu
    Copy the full SHA
    8b5aa2d View commit details
    Browse the repository at this point in the history

  45. Add filesystem drop_tables method

    @steinitzu
    steinitzu committed May 29, 2024
    Configuration menu
    Copy the full SHA
    aaaf519 View commit details
    Browse the repository at this point in the history

  46. Run all refresh tests on local filesystem destination

    @steinitzu
    steinitzu committed May 29, 2024
    Configuration menu
    Copy the full SHA
    e3b5b35 View commit details
    Browse the repository at this point in the history

  47. Fix test drop

    @steinitzu
    steinitzu committed May 29, 2024
    Configuration menu
    Copy the full SHA
    3dcd1ab View commit details
    Browse the repository at this point in the history

  48. Fix iter filesystem schemas

    @steinitzu
    steinitzu committed May 29, 2024
    Configuration menu
    Copy the full SHA
    8580ea4 View commit details
    Browse the repository at this point in the history

  49. Fix drop_resources

    @steinitzu
    steinitzu committed May 29, 2024
    Configuration menu
    Copy the full SHA
    6a06449 View commit details
    Browse the repository at this point in the history

  50. Default config.full_refresh also None

    @steinitzu
    steinitzu committed May 29, 2024
    Configuration menu
    Copy the full SHA
    87cfd16 View commit details
    Browse the repository at this point in the history

  51. Fix filesystem + test

    @steinitzu
    steinitzu committed May 29, 2024
    Configuration menu
    Copy the full SHA
    879e4e5 View commit details
    Browse the repository at this point in the history