Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Speed up LoadMode.DBT_LS by caching dbt ls output in Airflow Variable #1014

Merged
merged 84 commits into from
Jun 25, 2024
Merged

Speed up LoadMode.DBT_LS by caching dbt ls output in Airflow Variable #1014

merged 84 commits into from
Jun 25, 2024

Commits on Jun 25, 2024

  1. Add more logs to confirm bottleneck

    @tatiana
    tatiana committed Jun 25, 2024
    Configuration menu
    Copy the full SHA
    0ea8c19 View commit details
    Browse the repository at this point in the history

  2. Support retrieving dbt ls cache from Airflow Variable

    @tatiana
    tatiana committed Jun 25, 2024
    Configuration menu
    Copy the full SHA
    71dca3c View commit details
    Browse the repository at this point in the history

  3. Release 1.5.0a3

    @tatiana
    tatiana committed Jun 25, 2024
    Configuration menu
    Copy the full SHA
    c46e8b1 View commit details
    Browse the repository at this point in the history

  4. Fix log message for TaskGroups

    @tatiana
    tatiana committed Jun 25, 2024
    Configuration menu
    Copy the full SHA
    240b03c View commit details
    Browse the repository at this point in the history

  5. Fix log for Cosmos TaskGroup

    @tatiana
    tatiana committed Jun 25, 2024
    Configuration menu
    Copy the full SHA
    ed8b6fe View commit details
    Browse the repository at this point in the history

  6. Update changelog after last rebase

    @tatiana
    tatiana committed Jun 25, 2024
    Configuration menu
    Copy the full SHA
    5dc96af View commit details
    Browse the repository at this point in the history

  7. Introduce CachePurgeConfig

    @tatiana
    tatiana committed Jun 25, 2024
    Configuration menu
    Copy the full SHA
    4e0b71d View commit details
    Browse the repository at this point in the history

  8. Update release notes after rebase

    @tatiana
    tatiana committed Jun 25, 2024
    Configuration menu
    Copy the full SHA
    5602e9b View commit details
    Browse the repository at this point in the history

  9. Update release notes after rebase

    @tatiana
    tatiana committed Jun 25, 2024
    Configuration menu
    Copy the full SHA
    fb3c99c View commit details
    Browse the repository at this point in the history

  10. Fix alpha release version

    @tatiana
    tatiana committed Jun 25, 2024
    Configuration menu
    Copy the full SHA
    4768ea7 View commit details
    Browse the repository at this point in the history

  11. Improve var caching to be prefixed with cosmos_cache

    @tatiana
    tatiana committed Jun 25, 2024
    Configuration menu
    Copy the full SHA
    b5b0aff View commit details
    Browse the repository at this point in the history

  12. Temporarily remove fallback to custom load method in automatic, to tr… 

    …oubleshoot what is being implemented
    @tatiana
    tatiana committed Jun 25, 2024
    Configuration menu
    Copy the full SHA
    9d36171 View commit details
    Browse the repository at this point in the history

  13. Remove CachePurgeConfig and purge based on commands passed to dbt ls

    @tatiana
    tatiana committed Jun 25, 2024
    Configuration menu
    Copy the full SHA
    94f82a3 View commit details
    Browse the repository at this point in the history

  14. Add RenderConfig.airflow_vars_to_purge_cache

    @tatiana
    tatiana committed Jun 25, 2024
    Configuration menu
    Copy the full SHA
    e4ece12 View commit details
    Browse the repository at this point in the history

  15. Release 1.5.0a6

    @tatiana
    tatiana committed Jun 25, 2024
    Configuration menu
    Copy the full SHA
    625d980 View commit details
    Browse the repository at this point in the history

  16. Change identifying dbt project changes based on md5

    @tatiana
    tatiana committed Jun 25, 2024
    Configuration menu
    Copy the full SHA
    f29efc5 View commit details
    Browse the repository at this point in the history

  17. Use cached_property

    @tatiana
    tatiana committed Jun 25, 2024
    Configuration menu
    Copy the full SHA
    955c7b5 View commit details
    Browse the repository at this point in the history

  18. Use sha256 to hash the dir

    @tatiana
    tatiana committed Jun 25, 2024
    Configuration menu
    Copy the full SHA
    c0efbd3 View commit details
    Browse the repository at this point in the history

  19. Add logs

    @tatiana
    tatiana committed Jun 25, 2024
    Configuration menu
    Copy the full SHA
    d93381f View commit details
    Browse the repository at this point in the history

  20. Fix logs

    @tatiana
    tatiana committed Jun 25, 2024
    Configuration menu
    Copy the full SHA
    0fe9de7 View commit details
    Browse the repository at this point in the history

  21. Release 1.5.0a7

    @tatiana
    tatiana committed Jun 25, 2024
    Configuration menu
    Copy the full SHA
    b298a27 View commit details
    Browse the repository at this point in the history

  22. Add docstrings and rename args

    @tatiana
    tatiana committed Jun 25, 2024
    Configuration menu
    Copy the full SHA
    913e810 View commit details
    Browse the repository at this point in the history

  23. Remove stale comment

    @tatiana
    tatiana committed Jun 25, 2024
    Configuration menu
    Copy the full SHA
    1446987 View commit details
    Browse the repository at this point in the history

  24. Fix behaviour when user passes ProjectConfig(dbt_vars) in dbt ls

    @tatiana
    tatiana committed Jun 25, 2024
    Configuration menu
    Copy the full SHA
    1f6f62b View commit details
    Browse the repository at this point in the history

  25. Add more tests

    @tatiana
    tatiana committed Jun 25, 2024
    Configuration menu
    Copy the full SHA
    c19b80b View commit details
    Browse the repository at this point in the history

  26. Add more tests

    @tatiana
    tatiana committed Jun 25, 2024
    Configuration menu
    Copy the full SHA
    c811d23 View commit details
    Browse the repository at this point in the history

  27. Add more tests

    @tatiana
    tatiana committed Jun 25, 2024
    Configuration menu
    Copy the full SHA
    3e67d3a View commit details
    Browse the repository at this point in the history

  28. Fix unittests that were trying to fetch Airflow vars

    @tatiana
    tatiana committed Jun 25, 2024
    Configuration menu
    Copy the full SHA
    72b39b3 View commit details
    Browse the repository at this point in the history

  29. Try to fix test so it passes in linux and macos

    @tatiana
    tatiana committed Jun 25, 2024
    Configuration menu
    Copy the full SHA
    125d180 View commit details
    Browse the repository at this point in the history

  30. Fix dbt graph test/--vars

    @tatiana
    tatiana committed Jun 25, 2024
    Configuration menu
    Copy the full SHA
    840a85c View commit details
    Browse the repository at this point in the history

  31. Fix test passing vars

    @tatiana
    tatiana committed Jun 25, 2024
    Configuration menu
    Copy the full SHA
    bb9eca1 View commit details
    Browse the repository at this point in the history

  32. Fix test graph

    @tatiana
    tatiana committed Jun 25, 2024
    Configuration menu
    Copy the full SHA
    d73a8d2 View commit details
    Browse the repository at this point in the history

  33. Fix CI integrationt ests issue

    @tatiana
    tatiana committed Jun 25, 2024
    Configuration menu
    Copy the full SHA
    a830135 View commit details
    Browse the repository at this point in the history

  34. Fix issue uncovered by sqlite test

    @tatiana
    tatiana committed Jun 25, 2024
    Configuration menu
    Copy the full SHA
    f9571a1 View commit details
    Browse the repository at this point in the history

  35. Change tests that should not be using cache

    @tatiana
    tatiana committed Jun 25, 2024
    Configuration menu
    Copy the full SHA
    7624076 View commit details
    Browse the repository at this point in the history

  36. Change from sha256 to md5 to see if it is more deterministic

    @tatiana
    tatiana committed Jun 25, 2024
    Configuration menu
    Copy the full SHA
    20b636b View commit details
    Browse the repository at this point in the history

  37. Try to get deterministic value on hash for dbt project dir

    @tatiana
    tatiana committed Jun 25, 2024
    Configuration menu
    Copy the full SHA
    5e8daf5 View commit details
    Browse the repository at this point in the history

  38. Fix hash value in linux

    @tatiana
    tatiana committed Jun 25, 2024
    Configuration menu
    Copy the full SHA
    624d9e8 View commit details
    Browse the repository at this point in the history

  39. Try to fix CI unittests

    @tatiana
    tatiana committed Jun 25, 2024
    Configuration menu
    Copy the full SHA
    9b27522 View commit details
    Browse the repository at this point in the history

  40. fix unittest in the CI

    @tatiana
    tatiana committed Jun 25, 2024
    Configuration menu
    Copy the full SHA
    d860a6c View commit details
    Browse the repository at this point in the history

  41. fix unittest in the CI

    @tatiana
    tatiana committed Jun 25, 2024
    Configuration menu
    Copy the full SHA
    763036f View commit details
    Browse the repository at this point in the history

  42. Split test by linux/darwin

    @tatiana
    tatiana committed Jun 25, 2024
    Configuration menu
    Copy the full SHA
    0168516 View commit details
    Browse the repository at this point in the history

  43. Fix test_load_via_dbt_ls_caching_partial_parsing

    @tatiana
    tatiana committed Jun 25, 2024
    Configuration menu
    Copy the full SHA
    04a4ab7 View commit details
    Browse the repository at this point in the history

  44. Fix pyproject to fix integration tests

    @tatiana
    tatiana committed Jun 25, 2024
    Configuration menu
    Copy the full SHA
    64b9ea4 View commit details
    Browse the repository at this point in the history

  45. troubleshoot

    @tatiana
    tatiana committed Jun 25, 2024
    Configuration menu
    Copy the full SHA
    cf81a86 View commit details
    Browse the repository at this point in the history

  46. Troubleshoot CI

    @tatiana
    tatiana committed Jun 25, 2024
    Configuration menu
    Copy the full SHA
    50f2c10 View commit details
    Browse the repository at this point in the history

  47. Troubleshoot CI

    @tatiana
    tatiana committed Jun 25, 2024
    Configuration menu
    Copy the full SHA
    2bd7e50 View commit details
    Browse the repository at this point in the history

  48. Attempts to get tests to pass

    @tatiana
    tatiana committed Jun 25, 2024
    Configuration menu
    Copy the full SHA
    23c1bc8 View commit details
    Browse the repository at this point in the history

  49. Fix ci yml

    @tatiana
    tatiana committed Jun 25, 2024
    Configuration menu
    Copy the full SHA
    e24576f View commit details
    Browse the repository at this point in the history

  50. Disable 2.9 tests

    @tatiana
    tatiana committed Jun 25, 2024
    Configuration menu
    Copy the full SHA
    98232f7 View commit details
    Browse the repository at this point in the history

  51. Reduce test matrix to help troubleshooting

    @tatiana
    tatiana committed Jun 25, 2024
    Configuration menu
    Copy the full SHA
    5e53a4a View commit details
    Browse the repository at this point in the history

  52. Rename connection used in example dags

    @tatiana
    tatiana committed Jun 25, 2024
    Configuration menu
    Copy the full SHA
    37264d8 View commit details
    Browse the repository at this point in the history

  53. Try to reset Airflow db

    @tatiana
    tatiana committed Jun 25, 2024
    Configuration menu
    Copy the full SHA
    26207ae View commit details
    Browse the repository at this point in the history

  54. Force airflow db to be reset

    @tatiana
    tatiana committed Jun 25, 2024
    Configuration menu
    Copy the full SHA
    a3b0656 View commit details
    Browse the repository at this point in the history

  55. Run a subset of integration tests

    @tatiana
    tatiana committed Jun 25, 2024
    Configuration menu
    Copy the full SHA
    935cf51 View commit details
    Browse the repository at this point in the history

  56. Remove CI cache on int tests

    @tatiana
    tatiana committed Jun 25, 2024
    Configuration menu
    Copy the full SHA
    890ed11 View commit details
    Browse the repository at this point in the history

  57. Try to renable all integration tests

    @tatiana
    tatiana committed Jun 25, 2024
    Configuration menu
    Copy the full SHA
    afd3dbd View commit details
    Browse the repository at this point in the history

  58. Try to fix integration tests

    @tatiana
    tatiana committed Jun 25, 2024
    Configuration menu
    Copy the full SHA
    aaa3a74 View commit details
    Browse the repository at this point in the history

  59. Remove incorrrectly commited dev/dags/dbt/perf/models/

    @tatiana
    tatiana committed Jun 25, 2024
    Configuration menu
    Copy the full SHA
    36e38a3 View commit details
    Browse the repository at this point in the history

  60. 🎨 [pre-commit.ci] Auto format from pre-commit.com hooks

    @pre-commit-ci @tatiana
    pre-commit-ci[bot] authored and tatiana committed Jun 25, 2024
    Configuration menu
    Copy the full SHA
    a69542c View commit details
    Browse the repository at this point in the history

  61. Fix version of Airflow to supported version

    @tatiana
    tatiana committed Jun 25, 2024
    Configuration menu
    Copy the full SHA
    6c9bc63 View commit details
    Browse the repository at this point in the history

  62. Improve and tidy up tests

    @tatiana
    tatiana committed Jun 25, 2024
    Configuration menu
    Copy the full SHA
    b3c3716 View commit details
    Browse the repository at this point in the history

  63. Revert changes to workflows test

    @tatiana
    tatiana committed Jun 25, 2024
    Configuration menu
    Copy the full SHA
    254d276 View commit details
    Browse the repository at this point in the history

  64. Introduce method to delete stale Cosmos cache Airflow Variables

    @tatiana
    tatiana committed Jun 25, 2024
    Configuration menu
    Copy the full SHA
    9767c95 View commit details
    Browse the repository at this point in the history

  65. Release 1.5.0a8

    @tatiana
    tatiana committed Jun 25, 2024
    Configuration menu
    Copy the full SHA
    9bf1898 View commit details
    Browse the repository at this point in the history

  66. Change process_time to perf_counter

    @tatiana
    tatiana committed Jun 25, 2024
    Configuration menu
    Copy the full SHA
    859a257 View commit details
    Browse the repository at this point in the history

  67. Address PR feedback

    @tatiana
    tatiana committed Jun 25, 2024
    Configuration menu
    Copy the full SHA
    b9297c0 View commit details
    Browse the repository at this point in the history

  68. Address PR feedback

    @tatiana
    tatiana committed Jun 25, 2024
    Configuration menu
    Copy the full SHA
    16d288c View commit details
    Browse the repository at this point in the history

  69. Address PR feedback

    @tatiana
    tatiana committed Jun 25, 2024
    Configuration menu
    Copy the full SHA
    65a9f07 View commit details
    Browse the repository at this point in the history

  70. Address PR feedback

    @tatiana
    tatiana committed Jun 25, 2024
    Configuration menu
    Copy the full SHA
    b362afe View commit details
    Browse the repository at this point in the history

  71. Add an example DAG to cleanup Cosmos cache originated from running db… 

    …t ls
    @tatiana
    tatiana committed Jun 25, 2024
    Configuration menu
    Copy the full SHA
    56eeaba View commit details
    Browse the repository at this point in the history

  72. Make calculate_dbt_ls_cache_current_version private

    @tatiana
    tatiana committed Jun 25, 2024
    Configuration menu
    Copy the full SHA
    d0c323d View commit details
    Browse the repository at this point in the history

  73. Add debug statement for dbt_ls_cache_key_args

    @tatiana
    tatiana committed Jun 25, 2024
    Configuration menu
    Copy the full SHA
    ea452f7 View commit details
    Browse the repository at this point in the history

  74. Use timezone.utc as opposed to pytz

    @tatiana
    tatiana committed Jun 25, 2024
    Configuration menu
    Copy the full SHA
    7ba3b63 View commit details
    Browse the repository at this point in the history

  75. Change logger.warn to logger.info

    @tatiana
    tatiana committed Jun 25, 2024
    Configuration menu
    Copy the full SHA
    97dddba View commit details
    Browse the repository at this point in the history

  76. Remove debug file

    @tatiana
    tatiana committed Jun 25, 2024
    Configuration menu
    Copy the full SHA
    3cb6bed View commit details
    Browse the repository at this point in the history

  77. Add tests to cover delete_unused_dbt_ls_cache

    @tatiana
    tatiana committed Jun 25, 2024
    Configuration menu
    Copy the full SHA
    dae75fd View commit details
    Browse the repository at this point in the history

  78. Rename int test to int

    @tatiana
    tatiana committed Jun 25, 2024
    Configuration menu
    Copy the full SHA
    8ccf6ef View commit details
    Browse the repository at this point in the history

  79. Add docs

    @tatiana
    tatiana committed Jun 25, 2024
    Configuration menu
    Copy the full SHA
    908a48e View commit details
    Browse the repository at this point in the history

  80. Add docs

    @tatiana
    tatiana committed Jun 25, 2024
    Configuration menu
    Copy the full SHA
    00eff69 View commit details
    Browse the repository at this point in the history

  81. Improve docs, spelling, grammar

    @tatiana
    tatiana committed Jun 25, 2024
    Configuration menu
    Copy the full SHA
    b6a9077 View commit details
    Browse the repository at this point in the history

  82. Improve docs

    @tatiana
    tatiana committed Jun 25, 2024
    Configuration menu
    Copy the full SHA
    cedb178 View commit details
    Browse the repository at this point in the history

  83. Release 1.5.0a9

    @tatiana
    tatiana committed Jun 25, 2024
    Configuration menu
    Copy the full SHA
    46a9b78 View commit details
    Browse the repository at this point in the history

  84. Update changelog

    @tatiana
    tatiana committed Jun 25, 2024
    Configuration menu
    Copy the full SHA
    4dd9549 View commit details
    Browse the repository at this point in the history