Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[HUDI-2560][RFC-33] Support full Schema evolution for Spark #4910

Merged
merged 27 commits into from
Apr 1, 2022
Merged

[HUDI-2560][RFC-33] Support full Schema evolution for Spark #4910

merged 27 commits into from
Apr 1, 2022
+10,352 −106

Commits on Apr 1, 2022

  1. [HUDI-2560] introduce id_based schema to support full schema evolution.

    @xiarixiaoyao
    xiarixiaoyao committed Apr 1, 2022
    Configuration menu
    Copy the full SHA
    ced7477 View commit details
    Browse the repository at this point in the history

  2. add test for FileBasedInternalSchemaStorageManger and rebase code

    @xiarixiaoyao
    xiarixiaoyao committed Apr 1, 2022
    Configuration menu
    Copy the full SHA
    13fad93 View commit details
    Browse the repository at this point in the history

  3. add support for change column type and fix some test case

    @xiarixiaoyao
    xiarixiaoyao committed Apr 1, 2022
    Configuration menu
    Copy the full SHA
    3df69e7 View commit details
    Browse the repository at this point in the history

  4. fix some bugs encountered in the production env and delete useless code

    @xiarixiaoyao
    xiarixiaoyao committed Apr 1, 2022
    Configuration menu
    Copy the full SHA
    64b5492 View commit details
    Browse the repository at this point in the history

  5. fix test error

    @xiarixiaoyao
    xiarixiaoyao committed Apr 1, 2022
    Configuration menu
    Copy the full SHA
    0b468d1 View commit details
    Browse the repository at this point in the history

  6. rebase code

    @xiarixiaoyao
    xiarixiaoyao committed Apr 1, 2022
    Configuration menu
    Copy the full SHA
    c34fdf8 View commit details
    Browse the repository at this point in the history

  7. fixed some nested schema change bugs

    @xiarixiaoyao
    xiarixiaoyao committed Apr 1, 2022
    Configuration menu
    Copy the full SHA
    31a79d5 View commit details
    Browse the repository at this point in the history

  8. [HUDI-2429][Stacked On HUDI-2560]Support full schema evolution for spark

    @xiarixiaoyao
    xiarixiaoyao committed Apr 1, 2022
    Configuration menu
    Copy the full SHA
    926c0be View commit details
    Browse the repository at this point in the history

  9. [use dummyInternalSchema instead of null]

    @xiarixiaoyao
    xiarixiaoyao committed Apr 1, 2022
    Configuration menu
    Copy the full SHA
    b1aa45e View commit details
    Browse the repository at this point in the history

  10. add support for spark3.1.x

    @xiarixiaoyao
    xiarixiaoyao committed Apr 1, 2022
    Configuration menu
    Copy the full SHA
    0edac35 View commit details
    Browse the repository at this point in the history

  11. remove support for spark3.1.x , sicne some compile fail

    @xiarixiaoyao
    xiarixiaoyao committed Apr 1, 2022
    Configuration menu
    Copy the full SHA
    f7dde32 View commit details
    Browse the repository at this point in the history

  12. support spark3.1.x

    @xiarixiaoyao
    xiarixiaoyao committed Apr 1, 2022
    Configuration menu
    Copy the full SHA
    d808254 View commit details
    Browse the repository at this point in the history

  13. rebase and prepare solve all comments

    @xiarixiaoyao
    xiarixiaoyao committed Apr 1, 2022
    Configuration menu
    Copy the full SHA
    057e8f0 View commit details
    Browse the repository at this point in the history

  14. address all comments

    @xiarixiaoyao
    xiarixiaoyao committed Apr 1, 2022
    Configuration menu
    Copy the full SHA
    58cd3a6 View commit details
    Browse the repository at this point in the history

  15. rebase code

    @xiarixiaoyao
    xiarixiaoyao committed Apr 1, 2022
    Configuration menu
    Copy the full SHA
    8951618 View commit details
    Browse the repository at this point in the history

  16. fixed the count(*) bug

    @xiarixiaoyao
    xiarixiaoyao committed Apr 1, 2022
    Configuration menu
    Copy the full SHA
    7a0dc9e View commit details
    Browse the repository at this point in the history

  17. try to get internalSchema by parser commit file/history file directly… 

    …, not use metaclient which is time cost

address some comments
    @xiarixiaoyao
    xiarixiaoyao committed Apr 1, 2022
    Configuration menu
    Copy the full SHA
    eee91e5 View commit details
    Browse the repository at this point in the history

  18. fixed all comments

    @xiarixiaoyao
    xiarixiaoyao committed Apr 1, 2022
    Configuration menu
    Copy the full SHA
    3cde85a View commit details
    Browse the repository at this point in the history

  19. fix new comments

    @xiarixiaoyao
    xiarixiaoyao committed Apr 1, 2022
    Configuration menu
    Copy the full SHA
    d8fa679 View commit details
    Browse the repository at this point in the history

  20. rebase code,fix UT failed

    @xiarixiaoyao
    xiarixiaoyao committed Apr 1, 2022
    Configuration menu
    Copy the full SHA
    6828189 View commit details
    Browse the repository at this point in the history

  21. fixed mistake

    @xiarixiaoyao
    xiarixiaoyao committed Apr 1, 2022
    Configuration menu
    Copy the full SHA
    fc707d4 View commit details
    Browse the repository at this point in the history

  22. rebase code ,fixed new comments

    @xiarixiaoyao
    xiarixiaoyao committed Apr 1, 2022
    Configuration menu
    Copy the full SHA
    011e886 View commit details
    Browse the repository at this point in the history

  23. rebase code , and prepare for address new comments

    @xiarixiaoyao
    xiarixiaoyao committed Apr 1, 2022
    Configuration menu
    Copy the full SHA
    dc3b29a View commit details
    Browse the repository at this point in the history

  24. address commits

    @xiarixiaoyao
    xiarixiaoyao committed Apr 1, 2022
    Configuration menu
    Copy the full SHA
    3c9bfc9 View commit details
    Browse the repository at this point in the history

  25. address new comments

    @xiarixiaoyao
    xiarixiaoyao committed Apr 1, 2022
    Configuration menu
    Copy the full SHA
    71fcd14 View commit details
    Browse the repository at this point in the history

  26. fix new issues

    @xiarixiaoyao
    xiarixiaoyao committed Apr 1, 2022
    Configuration menu
    Copy the full SHA
    c7c98b0 View commit details
    Browse the repository at this point in the history

  27. control fallback original write logical

    @xiarixiaoyao
    xiarixiaoyao committed Apr 1, 2022
    Configuration menu
    Copy the full SHA
    2415e5c View commit details
    Browse the repository at this point in the history