Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[SPARK-36206][CORE] Support shuffle data corruption diagnosis via shuffle checksum #33451

Closed
wants to merge 48 commits into from
Closed

[SPARK-36206][CORE] Support shuffle data corruption diagnosis via shuffle checksum #33451

wants to merge 48 commits into from

Commits on Aug 2, 2021

  1. init

    @Ngone51
    Ngone51 committed Aug 2, 2021
    Configuration menu
    Copy the full SHA
    7b40e2e View commit details
    Browse the repository at this point in the history

  2. update

    @Ngone51
    Ngone51 committed Aug 2, 2021
    Configuration menu
    Copy the full SHA
    3cdd858 View commit details
    Browse the repository at this point in the history

  3. move comment

    @Ngone51
    Ngone51 committed Aug 2, 2021
    Configuration menu
    Copy the full SHA
    4c01069 View commit details
    Browse the repository at this point in the history

  4. fix ExternalBlockStoreClient

    @Ngone51
    Ngone51 committed Aug 2, 2021
    Configuration menu
    Copy the full SHA
    7bd3788 View commit details
    Browse the repository at this point in the history

  5. mark Cause as private

    @Ngone51
    Ngone51 committed Aug 2, 2021
    Configuration menu
    Copy the full SHA
    8eb667f View commit details
    Browse the repository at this point in the history

  6. remove unused Cause import

    @Ngone51
    Ngone51 committed Aug 2, 2021
    Configuration menu
    Copy the full SHA
    8015e86 View commit details
    Browse the repository at this point in the history

  7. fix indents of BlockStoreClient.diagnoseCorruption

    @Ngone51
    Ngone51 committed Aug 2, 2021
    Configuration menu
    Copy the full SHA
    b25914d View commit details
    Browse the repository at this point in the history

  8. verify -> verifying

    @Ngone51
    Ngone51 committed Aug 2, 2021
    Configuration menu
    Copy the full SHA
    03b26fe View commit details
    Browse the repository at this point in the history

  9. fix comment

    @Ngone51
    Ngone51 committed Aug 2, 2021
    Configuration menu
    Copy the full SHA
    10d60f8 View commit details
    Browse the repository at this point in the history

  10. include checksumfile path

    @Ngone51
    Ngone51 committed Aug 2, 2021
    Configuration menu
    Copy the full SHA
    b0b17ab View commit details
    Browse the repository at this point in the history

  11. remove throws

    @Ngone51
    Ngone51 committed Aug 2, 2021
    Configuration menu
    Copy the full SHA
    c3dcef0 View commit details
    Browse the repository at this point in the history

  12. use ByteStreams.skipFully

    @Ngone51
    Ngone51 committed Aug 2, 2021
    Configuration menu
    Copy the full SHA
    a8f15bd View commit details
    Browse the repository at this point in the history

  13. check cheaper fields first

    @Ngone51
    Ngone51 committed Aug 2, 2021
    Configuration menu
    Copy the full SHA
    815fdf6 View commit details
    Browse the repository at this point in the history

  14. use this.transportConf

    @Ngone51
    Ngone51 committed Aug 2, 2021
    Configuration menu
    Copy the full SHA
    f193368 View commit details
    Browse the repository at this point in the history

  15. add todo for pushbased shuffle

    @Ngone51
    Ngone51 committed Aug 2, 2021
    Configuration menu
    Copy the full SHA
    dfb594d View commit details
    Browse the repository at this point in the history

  16. resolve magic number

    @Ngone51
    Ngone51 committed Aug 2, 2021
    Configuration menu
    Copy the full SHA
    30ed389 View commit details
    Browse the repository at this point in the history

  17. use fileName strip the suffix

    @Ngone51
    Ngone51 committed Aug 2, 2021
    Configuration menu
    Copy the full SHA
    c70fa09 View commit details
    Browse the repository at this point in the history

  18. use Files.newDirectoryStream

    @Ngone51
    Ngone51 committed Aug 2, 2021
    Configuration menu
    Copy the full SHA
    94576d1 View commit details
    Browse the repository at this point in the history

  19. remove checksum file existence check

    @Ngone51
    Ngone51 committed Aug 2, 2021
    Configuration menu
    Copy the full SHA
    e606ca6 View commit details
    Browse the repository at this point in the history

  20. combine ShuffleChecksumHelper&ShuffleCorruptionDiagnosisHelper

    @Ngone51
    Ngone51 committed Aug 2, 2021
    Configuration menu
    Copy the full SHA
    acf20cf View commit details
    Browse the repository at this point in the history

  21. fix tests

    @Ngone51
    Ngone51 committed Aug 2, 2021
    Configuration menu
    Copy the full SHA
    895aad5 View commit details
    Browse the repository at this point in the history

  22. diagnose corruption when the block corrupted twice

    @Ngone51
    Ngone51 committed Aug 2, 2021
    Configuration menu
    Copy the full SHA
    91610b0 View commit details
    Browse the repository at this point in the history

  23. send checksum algorithm together

    @Ngone51
    Ngone51 committed Aug 2, 2021
    Configuration menu
    Copy the full SHA
    6e5d2c0 View commit details
    Browse the repository at this point in the history

  24. fix tests

    @Ngone51
    Ngone51 committed Aug 2, 2021
    Configuration menu
    Copy the full SHA
    993ea3d View commit details
    Browse the repository at this point in the history

  25. fix rat of ShuffleChecksumSupport

    @Ngone51
    Ngone51 committed Aug 2, 2021
    Configuration menu
    Copy the full SHA
    e2e5fa0 View commit details
    Browse the repository at this point in the history

  26. add since for Cause

    @Ngone51
    Ngone51 committed Aug 2, 2021
    Configuration menu
    Copy the full SHA
    c407962 View commit details
    Browse the repository at this point in the history

  27. add comment for fileName

    @Ngone51
    Ngone51 committed Aug 2, 2021
    Configuration menu
    Copy the full SHA
    1b612a2 View commit details
    Browse the repository at this point in the history

  28. fix ExternalBlockStoreClient

    @Ngone51
    Ngone51 committed Aug 2, 2021
    Configuration menu
    Copy the full SHA
    3a41a92 View commit details
    Browse the repository at this point in the history

  29. use the explicit checksum algorithm

    @Ngone51
    Ngone51 committed Aug 2, 2021
    Configuration menu
    Copy the full SHA
    cab36a2 View commit details
    Browse the repository at this point in the history

  30. update comment of diagnoseCorruption

    @Ngone51
    Ngone51 committed Aug 2, 2021
    Configuration menu
    Copy the full SHA
    1292390 View commit details
    Browse the repository at this point in the history

  31. improve error msg for different causes

    @Ngone51
    Ngone51 committed Aug 2, 2021
    Configuration menu
    Copy the full SHA
    8610333 View commit details
    Browse the repository at this point in the history

  32. inline diagnosisResponse into diagnoseCorruption

    @Ngone51
    Ngone51 committed Aug 2, 2021
    Configuration menu
    Copy the full SHA
    47764e3 View commit details
    Browse the repository at this point in the history

  33. update warning msg

    @Ngone51
    Ngone51 committed Aug 2, 2021
    Configuration menu
    Copy the full SHA
    c041f01 View commit details
    Browse the repository at this point in the history

  34. fix ShuffleBlockFetcherIteratorSuite

    @Ngone51
    Ngone51 committed Aug 2, 2021
    Configuration menu
    Copy the full SHA
    4cd8350 View commit details
    Browse the repository at this point in the history

  35. fix java lint

    @Ngone51
    Ngone51 committed Aug 2, 2021
    Configuration menu
    Copy the full SHA
    2da1b91 View commit details
    Browse the repository at this point in the history

  36. fix tests

    @Ngone51
    Ngone51 committed Aug 2, 2021
    Configuration menu
    Copy the full SHA
    95ef9db View commit details
    Browse the repository at this point in the history

  37. swallow exception from diagnoseCorruption

    @Ngone51
    Ngone51 committed Aug 2, 2021
    Configuration menu
    Copy the full SHA
    925491c View commit details
    Browse the repository at this point in the history

  38. ensure test stability

    @Ngone51
    Ngone51 committed Aug 2, 2021
    Configuration menu
    Copy the full SHA
    73b7b70 View commit details
    Browse the repository at this point in the history

  39. update version to 3.2.0

    @Ngone51
    Ngone51 committed Aug 2, 2021
    Configuration menu
    Copy the full SHA
    be116fb View commit details
    Browse the repository at this point in the history

  40. fix import style

    @Ngone51
    Ngone51 committed Aug 2, 2021
    Configuration menu
    Copy the full SHA
    64071ee View commit details
    Browse the repository at this point in the history

  41. add tests

    @Ngone51
    Ngone51 committed Aug 2, 2021
    Configuration menu
    Copy the full SHA
    a994c0d View commit details
    Browse the repository at this point in the history

  42. refactor bufIn

    @Ngone51
    Ngone51 committed Aug 2, 2021
    Configuration menu
    Copy the full SHA
    261f5ec View commit details
    Browse the repository at this point in the history

  43. use nano time

    @Ngone51
    Ngone51 committed Aug 2, 2021
    Configuration menu
    Copy the full SHA
    8d9db93 View commit details
    Browse the repository at this point in the history

  44. Use Option

    @Ngone51
    Ngone51 committed Aug 2, 2021
    Configuration menu
    Copy the full SHA
    6f72ab1 View commit details
    Browse the repository at this point in the history

  45. move Cause

    @Ngone51
    Ngone51 committed Aug 2, 2021
    Configuration menu
    Copy the full SHA
    e5e58d5 View commit details
    Browse the repository at this point in the history

  46. fix tests

    @Ngone51
    Ngone51 committed Aug 2, 2021
    Configuration menu
    Copy the full SHA
    7e4c91d View commit details
    Browse the repository at this point in the history

  47. fix

    @Ngone51
    Ngone51 committed Aug 2, 2021
    Configuration menu
    Copy the full SHA
    67262c9 View commit details
    Browse the repository at this point in the history

  48. address comment

    @Ngone51
    Ngone51 committed Aug 2, 2021
    Configuration menu
    Copy the full SHA
    ca1b058 View commit details
    Browse the repository at this point in the history