Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[SPARK-38809][SS] Implement option to skip null values in symmetric hash implementation of stream-stream joins #36090

Closed
wants to merge 6 commits into from
Closed

[SPARK-38809][SS] Implement option to skip null values in symmetric hash implementation of stream-stream joins #36090

wants to merge 6 commits into from

Commits on Apr 6, 2022

  1. [SPARK-38809][SS] Implement option to skip null values in symmetric h… 

    …ash impl of stream-stream joins

In the symmetric has join state manager, we can receive entries with null values for a key and that caused the `removeByValue` and get iterators to fail and run into the NullPointerException. This is possible if the state recovered is written from a old spark version or its corrupted on disk. Since we don't have a utility to query this state, we would like to provide a conf option to skip nulls for the symmetric hash impl in stream stream joins.
    @anishshri-db
    anishshri-db committed Apr 6, 2022
    Configuration menu
    Copy the full SHA
    3baf99c View commit details
    Browse the repository at this point in the history

  2. Change to fix comments in test

    @anishshri-db
    anishshri-db committed Apr 6, 2022
    Configuration menu
    Copy the full SHA
    066a03b View commit details
    Browse the repository at this point in the history

Commits on Apr 7, 2022

  1. Change to address Jerry's comments

    @anishshri-db
    anishshri-db committed Apr 7, 2022
    Configuration menu
    Copy the full SHA
    f6371e0 View commit details
    Browse the repository at this point in the history

  2. Change to address Jungtaek's comments

    @anishshri-db
    anishshri-db committed Apr 7, 2022
    Configuration menu
    Copy the full SHA
    364790d View commit details
    Browse the repository at this point in the history

  3. Change to address Jungtaek's comments

    @anishshri-db
    anishshri-db committed Apr 7, 2022
    Configuration menu
    Copy the full SHA
    0d5ef66 View commit details
    Browse the repository at this point in the history

Commits on Apr 8, 2022

  1. Change to add version to conf

    @anishshri-db
    anishshri-db committed Apr 8, 2022
    Configuration menu
    Copy the full SHA
    45dee14 View commit details
    Browse the repository at this point in the history