Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[SPARK-42585][CONNECT] Streaming of local relations #40827

Closed
wants to merge 61 commits into from
Closed

[SPARK-42585][CONNECT] Streaming of local relations #40827

wants to merge 61 commits into from

Commits on Apr 14, 2023

  1. Add an end-to-end test

    @MaxGekk
    MaxGekk committed Apr 14, 2023
    Configuration menu
    Copy the full SHA
    8f4c0d0 View commit details
    Browse the repository at this point in the history

Commits on Apr 15, 2023

  1. Add the SQL config spark.sql.session.localRelationCacheThreshold

    @MaxGekk
    MaxGekk committed Apr 15, 2023
    Configuration menu
    Copy the full SHA
    b43e79c View commit details
    Browse the repository at this point in the history

  2. Update createDataset()

    @MaxGekk
    MaxGekk committed Apr 15, 2023
    Configuration menu
    Copy the full SHA
    d89d004 View commit details
    Browse the repository at this point in the history

Commits on Apr 17, 2023

  1. Add the proto message: CachedLocalRelation

    @MaxGekk
    MaxGekk committed Apr 17, 2023
    Configuration menu
    Copy the full SHA
    4adee82 View commit details
    Browse the repository at this point in the history

  2. Re-gen relations_pb2.py and relations_pb2.pyi

    @MaxGekk
    MaxGekk committed Apr 17, 2023
    Configuration menu
    Copy the full SHA
    33a2baf View commit details
    Browse the repository at this point in the history

Commits on Apr 18, 2023

  1. Merge remote-tracking branch 'origin/master' into streaming-createDat… 

    …aFrame-2
    @MaxGekk
    MaxGekk committed Apr 18, 2023
    Configuration menu
    Copy the full SHA
    55bfac5 View commit details
    Browse the repository at this point in the history

  2. Re-gen relations_pb2.py and relations_pb2.pyi

    @MaxGekk
    MaxGekk committed Apr 18, 2023
    Configuration menu
    Copy the full SHA
    0a4639e View commit details
    Browse the repository at this point in the history

  3. Re-gen golden files for PlanGenerationTestSuite

    @MaxGekk
    MaxGekk committed Apr 18, 2023
    Configuration menu
    Copy the full SHA
    a78b7c0 View commit details
    Browse the repository at this point in the history

  4. Re-gen golden files

    @MaxGekk
    MaxGekk committed Apr 18, 2023
    Configuration menu
    Copy the full SHA
    3f22c9f View commit details
    Browse the repository at this point in the history

  5. Send the local relation to the remote cache

    @MaxGekk
    MaxGekk committed Apr 18, 2023
    Configuration menu
    Copy the full SHA
    74b409e View commit details
    Browse the repository at this point in the history

Commits on Apr 19, 2023

  1. Partial implementation

    @MaxGekk
    MaxGekk committed Apr 19, 2023
    Configuration menu
    Copy the full SHA
    526e696 View commit details
    Browse the repository at this point in the history

  2. Impl w/o exist checks

    @MaxGekk
    MaxGekk committed Apr 19, 2023
    Configuration menu
    Copy the full SHA
    c8b0bbc View commit details
    Browse the repository at this point in the history

  3. Bug fixes

    @MaxGekk
    MaxGekk committed Apr 19, 2023
    Configuration menu
    Copy the full SHA
    25c77ce View commit details
    Browse the repository at this point in the history

Commits on Apr 20, 2023

  1. Merge remote-tracking branch 'origin/master' into streaming-createDat… 

    …aFrame-2
    @MaxGekk
    MaxGekk committed Apr 20, 2023
    Configuration menu
    Copy the full SHA
    f603081 View commit details
    Browse the repository at this point in the history

  2. Re-gen relations_pb2.py

    @MaxGekk
    MaxGekk committed Apr 20, 2023
    Configuration menu
    Copy the full SHA
    b37b595 View commit details
    Browse the repository at this point in the history

  3. Bug fix: serialization

    @MaxGekk
    MaxGekk committed Apr 20, 2023
    Configuration menu
    Copy the full SHA
    88b0e47 View commit details
    Browse the repository at this point in the history

  4. Add a test to ArtifactManagerSuite

    @MaxGekk
    MaxGekk committed Apr 20, 2023
    Configuration menu
    Copy the full SHA
    09f6fc5 View commit details
    Browse the repository at this point in the history

  5. Add a test to ClientE2ETestSuite

    @MaxGekk
    MaxGekk committed Apr 20, 2023
    Configuration menu
    Copy the full SHA
    33b1650 View commit details
    Browse the repository at this point in the history

  6. Trigger build

    @MaxGekk
    MaxGekk committed Apr 20, 2023
    Configuration menu
    Copy the full SHA
    5932218 View commit details
    Browse the repository at this point in the history

  7. Reformat SparkConnectArtifactManager and SparkConnectPlanner

    @MaxGekk
    MaxGekk committed Apr 20, 2023
    Configuration menu
    Copy the full SHA
    d369cf4 View commit details
    Browse the repository at this point in the history

Commits on Apr 21, 2023

  1. Trigger build

    @MaxGekk
    MaxGekk committed Apr 21, 2023
    Configuration menu
    Copy the full SHA
    ba72916 View commit details
    Browse the repository at this point in the history

  2. Trigger build

    @MaxGekk
    MaxGekk committed Apr 21, 2023
    Configuration menu
    Copy the full SHA
    3d8d157 View commit details
    Browse the repository at this point in the history

Commits on Apr 22, 2023

  1. Merge remote-tracking branch 'origin/master' into streaming-createDat… 

    …aFrame-2
    @MaxGekk
    MaxGekk committed Apr 22, 2023
    Configuration menu
    Copy the full SHA
    856b5e7 View commit details
    Browse the repository at this point in the history

  2. Re-gen proto/relations_pb2.py

    @MaxGekk
    MaxGekk committed Apr 22, 2023
    Configuration menu
    Copy the full SHA
    bbc2632 View commit details
    Browse the repository at this point in the history

Commits on Apr 24, 2023

  1. Add the rpc endpoint: ArtifactStatus

    @MaxGekk
    MaxGekk committed Apr 24, 2023
    Configuration menu
    Copy the full SHA
    7af7bac View commit details
    Browse the repository at this point in the history

  2. Re-gen proto/base_pb2

    @MaxGekk
    MaxGekk committed Apr 24, 2023
    Configuration menu
    Copy the full SHA
    e6272b6 View commit details
    Browse the repository at this point in the history

  3. Implement SparkConnectArtifactStatusesHandler

    @MaxGekk
    MaxGekk committed Apr 24, 2023
    Configuration menu
    Copy the full SHA
    eec35b1 View commit details
    Browse the repository at this point in the history

  4. Fix compilation errors

    @MaxGekk
    MaxGekk committed Apr 24, 2023
    Configuration menu
    Copy the full SHA
    d88d68f View commit details
    Browse the repository at this point in the history

  5. Merge remote-tracking branch 'origin/master' into streaming-createDat… 

    …aFrame-2
    @MaxGekk
    MaxGekk committed Apr 24, 2023
    Configuration menu
    Copy the full SHA
    98b42e5 View commit details
    Browse the repository at this point in the history

  6. Fix coding style and compiler errors

    @MaxGekk
    MaxGekk committed Apr 24, 2023
    Configuration menu
    Copy the full SHA
    b6c9a9f View commit details
    Browse the repository at this point in the history

  7. Add the test suite: ArtifactStatusesHandlerSuite

    @MaxGekk
    MaxGekk committed Apr 24, 2023
    Configuration menu
    Copy the full SHA
    11e4f42 View commit details
    Browse the repository at this point in the history

Commits on Apr 25, 2023

  1. Bug fix: artifact has a prefix

    @MaxGekk
    MaxGekk committed Apr 25, 2023
    Configuration menu
    Copy the full SHA
    3e18eaf View commit details
    Browse the repository at this point in the history

  2. Is the artifact cached: check from the client.

    @MaxGekk
    MaxGekk committed Apr 25, 2023
    Configuration menu
    Copy the full SHA
    73d0236 View commit details
    Browse the repository at this point in the history

  3. Merge remote-tracking branch 'origin/master' into streaming-createDat… 

    …aFrame-2
    @MaxGekk
    MaxGekk committed Apr 25, 2023
    Configuration menu
    Copy the full SHA
    d4f603a View commit details
    Browse the repository at this point in the history

  4. Re-gen base_pb2.py

    @MaxGekk
    MaxGekk committed Apr 25, 2023
    Configuration menu
    Copy the full SHA
    e99bf32 View commit details
    Browse the repository at this point in the history

  5. Add user and session id to BlockId

    @MaxGekk
    MaxGekk committed Apr 25, 2023
    Configuration menu
    Copy the full SHA
    d37a24d View commit details
    Browse the repository at this point in the history

  6. Re-gen relations_pb2

    @MaxGekk
    MaxGekk committed Apr 25, 2023
    Configuration menu
    Copy the full SHA
    e11bd47 View commit details
    Browse the repository at this point in the history

  7. Reformat connect files

    @MaxGekk
    MaxGekk committed Apr 25, 2023
    Configuration menu
    Copy the full SHA
    7c03387 View commit details
    Browse the repository at this point in the history

Commits on Apr 26, 2023

  1. Merge remote-tracking branch 'origin/master' into streaming-createDat… 

    …aFrame-2
    @MaxGekk
    MaxGekk committed Apr 26, 2023
    Configuration menu
    Copy the full SHA
    f707659 View commit details
    Browse the repository at this point in the history

  2. Re-gen relations_pb2

    @MaxGekk
    MaxGekk committed Apr 26, 2023
    Configuration menu
    Copy the full SHA
    6e86b1b View commit details
    Browse the repository at this point in the history

  3. Clean up cache entries in the block manager.

    @MaxGekk
    MaxGekk committed Apr 26, 2023
    Configuration menu
    Copy the full SHA
    e1a5f70 View commit details
    Browse the repository at this point in the history

  4. Fix scala 2.13 errors

    @MaxGekk
    MaxGekk committed Apr 26, 2023
    Configuration menu
    Copy the full SHA
    db404f4 View commit details
    Browse the repository at this point in the history

  5. Trigger build

    @MaxGekk
    MaxGekk committed Apr 26, 2023
    Configuration menu
    Copy the full SHA
    1e93ba4 View commit details
    Browse the repository at this point in the history

Commits on Apr 27, 2023

  1. Trigger build

    @MaxGekk
    MaxGekk committed Apr 27, 2023
    Configuration menu
    Copy the full SHA
    68d1144 View commit details
    Browse the repository at this point in the history

  2. Use Arrays.asList

    @MaxGekk
    MaxGekk committed Apr 27, 2023
    Configuration menu
    Copy the full SHA
    f73730a View commit details
    Browse the repository at this point in the history

  3. Address Wenchen's review comments

    @MaxGekk
    MaxGekk committed Apr 27, 2023
    Configuration menu
    Copy the full SHA
    9cf03de View commit details
    Browse the repository at this point in the history

  4. Re-gen base_pb2

    @MaxGekk
    MaxGekk committed Apr 27, 2023
    Configuration menu
    Copy the full SHA
    be76d83 View commit details
    Browse the repository at this point in the history

Commits on Apr 28, 2023

  1. Trigger build

    @MaxGekk
    MaxGekk committed Apr 28, 2023
    Configuration menu
    Copy the full SHA
    e1cc5f6 View commit details
    Browse the repository at this point in the history

  2. Trigger build

    @MaxGekk
    MaxGekk committed Apr 28, 2023
    Configuration menu
    Copy the full SHA
    5952148 View commit details
    Browse the repository at this point in the history

  3. Implement remove cache in the unified way

    @MaxGekk
    MaxGekk committed Apr 28, 2023
    Configuration menu
    Copy the full SHA
    b7bbad3 View commit details
    Browse the repository at this point in the history

  4. Remove a duplicate test

    @MaxGekk
    MaxGekk committed Apr 28, 2023
    Configuration menu
    Copy the full SHA
    ea0cc60 View commit details
    Browse the repository at this point in the history

  5. Merge remote-tracking branch 'origin/master' into streaming-createDat… 

    …aFrame-2
    @MaxGekk
    MaxGekk committed Apr 28, 2023
    Configuration menu
    Copy the full SHA
    22841da View commit details
    Browse the repository at this point in the history

  6. Revert "Remove a duplicate test" 

    This reverts commit ea0cc60.
    @MaxGekk
    MaxGekk committed Apr 28, 2023
    Configuration menu
    Copy the full SHA
    49c0c10 View commit details
    Browse the repository at this point in the history

  7. Don't tell to the master about caches

    @MaxGekk
    MaxGekk committed Apr 28, 2023
    Configuration menu
    Copy the full SHA
    0e87751 View commit details
    Browse the repository at this point in the history

  8. Release the read lock of the caches

    @MaxGekk
    MaxGekk committed Apr 28, 2023
    Configuration menu
    Copy the full SHA
    61e8e2e View commit details
    Browse the repository at this point in the history

  9. Make the artifact test more stable

    @MaxGekk
    MaxGekk committed Apr 28, 2023
    Configuration menu
    Copy the full SHA
    7dc4d1c View commit details
    Browse the repository at this point in the history

Commits on Apr 29, 2023

  1. Reformat SparkConnectPlanner

    @MaxGekk
    MaxGekk committed Apr 29, 2023
    Configuration menu
    Copy the full SHA
    1ec8fb5 View commit details
    Browse the repository at this point in the history

  2. Merge remote-tracking branch 'origin/master' into streaming-createDat… 

    …aFrame-2
    @MaxGekk
    MaxGekk committed Apr 29, 2023
    Configuration menu
    Copy the full SHA
    900867e View commit details
    Browse the repository at this point in the history

  3. Reformat SparkConnectService

    @MaxGekk
    MaxGekk committed Apr 29, 2023
    Configuration menu
    Copy the full SHA
    710d224 View commit details
    Browse the repository at this point in the history

Commits on May 1, 2023

  1. Remove the dependency of SQLConf

    @MaxGekk
    MaxGekk committed May 1, 2023
    Configuration menu
    Copy the full SHA
    46eb354 View commit details
    Browse the repository at this point in the history

  2. Trigger build

    @MaxGekk
    MaxGekk committed May 1, 2023
    Configuration menu
    Copy the full SHA
    2351069 View commit details
    Browse the repository at this point in the history