Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Exploring streaming #133

Merged
merged 50 commits into from
Apr 21, 2022
Merged

Exploring streaming #133

merged 50 commits into from
Apr 21, 2022

Commits on Feb 21, 2022

  1. added withSparkStreaming function and example. Let's see if something… 

    … like this could form the basis of easy streaming support :)
    @Jolanrensen
    Jolanrensen committed Feb 21, 2022
    Configuration menu
    Copy the full SHA
    b24172f View commit details
    Browse the repository at this point in the history

  2. makes withSparkStreaming reuse the normal withSpark

    @Jolanrensen
    Jolanrensen committed Feb 21, 2022
    Configuration menu
    Copy the full SHA
    a378070 View commit details
    Browse the repository at this point in the history

Commits on Feb 24, 2022

  1. removed sc.stop()

    @Jolanrensen
    Jolanrensen committed Feb 24, 2022
    Configuration menu
    Copy the full SHA
    47e1ccb View commit details
    Browse the repository at this point in the history

Commits on Mar 7, 2022

  1. added withSparkStreaming function and example. Let's see if something… 

    … like this could form the basis of easy streaming support :)
    @Jolanrensen
    Jolanrensen committed Mar 7, 2022
    Configuration menu
    Copy the full SHA
    6c1b0d9 View commit details
    Browse the repository at this point in the history

  2. makes withSparkStreaming reuse the normal withSpark

    @Jolanrensen
    Jolanrensen committed Mar 7, 2022
    Configuration menu
    Copy the full SHA
    95a9563 View commit details
    Browse the repository at this point in the history

  3. removed sc.stop()

    @Jolanrensen
    Jolanrensen committed Mar 7, 2022
    Configuration menu
    Copy the full SHA
    9d1450b View commit details
    Browse the repository at this point in the history

  4. Merge remote-tracking branch 'origin/exploring-streaming' into explor… 

    …ing-streaming

# Conflicts:
#	kotlin-spark-api/3.2/src/main/kotlin/org/jetbrains/kotlinx/spark/api/SparkHelper.kt
    @Jolanrensen
    Jolanrensen committed Mar 7, 2022
    Configuration menu
    Copy the full SHA
    60569f9 View commit details
    Browse the repository at this point in the history

  5. fixed merge

    @Jolanrensen
    Jolanrensen committed Mar 7, 2022
    Configuration menu
    Copy the full SHA
    f2ce000 View commit details
    Browse the repository at this point in the history

  6. working on tests

    @Jolanrensen
    Jolanrensen committed Mar 7, 2022
    Configuration menu
    Copy the full SHA
    cdf7296 View commit details
    Browse the repository at this point in the history

Commits on Mar 8, 2022

  1. added timeout and working streaming test

    @Jolanrensen
    Jolanrensen committed Mar 8, 2022
    Configuration menu
    Copy the full SHA
    330536c View commit details
    Browse the repository at this point in the history

  2. added timeout and working streaming test

    @Jolanrensen
    Jolanrensen committed Mar 8, 2022
    Configuration menu
    Copy the full SHA
    d2e792a View commit details
    Browse the repository at this point in the history

Commits on Mar 17, 2022

  1. still exploring, just pushing to keep my work safe :)

    @Jolanrensen
    Jolanrensen committed Mar 17, 2022
    Configuration menu
    Copy the full SHA
    4222a03 View commit details
    Browse the repository at this point in the history

  2. still exploring, just pushing to keep my work safe :)

    @Jolanrensen
    Jolanrensen committed Mar 17, 2022
    Configuration menu
    Copy the full SHA
    5769e92 View commit details
    Browse the repository at this point in the history

  3. pairs are recognized too

    @Jolanrensen
    Jolanrensen committed Mar 17, 2022
    Configuration menu
    Copy the full SHA
    14278ff View commit details
    Browse the repository at this point in the history

  4. and tuples cause why not

    @Jolanrensen
    Jolanrensen committed Mar 17, 2022
    Configuration menu
    Copy the full SHA
    eec03cb View commit details
    Browse the repository at this point in the history

Commits on Mar 21, 2022

  1. added Option(al) converters, testing fakeClassTag(). Working with str… 

    …eaming pair-functions, decided to go tuple-first instead of arities
    @Jolanrensen
    Jolanrensen committed Mar 21, 2022
    Configuration menu
    Copy the full SHA
    bc9fd3e View commit details
    Browse the repository at this point in the history

Commits on Mar 31, 2022

  1. removed commented out code, fixed checkpointPath (needs cleaning up c… 

    …ode), new notation for entering KSparkSession scope inside streaming scope
    @Jolanrensen
    Jolanrensen committed Mar 31, 2022
    Configuration menu
    Copy the full SHA
    ea3e7d8 View commit details
    Browse the repository at this point in the history

Commits on Apr 1, 2022

  1. Merge branch 'tuple-first' into streaming+tuples 

    # Conflicts:
#	kotlin-spark-api/3.2/src/main/kotlin/org/jetbrains/kotlinx/spark/api/Conversions.kt
    @Jolanrensen
    Jolanrensen committed Apr 1, 2022
    Configuration menu
    Copy the full SHA
    d672ffb View commit details
    Browse the repository at this point in the history

  2. small streaming updates

    @Jolanrensen
    Jolanrensen committed Apr 1, 2022
    Configuration menu
    Copy the full SHA
    f523d9d View commit details
    Browse the repository at this point in the history

  3. Merge branch 'tuple-first' into streaming+tuples

    @Jolanrensen
    Jolanrensen committed Apr 1, 2022
    Configuration menu
    Copy the full SHA
    62daa82 View commit details
    Browse the repository at this point in the history

  4. created temporary branch merging tuples with streaming. Will merge in… 

    …to normal "exploring-streaming" after vacation
    @Jolanrensen
    Jolanrensen committed Apr 1, 2022
    Configuration menu
    Copy the full SHA
    2708a5c View commit details
    Browse the repository at this point in the history

Commits on Apr 11, 2022

  1. Merge branch 'spark-3.2' into streaming+tuples

    @Jolanrensen
    Jolanrensen committed Apr 11, 2022
    Configuration menu
    Copy the full SHA
    1e8ab36 View commit details
    Browse the repository at this point in the history

  2. Merge branch 'spark-3.2' into exploring-streaming

    @Jolanrensen
    Jolanrensen committed Apr 11, 2022
    Configuration menu
    Copy the full SHA
    52e5b0d View commit details
    Browse the repository at this point in the history

Commits on Apr 12, 2022

  1. working on testing, not yet finished

    @Jolanrensen
    Jolanrensen committed Apr 12, 2022
    Configuration menu
    Copy the full SHA
    c70c0f3 View commit details
    Browse the repository at this point in the history

Commits on Apr 13, 2022

  1. updated to kotlin 1.6.20, refactored withSparkStreaming and fixed tests

    @Jolanrensen
    Jolanrensen committed Apr 13, 2022
    Configuration menu
    Copy the full SHA
    02d50cc View commit details
    Browse the repository at this point in the history

Commits on Apr 14, 2022

  1. added conversions for Option(al), State. 

    Added DataStreamWriter forEachBatch helper
added KotlinStatefulNetworkCount.kt
cleaning up
    @Jolanrensen
    Jolanrensen committed Apr 14, 2022
    Configuration menu
    Copy the full SHA
    219b949 View commit details
    Browse the repository at this point in the history

  2. added toDataFrame conversions for RDDs 

    added KotlinSqlNetworkWordCount.kt
    @Jolanrensen
    Jolanrensen committed Apr 14, 2022
    Configuration menu
    Copy the full SHA
    59f658e View commit details
    Browse the repository at this point in the history

  3. added toDataFrame conversions for RDDs 

    added KotlinSqlNetworkWordCount.kt
    @Jolanrensen
    Jolanrensen committed Apr 14, 2022
    Configuration menu
    Copy the full SHA
    547ab14 View commit details
    Browse the repository at this point in the history

  4. more tests

    @Jolanrensen
    Jolanrensen committed Apr 14, 2022
    Configuration menu
    Copy the full SHA
    6a5fce4 View commit details
    Browse the repository at this point in the history

  5. added kafka test, but issue with kotest extension

    @Jolanrensen
    Jolanrensen committed Apr 14, 2022
    Configuration menu
    Copy the full SHA
    9449b9f View commit details
    Browse the repository at this point in the history

Commits on Apr 19, 2022

  1. added embedded kafka so kafka tests work

    @Jolanrensen
    Jolanrensen committed Apr 19, 2022
    Configuration menu
    Copy the full SHA
    70949ef View commit details
    Browse the repository at this point in the history

  2. Optional and nullable options for updateStateByKey, changing port for… 

    … kafka test
    @Jolanrensen
    Jolanrensen committed Apr 19, 2022
    Configuration menu
    Copy the full SHA
    eaf13ce View commit details
    Browse the repository at this point in the history

  3. qodana suggestions

    @Jolanrensen
    Jolanrensen committed Apr 19, 2022
    Configuration menu
    Copy the full SHA
    8ccf5ea View commit details
    Browse the repository at this point in the history

  4. Merge branch 'spark-3.2' into exploring-streaming

    @Jolanrensen
    Jolanrensen committed Apr 19, 2022
    Configuration menu
    Copy the full SHA
    db566be View commit details
    Browse the repository at this point in the history

  5. qodana suggestions

    @Jolanrensen
    Jolanrensen committed Apr 19, 2022
    Configuration menu
    Copy the full SHA
    8e3b952 View commit details
    Browse the repository at this point in the history

  6. let's see if adding a container does anything

    @Jolanrensen
    Jolanrensen committed Apr 19, 2022
    Configuration menu
    Copy the full SHA
    1af4c04 View commit details
    Browse the repository at this point in the history

Commits on Apr 20, 2022

  1. changing to ip 0.0.0.0

    @Jolanrensen
    Jolanrensen committed Apr 20, 2022
    Configuration menu
    Copy the full SHA
    e4d6e2a View commit details
    Browse the repository at this point in the history

  2. changing to ip localhost

    @Jolanrensen
    Jolanrensen committed Apr 20, 2022
    Configuration menu
    Copy the full SHA
    25464d8 View commit details
    Browse the repository at this point in the history

  3. attempt to add exclusion for kafka streaming test for github

    @Jolanrensen
    Jolanrensen committed Apr 20, 2022
    Configuration menu
    Copy the full SHA
    9901c5b View commit details
    Browse the repository at this point in the history

  4. attempting to exclude entire file

    @Jolanrensen
    Jolanrensen committed Apr 20, 2022
    Configuration menu
    Copy the full SHA
    1090e2b View commit details
    Browse the repository at this point in the history

  5. attempting to exclude entire file

    @Jolanrensen
    Jolanrensen committed Apr 20, 2022
    Configuration menu
    Copy the full SHA
    729279e View commit details
    Browse the repository at this point in the history

  6. attempting to exclude entire file

    @Jolanrensen
    Jolanrensen committed Apr 20, 2022
    Configuration menu
    Copy the full SHA
    9b5c1fb View commit details
    Browse the repository at this point in the history

  7. attempting to exclude entire file

    @Jolanrensen
    Jolanrensen committed Apr 20, 2022
    Configuration menu
    Copy the full SHA
    290fc9e View commit details
    Browse the repository at this point in the history

Commits on Apr 21, 2022

  1. exclusion kafka works!

    @Jolanrensen
    Jolanrensen committed Apr 21, 2022
    Configuration menu
    Copy the full SHA
    fdee3a2 View commit details
    Browse the repository at this point in the history

  2. exclusion kafka works!

    @Jolanrensen
    Jolanrensen committed Apr 21, 2022
    Configuration menu
    Copy the full SHA
    4dfb747 View commit details
    Browse the repository at this point in the history

  3. updating readme and example

    @Jolanrensen
    Jolanrensen committed Apr 21, 2022
    Configuration menu
    Copy the full SHA
    54b9d10 View commit details
    Browse the repository at this point in the history

  4. attempt to add qodana scan action to github actions

    @Jolanrensen
    Jolanrensen committed Apr 21, 2022
    Configuration menu
    Copy the full SHA
    f694d07 View commit details
    Browse the repository at this point in the history

  5. removed qodana app, let's try

    @Jolanrensen
    Jolanrensen committed Apr 21, 2022
    Configuration menu
    Copy the full SHA
    299fb75 View commit details
    Browse the repository at this point in the history

  6. removed todo test and updated unused imports

    @Jolanrensen
    Jolanrensen committed Apr 21, 2022
    Configuration menu
    Copy the full SHA
    f83727d View commit details
    Browse the repository at this point in the history

  7. last cleanups

    @Jolanrensen
    Jolanrensen committed Apr 21, 2022
    Configuration menu
    Copy the full SHA
    4ece47e View commit details
    Browse the repository at this point in the history