Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[SPARK-24763][SS] Remove redundant key data from value in streaming aggregation #21733

Closed
wants to merge 13 commits into from
Closed

[SPARK-24763][SS] Remove redundant key data from value in streaming aggregation #21733

wants to merge 13 commits into from

Commits on Jul 20, 2018

  1. [SPARK-24763][SS] Remove redundant key data from value in streaming a… 

    …ggregation

* add option to configure enabling new feature: remove redundant key data from value
* modify code to respect new option (turning on/off feature)
* modify tests to run tests with both on/off
* Add guard in OffsetSeqMetadata to prevent modifying option after executing query
    @HeartSaVioR
    HeartSaVioR committed Jul 20, 2018
    Copy the full SHA
    4252f41 View commit details
    Browse the repository at this point in the history

  2. Fix scala checkstyle issue

    @HeartSaVioR
    HeartSaVioR committed Jul 20, 2018
    Copy the full SHA
    941b88d View commit details
    Browse the repository at this point in the history

  3. Remove duplicating code, use configuration key instead of string literal

    @HeartSaVioR
    HeartSaVioR committed Jul 20, 2018
    Copy the full SHA
    abec57f View commit details
    Browse the repository at this point in the history

  4. Refine code change: introduce trait and classes to group duplicate me… 

    …thods
    @HeartSaVioR
    HeartSaVioR committed Jul 20, 2018
    Copy the full SHA
    977428c View commit details
    Browse the repository at this point in the history

  5. Change the strategy: "add new option" -> "apply by default, but keep … 

    …backward compatible"
    @HeartSaVioR
    HeartSaVioR committed Jul 20, 2018
    Copy the full SHA
    63dfb5d View commit details
    Browse the repository at this point in the history

  6. Add tests for StatefulOperatorsHelper as well

    @HeartSaVioR
    HeartSaVioR committed Jul 20, 2018
    Copy the full SHA
    e844636 View commit details
    Browse the repository at this point in the history

Commits on Aug 1, 2018

  1. WIP Address a part of review comments from @tdas 

    * TODO list
  * add docs
  * move StreamingAggregationStateManager to execution.streaming.state package
  * add iterator / remove in StreamingAggregationStateManager to remove restoreOriginRow
  * replace all the usages for direct call of store.xxx whenever state manager is available
    @HeartSaVioR
    HeartSaVioR committed Aug 1, 2018
    Copy the full SHA
    26701a3 View commit details
    Browse the repository at this point in the history

  2. WIP Address a part of review comments from @tdas 

    * TODO list
  * replace all the usages for direct call of store.xxx whenever state manager is available
  * add iterator / remove in StreamingAggregationStateManager to remove restoreOriginRow
  * add docs
    @HeartSaVioR
    HeartSaVioR committed Aug 1, 2018
    Copy the full SHA
    60c231e View commit details
    Browse the repository at this point in the history

  3. WIP Address a part of review comments from @tdas 

    * TODO list
  * add docs
    @HeartSaVioR
    HeartSaVioR committed Aug 1, 2018
    Copy the full SHA
    b4a3807 View commit details
    Browse the repository at this point in the history

Commits on Aug 8, 2018

  1. Address a part of review comments from @tdas 

    * add docs
    @HeartSaVioR
    HeartSaVioR committed Aug 8, 2018
    Copy the full SHA
    e0ee04a View commit details
    Browse the repository at this point in the history

Commits on Aug 9, 2018

  1. WIP address review comments from @tdas 

    TODO list
* add test which reads checkpoint from 2.3.1 and runs query
    @HeartSaVioR
    HeartSaVioR committed Aug 9, 2018
    Copy the full SHA
    8629f59 View commit details
    Browse the repository at this point in the history

  2. Address review comments from @tdas 

    * add test which restores query from Spark 2.3.1
    @HeartSaVioR
    HeartSaVioR committed Aug 9, 2018
    Copy the full SHA
    65801a6 View commit details
    Browse the repository at this point in the history

Commits on Aug 21, 2018

  1. Remove .crc files as following up @tdas guide

    @HeartSaVioR
    HeartSaVioR committed Aug 21, 2018
    Copy the full SHA
    19888ab View commit details
    Browse the repository at this point in the history