Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

TSDB: Implement downsampling on time-series indices #85708

Merged
merged 78 commits into from
May 25, 2022
Merged

TSDB: Implement downsampling on time-series indices #85708

merged 78 commits into from
May 25, 2022

Commits on Apr 5, 2022

  1. Remove feature flag es.rollup_v2_feature_flag_enabled

    @csoulios
    csoulios committed Apr 5, 2022
    Configuration menu
    Copy the full SHA
    94fb431 View commit details
    Browse the repository at this point in the history

  2. RollupShardIndexer

    @csoulios
    csoulios committed Apr 5, 2022
    Configuration menu
    Copy the full SHA
    b8e9d2b View commit details
    Browse the repository at this point in the history

  3. Cleanup

    @csoulios
    csoulios committed Apr 5, 2022
    Configuration menu
    Copy the full SHA
    7b7d06b View commit details
    Browse the repository at this point in the history

  4. Test should not merge segments

    @csoulios
    csoulios committed Apr 5, 2022
    Configuration menu
    Copy the full SHA
    75ce70c View commit details
    Browse the repository at this point in the history

  5. Refactoring code

    @csoulios
    csoulios committed Apr 5, 2022
    Configuration menu
    Copy the full SHA
    98627b1 View commit details
    Browse the repository at this point in the history

  6. Fix wrong timestamp order

    @csoulios
    csoulios committed Apr 5, 2022
    Configuration menu
    Copy the full SHA
    3215864 View commit details
    Browse the repository at this point in the history

  7. Cleanup

    @csoulios
    csoulios committed Apr 5, 2022
    Configuration menu
    Copy the full SHA
    34f42bd View commit details
    Browse the repository at this point in the history

  8. Cleanup

    @csoulios
    csoulios committed Apr 5, 2022
    Configuration menu
    Copy the full SHA
    29bc089 View commit details
    Browse the repository at this point in the history

  9. Removed CompressingOfflineSorter implementation 

    Not needed for timeseries indices
    @csoulios
    csoulios committed Apr 5, 2022
    Configuration menu
    Copy the full SHA
    939a1c0 View commit details
    Browse the repository at this point in the history

  10. class to record

    @csoulios
    csoulios committed Apr 5, 2022
    Configuration menu
    Copy the full SHA
    23d39a8 View commit details
    Browse the repository at this point in the history

  11. Added more tests

    @csoulios
    csoulios committed Apr 5, 2022
    Configuration menu
    Copy the full SHA
    0f40815 View commit details
    Browse the repository at this point in the history

  12. Refactored rollup action config 

    Removed most of the configuration, which is extracted from
the index mapping.

Modified TransportRollupAction to extract the rollup config from
the field caps
    @csoulios
    csoulios committed Apr 5, 2022
    Configuration menu
    Copy the full SHA
    7a44a68 View commit details
    Browse the repository at this point in the history

  13. Refactored rollup action config 

    Removed most of the configuration, which is extracted from
the index mapping.

Modified TransportRollupAction to extract the rollup config from
the field caps.
    @csoulios
    csoulios committed Apr 5, 2022
    Configuration menu
    Copy the full SHA
    b94e82d View commit details
    Browse the repository at this point in the history

  14. Update docs/changelog/85708.yaml

    @csoulios
    csoulios committed Apr 5, 2022
    Configuration menu
    Copy the full SHA
    e514fbb View commit details
    Browse the repository at this point in the history

  15. Version updates

    @csoulios
    csoulios committed Apr 5, 2022
    Configuration menu
    Copy the full SHA
    219da73 View commit details
    Browse the repository at this point in the history

Commits on Apr 6, 2022

  1. CI Fixes

    @csoulios
    csoulios committed Apr 6, 2022
    Configuration menu
    Copy the full SHA
    1a385e7 View commit details
    Browse the repository at this point in the history

  2. checkstyle

    @csoulios
    csoulios committed Apr 6, 2022
    Configuration menu
    Copy the full SHA
    633abac View commit details
    Browse the repository at this point in the history

  3. checkstyle

    @csoulios
    csoulios committed Apr 6, 2022
    Configuration menu
    Copy the full SHA
    d4daeb8 View commit details
    Browse the repository at this point in the history

  4. Skip some failing tests 

    I will deal with them later
    @csoulios
    csoulios committed Apr 6, 2022
    Configuration menu
    Copy the full SHA
    496e00f View commit details
    Browse the repository at this point in the history

  5. Fix broken test with wrong index sort order

    @csoulios
    csoulios committed Apr 6, 2022
    Configuration menu
    Copy the full SHA
    5a0642c View commit details
    Browse the repository at this point in the history

  6. Fix ILM test

    @csoulios
    csoulios committed Apr 6, 2022
    Configuration menu
    Copy the full SHA
    d0222f0 View commit details
    Browse the repository at this point in the history

Commits on Apr 12, 2022

  1. Merge branch 'master' into tsdb-rollup

    @csoulios
    csoulios committed Apr 12, 2022
    Configuration menu
    Copy the full SHA
    b8bdcf9 View commit details
    Browse the repository at this point in the history

  2. Enable test

    @csoulios
    csoulios committed Apr 12, 2022
    Configuration menu
    Copy the full SHA
    d9c6f6e View commit details
    Browse the repository at this point in the history

Commits on Apr 15, 2022

  1. Update docs/changelog/85708.yaml

    @csoulios
    csoulios committed Apr 15, 2022
    Configuration menu
    Copy the full SHA
    ff864f2 View commit details
    Browse the repository at this point in the history

Commits on Apr 18, 2022

  1. Merge branch 'master' into tsdb-rollup

    @csoulios
    csoulios committed Apr 18, 2022
    Configuration menu
    Copy the full SHA
    0464806 View commit details
    Browse the repository at this point in the history

  2. Pull timestamp from AggregationExecutionContext

    @csoulios
    csoulios committed Apr 18, 2022
    Configuration menu
    Copy the full SHA
    643962f View commit details
    Browse the repository at this point in the history

  3. Remove time_zone param from rollup config

    @csoulios
    csoulios committed Apr 18, 2022
    Configuration menu
    Copy the full SHA
    d905038 View commit details
    Browse the repository at this point in the history

  4. Fix build failure

    @csoulios
    csoulios committed Apr 18, 2022
    Configuration menu
    Copy the full SHA
    29b77bc View commit details
    Browse the repository at this point in the history

  5. Merge branch 'master' into tsdb-rollup

    @csoulios
    csoulios committed Apr 18, 2022
    Configuration menu
    Copy the full SHA
    6237029 View commit details
    Browse the repository at this point in the history

  6. Added validations for rollup request parameters 

    Clean up handling of the temporary index name
Fixes elastic#65769
    @csoulios
    csoulios committed Apr 18, 2022
    Configuration menu
    Copy the full SHA
    a3e3e42 View commit details
    Browse the repository at this point in the history

  7. Update docs/changelog/85708.yaml

    @csoulios
    csoulios committed Apr 18, 2022
    Configuration menu
    Copy the full SHA
    0abece8 View commit details
    Browse the repository at this point in the history

  8. Set the number of shards and replicas 

    of the source index to the rollup index
    @csoulios
    csoulios committed Apr 18, 2022
    Configuration menu
    Copy the full SHA
    5dc065c View commit details
    Browse the repository at this point in the history

Commits on Apr 19, 2022

  1. Validate that index is read-only before rolling up

    @csoulios
    csoulios committed Apr 19, 2022
    Configuration menu
    Copy the full SHA
    6a8976a View commit details
    Browse the repository at this point in the history

  2. Delete source index after it has been downsampled

    @csoulios
    csoulios committed Apr 19, 2022
    Configuration menu
    Copy the full SHA
    d6adb3c View commit details
    Browse the repository at this point in the history

  3. Added test and fix for rolling up data stream idx

    @csoulios
    csoulios committed Apr 19, 2022
    Configuration menu
    Copy the full SHA
    4492090 View commit details
    Browse the repository at this point in the history

  4. minor change

    @csoulios
    csoulios committed Apr 19, 2022
    Configuration menu
    Copy the full SHA
    fc255f6 View commit details
    Browse the repository at this point in the history

  5. More changes

    @csoulios
    csoulios committed Apr 19, 2022
    Configuration menu
    Copy the full SHA
    2bc4600 View commit details
    Browse the repository at this point in the history

  6. Set max as default_metric for gauge metrics

    @csoulios
    csoulios committed Apr 19, 2022
    Configuration menu
    Copy the full SHA
    b10254d View commit details
    Browse the repository at this point in the history

Commits on Apr 20, 2022

  1. Merge branch 'master' into tsdb-rollup

    @csoulios
    csoulios committed Apr 20, 2022
    Configuration menu
    Copy the full SHA
    405915b View commit details
    Browse the repository at this point in the history

  2. Minor change to address reviewer comments

    @csoulios
    csoulios committed Apr 20, 2022
    Configuration menu
    Copy the full SHA
    d5de1aa View commit details
    Browse the repository at this point in the history

  3. Tidy up code for data streams

    @csoulios
    csoulios committed Apr 20, 2022
    Configuration menu
    Copy the full SHA
    bef7341 View commit details
    Browse the repository at this point in the history

  4. Changed hashCode

    @csoulios
    csoulios committed Apr 20, 2022
    Configuration menu
    Copy the full SHA
    af006e4 View commit details
    Browse the repository at this point in the history

Commits on Apr 26, 2022

  1. Merge branch 'master' into tsdb-rollup

    @csoulios
    csoulios committed Apr 26, 2022
    Configuration menu
    Copy the full SHA
    a1988a9 View commit details
    Browse the repository at this point in the history

  2. Return statement after error:

    @csoulios
    csoulios committed Apr 26, 2022
    Configuration menu
    Copy the full SHA
    cd23fd7 View commit details
    Browse the repository at this point in the history

  3. Merge branch 'master' into tsdb-rollup

    @csoulios
    csoulios committed Apr 26, 2022
    Configuration menu
    Copy the full SHA
    7973ca2 View commit details
    Browse the repository at this point in the history

Commits on Apr 29, 2022

  1. Merge branch 'master' into tsdb-rollup

    @csoulios
    csoulios committed Apr 29, 2022
    Configuration menu
    Copy the full SHA
    98fd352 View commit details
    Browse the repository at this point in the history

Commits on May 3, 2022

  1. Fix compilation error

    @csoulios
    csoulios committed May 3, 2022
    Configuration menu
    Copy the full SHA
    c46b126 View commit details
    Browse the repository at this point in the history

  2. Merge branch 'master' into tsdb-rollup

    @csoulios
    csoulios committed May 3, 2022
    Configuration menu
    Copy the full SHA
    fb064c6 View commit details
    Browse the repository at this point in the history

Commits on May 9, 2022

  1. Merge branch 'master' into tsdb-rollup

    @csoulios
    csoulios committed May 9, 2022
    Configuration menu
    Copy the full SHA
    3167f92 View commit details
    Browse the repository at this point in the history

  2. Fix broken test

    @csoulios
    csoulios committed May 9, 2022
    Configuration menu
    Copy the full SHA
    ed94943 View commit details
    Browse the repository at this point in the history

  3. Merge branch 'master' into tsdb-rollup

    @csoulios
    csoulios committed May 9, 2022
    Configuration menu
    Copy the full SHA
    48796c2 View commit details
    Browse the repository at this point in the history

Commits on May 10, 2022

  1. Merge branch 'master' into tsdb-rollup

    @csoulios
    csoulios committed May 10, 2022
    Configuration menu
    Copy the full SHA
    78d110b View commit details
    Browse the repository at this point in the history

  2. Compute last_value for metric fields

    @csoulios
    csoulios committed May 10, 2022
    Configuration menu
    Copy the full SHA
    f41343b View commit details
    Browse the repository at this point in the history

Commits on May 12, 2022

  1. Removed temporary rollup index 

    Rollups will be stored in the rollup index directly
    @csoulios
    csoulios committed May 12, 2022
    Configuration menu
    Copy the full SHA
    286cdbd View commit details
    Browse the repository at this point in the history

  2. Refresh the rollup index in the end

    @csoulios
    csoulios committed May 12, 2022
    Configuration menu
    Copy the full SHA
    ab1a4d8 View commit details
    Browse the repository at this point in the history

  3. Use randomValueOtherThan

    @csoulios
    csoulios committed May 12, 2022
    Configuration menu
    Copy the full SHA
    c22b6ba View commit details
    Browse the repository at this point in the history

  4. Do not serialize constants in RollupActionConfig

    @csoulios
    csoulios committed May 12, 2022
    Configuration menu
    Copy the full SHA
    4327b17 View commit details
    Browse the repository at this point in the history

  5. Add tsdb feature flag

    @csoulios
    csoulios committed May 12, 2022
    Configuration menu
    Copy the full SHA
    86c3347 View commit details
    Browse the repository at this point in the history

Commits on May 16, 2022

  1. Merge branch 'master' into tsdb-rollup

    @csoulios
    csoulios committed May 16, 2022
    Configuration menu
    Copy the full SHA
    ed51704 View commit details
    Browse the repository at this point in the history

  2. minor change

    @csoulios
    csoulios committed May 16, 2022
    Configuration menu
    Copy the full SHA
    22a7589 View commit details
    Browse the repository at this point in the history

Commits on May 17, 2022

  1. Perform the following actions in one state update: 

    - Replace source with rollup in the datastream
- Delete source index
- Mark rollup index as status=success

The above actions will be performed in a single atomic step
    @csoulios
    csoulios committed May 17, 2022
    Configuration menu
    Copy the full SHA
    ee42ef5 View commit details
    Browse the repository at this point in the history

Commits on May 19, 2022

  1. Perform batched cluster state update

    @csoulios
    csoulios committed May 19, 2022
    Configuration menu
    Copy the full SHA
    d7d9b72 View commit details
    Browse the repository at this point in the history

  2. Cleanup the mess

    @csoulios
    csoulios committed May 19, 2022
    Configuration menu
    Copy the full SHA
    332bef6 View commit details
    Browse the repository at this point in the history

  3. Merge branch 'master' into tsdb-rollup

    @csoulios
    csoulios committed May 19, 2022
    Configuration menu
    Copy the full SHA
    17eb72b View commit details
    Browse the repository at this point in the history

  4. More cleanup

    @csoulios
    csoulios committed May 19, 2022
    Configuration menu
    Copy the full SHA
    550a368 View commit details
    Browse the repository at this point in the history

  5. Added feature flag to security tests

    @csoulios
    csoulios committed May 19, 2022
    Configuration menu
    Copy the full SHA
    270ea69 View commit details
    Browse the repository at this point in the history

  6. Added feature flag to build files

    @csoulios
    csoulios committed May 19, 2022
    Configuration menu
    Copy the full SHA
    7197a5f View commit details
    Browse the repository at this point in the history

  7. Reverted wrong version bump

    @csoulios
    csoulios committed May 19, 2022
    Configuration menu
    Copy the full SHA
    8b96364 View commit details
    Browse the repository at this point in the history

Commits on May 20, 2022

  1. Add the number of indexed docs to the shard response

    @csoulios
    csoulios committed May 20, 2022
    Configuration menu
    Copy the full SHA
    839a29d View commit details
    Browse the repository at this point in the history

  2. Added javadoc

    @csoulios
    csoulios committed May 20, 2022
    Configuration menu
    Copy the full SHA
    de6b766 View commit details
    Browse the repository at this point in the history

Commits on May 23, 2022

  1. Merge branch 'master' into tsdb-rollup

    @csoulios
    csoulios committed May 23, 2022
    Configuration menu
    Copy the full SHA
    0f3f827 View commit details
    Browse the repository at this point in the history

  2. Changes to rollup index: 

    - Set refresh_interval to -1
- Refresh index before adding to data stream
- Force merge index in the end
    @csoulios
    csoulios committed May 23, 2022
    Configuration menu
    Copy the full SHA
    0ff80dc View commit details
    Browse the repository at this point in the history

  3. Remove org.elasticsearch.rollup from module-info

    @csoulios
    csoulios committed May 23, 2022
    Configuration menu
    Copy the full SHA
    7c4c441 View commit details
    Browse the repository at this point in the history

  4. Merge branch 'master' into tsdb-rollup

    @csoulios
    csoulios committed May 23, 2022
    Configuration menu
    Copy the full SHA
    5856f1d View commit details
    Browse the repository at this point in the history

  5. Added more validation for source index

    @csoulios
    csoulios committed May 23, 2022
    Configuration menu
    Copy the full SHA
    1dd28fd View commit details
    Browse the repository at this point in the history

Commits on May 24, 2022

  1. Do not fail rollup if force-merge fails

    @csoulios
    csoulios committed May 24, 2022
    Configuration menu
    Copy the full SHA
    393d7c2 View commit details
    Browse the repository at this point in the history

  2. Merge branch 'master' into tsdb-rollup

    @csoulios
    csoulios committed May 24, 2022
    Configuration menu
    Copy the full SHA
    bfd9ce9 View commit details
    Browse the repository at this point in the history

Commits on May 25, 2022

  1. Merge branch 'master' into tsdb-rollup

    @csoulios
    csoulios committed May 25, 2022
    Configuration menu
    Copy the full SHA
    0a06f41 View commit details
    Browse the repository at this point in the history