Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Adding action to cleanup & rebuild all index ranges. #15808

Merged
merged 4 commits into from Jul 7, 2023
Merged

Adding action to cleanup & rebuild all index ranges. #15808

merged 4 commits into from Jul 7, 2023

Conversation

dennisoelkers
Copy link
Member

@dennisoelkers dennisoelkers commented Jun 23, 2023 •
edited

Description

Motivation and Context

This PR is doing two things:

  • Cleans up index ranges when IndexRangesResource#rebuild is triggered
  • Adds a maintenance dropdown with a "Cleanup & recalculate all index ranges" button to the general index sets overview page

This helps a lot with getting index ranges synced after Opensearch was wiped/updated, indices were manually deleted etc.

What is missing and should be implemented in a follow-up PR:

  • Also including index range cleanup for the "Recalculate index ranges" action for a specific index set
  • Refactor the cleanup periodical for easier reuse
  • Wrap it in a system job instead of a sync execution

Fixes #11919.

How Has This Been Tested?

Screenshots (if appropriate):

cleanup-all

Types of changes

  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Refactoring (non-breaking change)
  • Breaking change (fix or feature that would cause existing functionality to change)

Checklist:

  • My code follows the code style of this project.
  • My change requires a change to the documentation.
  • I have updated the documentation accordingly.
  • I have read the CONTRIBUTING document.
  • I have added tests to cover my changes.

linuspahl
linuspahl approved these changes Jul 5, 2023
View reviewed changes
Copy link
Contributor

@linuspahl linuspahl left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Frontend looks good to me and the index recalculation seems to work as expected.

@mpfz0r mpfz0r self-assigned this Jul 6, 2023
mpfz0r
mpfz0r approved these changes Jul 6, 2023
View reviewed changes
Copy link
Member

@mpfz0r mpfz0r left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Works and LGTM. I agree about the follow up PRs. The fact that the cleanup is done asynchronously via the event bus makes a bit nervous

@dennisoelkers dennisoelkers merged commit 450b52c into master Jul 7, 2023
5 checks passed
@dennisoelkers dennisoelkers deleted the feat/add-action-to-cleanup-all-index-sets branch July 7, 2023 06:03
todvora pushed a commit that referenced this pull request Jul 11, 2023
@dennisoelkers @todvora
Adding action to cleanup & rebuild all index ranges. (#15808)
0c455d2 
* Adding action to cleanup & rebuild all index ranges.

* Adding changelog snippet.

* Suppressing linter hints.
dennisoelkers added a commit that referenced this pull request Jul 19, 2023
@todvora @kodjo-anipah
@dependabot @thll @bernd @patrickmann @dennisoelkers @janheise @zeeklop @github-actions @luk-kaminski @mpfz0r @linuspahl
Decorate aggregation query in event processor (#15793)
a023957 
* Decorate aggregation query in event processor

* unit test for query decoration in aggregation events

* added changelog

* Bump @babel/preset-env (#15876)

Bumps [@babel/preset-env](https://github.com/babel/babel/tree/HEAD/packages/babel-preset-env) from 7.22.5 to 7.22.6.
- [Release notes](https://github.com/babel/babel/releases)
- [Changelog](https://github.com/babel/babel/blob/main/CHANGELOG.md)
- [Commits](https://github.com/babel/babel/commits/v7.22.6/packages/babel-preset-env)

---
updated-dependencies:
- dependency-name: "@babel/preset-env"
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Bump @jest/types in /graylog2-web-interface/packages/jest-preset-graylog (#15873)

Bumps [@jest/types](https://github.com/facebook/jest/tree/HEAD/packages/jest-types) from 29.5.0 to 29.6.0.
- [Release notes](https://github.com/facebook/jest/releases)
- [Changelog](https://github.com/jestjs/jest/blob/main/CHANGELOG.md)
- [Commits](https://github.com/facebook/jest/commits/v29.6.0/packages/jest-types)

---
updated-dependencies:
- dependency-name: "@jest/types"
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Bump jest-environment-jsdom (#15871)

Bumps [jest-environment-jsdom](https://github.com/facebook/jest/tree/HEAD/packages/jest-environment-jsdom) from 29.5.0 to 29.6.0.
- [Release notes](https://github.com/facebook/jest/releases)
- [Changelog](https://github.com/jestjs/jest/blob/main/CHANGELOG.md)
- [Commits](https://github.com/facebook/jest/commits/v29.6.0/packages/jest-environment-jsdom)

---
updated-dependencies:
- dependency-name: jest-environment-jsdom
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Bump babel-jest in /graylog2-web-interface/packages/jest-preset-graylog (#15872)

Bumps [babel-jest](https://github.com/facebook/jest/tree/HEAD/packages/babel-jest) from 29.5.0 to 29.6.0.
- [Release notes](https://github.com/facebook/jest/releases)
- [Changelog](https://github.com/jestjs/jest/blob/main/CHANGELOG.md)
- [Commits](https://github.com/facebook/jest/commits/v29.6.0/packages/babel-jest)

---
updated-dependencies:
- dependency-name: babel-jest
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Add aws-msk-iam-auth (#15870)

Enables authentication with IAM for AWS MSK.
See https://github.com/aws/aws-msk-iam-auth

* Bump @babel/plugin-transform-runtime (#15875)

Bumps [@babel/plugin-transform-runtime](https://github.com/babel/babel/tree/HEAD/packages/babel-plugin-transform-runtime) from 7.22.5 to 7.22.6.
- [Release notes](https://github.com/babel/babel/releases)
- [Changelog](https://github.com/babel/babel/blob/main/CHANGELOG.md)
- [Commits](https://github.com/babel/babel/commits/v7.22.6/packages/babel-plugin-transform-runtime)

---
updated-dependencies:
- dependency-name: "@babel/plugin-transform-runtime"
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Merge pull request from GHSA-g96c-x7rh-99r3

* Add support for randomizing DNS Lookup source port

* Clarify purpose of lease

* Skip initial refresh

Previously, the pool was being refreshed immediately upon initialization. Now, the refresh waits until the `poolRefreshSeconds` duration has elapsed.

* Ensure thread safety, skip unused poller refreshes

* Add change log

* Merge pull request from GHSA-3fqm-frhg-7c85

- Clear session from cache on all nodes after deletion
- Add changelog

* Merge pull request from GHSA-2q4p-f6gf-mqr5

* Fix filename validation in Support Bundle handling

The previous implementation of "SupportBundle#ensureFileWithinBundleDir"
was affected by a partial path traversal vulnerability and allowed
authenticated users with the Admin role to download or delete files in
sibling directories of the support bundle data directory.

See: GHSA-2q4p-f6gf-mqr5

* Add changelog snippet

* Fix bundle download with relative data_dir config

* Fix another test case

* Add missing license header to DnsResolverPoolTest (#15880)

* Read static CSP configuration values from file (#15439)

* parse CSP config file

* parse hierarchical property names

* merge method

* checkpoint

* store csp values uniquely in sets

* refactor and cleanup

* update IT test

* allow duplicate keys in config file

* forbidden API

* typo in config file

* refine default csp.config

* revert back to regular java property file

* fix IT test

---------

Co-authored-by: Dennis Oelkers <dennis@graylog.com>

* User has to select an entity if he selects a Start Page type (#15869)

* force selection of an entity

* added changelog

* feat: Modifies route active check to account for nested routes (#15842)

* Updating yarn lockfile (#15819)

Co-authored-by: Gary Bot <garybot2@graylog.com>

* Decreasing z-index of login background. (#15887)

* Sorting on metrics like variance or std deviation is now possible (#15878)

* Sorting on metrics like variance or std deviation is now possible

* Changelog added

* added integration test for stddev sorting

---------

Co-authored-by: Tomas Dvorak <tomas.dvorak@graylog.com>

* Adding action to cleanup & rebuild all index ranges. (#15808)

* Adding action to cleanup & rebuild all index ranges.

* Adding changelog snippet.

* Suppressing linter hints.

* Bump @jest/types in /graylog2-web-interface/packages/jest-preset-graylog (#15893)

Bumps [@jest/types](https://github.com/facebook/jest/tree/HEAD/packages/jest-types) from 29.6.0 to 29.6.1.
- [Release notes](https://github.com/facebook/jest/releases)
- [Changelog](https://github.com/jestjs/jest/blob/main/CHANGELOG.md)
- [Commits](https://github.com/facebook/jest/commits/v29.6.1/packages/jest-types)

---
updated-dependencies:
- dependency-name: "@jest/types"
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Fix join pipeline function parameter name (#15891)

We cannot use `end` because it's a reserved keyword in the pipeline language and causes
a syntax error.

Instead of adding special handling for the `end` keyword to the pipeline rule language
we just rename the parameter to `endIndex`.

This shouldn't break existing rules, because they could've never been saved.

Fixes #11868

The substring function got already fixed by #15470 and should be
included in a backport.

* Bump @babel/plugin-transform-runtime (#15898)

Bumps [@babel/plugin-transform-runtime](https://github.com/babel/babel/tree/HEAD/packages/babel-plugin-transform-runtime) from 7.22.6 to 7.22.7.
- [Release notes](https://github.com/babel/babel/releases)
- [Changelog](https://github.com/babel/babel/blob/main/CHANGELOG.md)
- [Commits](https://github.com/babel/babel/commits/v7.22.7/packages/babel-plugin-transform-runtime)

---
updated-dependencies:
- dependency-name: "@babel/plugin-transform-runtime"
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Bump jest in /graylog2-web-interface/packages/jest-preset-graylog (#15896)

Bumps [jest](https://github.com/facebook/jest/tree/HEAD/packages/jest) from 29.5.0 to 29.6.1.
- [Release notes](https://github.com/facebook/jest/releases)
- [Changelog](https://github.com/jestjs/jest/blob/main/CHANGELOG.md)
- [Commits](https://github.com/facebook/jest/commits/v29.6.1/packages/jest)

---
updated-dependencies:
- dependency-name: jest
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Bump jest-environment-jsdom (#15894)

Bumps [jest-environment-jsdom](https://github.com/facebook/jest/tree/HEAD/packages/jest-environment-jsdom) from 29.6.0 to 29.6.1.
- [Release notes](https://github.com/facebook/jest/releases)
- [Changelog](https://github.com/jestjs/jest/blob/main/CHANGELOG.md)
- [Commits](https://github.com/facebook/jest/commits/v29.6.1/packages/jest-environment-jsdom)

---
updated-dependencies:
- dependency-name: jest-environment-jsdom
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Only display query input suggestions while input has focus. (#15885)

* Only display query input suggestions while input has focus.

* Adding changelog.

* Bump babel-jest in /graylog2-web-interface/packages/jest-preset-graylog (#15895)

Bumps [babel-jest](https://github.com/facebook/jest/tree/HEAD/packages/babel-jest) from 29.6.0 to 29.6.1.
- [Release notes](https://github.com/facebook/jest/releases)
- [Changelog](https://github.com/jestjs/jest/blob/main/CHANGELOG.md)
- [Commits](https://github.com/facebook/jest/commits/v29.6.1/packages/babel-jest)

---
updated-dependencies:
- dependency-name: babel-jest
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Adding groupings for dependencies. (#15899)

* Making value rendering consistent. (#15864)

* Making value rendering consistent.

* Adding changelog snippet.

* Improve readability of hover legend in aggregations. (#15888)

* Improve readability of hover labels in aggregations.

* Improve styling structure.

* Adding changelog

* Fixing linter hints.

* Bump the babel group (#15904)

Bumps the babel group in /graylog2-web-interface/packages/babel-preset-graylog with 1 update: [@babel/preset-env](https://github.com/babel/babel/tree/HEAD/packages/babel-preset-env).

- [Release notes](https://github.com/babel/babel/releases)
- [Changelog](https://github.com/babel/babel/blob/main/CHANGELOG.md)
- [Commits](https://github.com/babel/babel/commits/v7.22.7/packages/babel-preset-env)

---
updated-dependencies:
- dependency-name: "@babel/preset-env"
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: babel
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Update info text for empty aggregations. (#15906)

* Updating yarn lockfile (#15907)

Co-authored-by: Gary Bot <garybot2@graylog.com>

* Updating yarn lockfile (#15917)

Co-authored-by: Gary Bot <garybot2@graylog.com>

* Add changes required for e2e tests. (#15881)

* GL shutdown on catchable OutOfMemoryError (#15889)

* GL shutdown on catchable OutOfMemoryError

* Changelog added

* Changelog renamed, because of mixed-repo situation

* New entry in UPGRADING.md

* Setting `networkMode` to `always` for TanStack Query. (#15914)

* Setting `networkMode` to `always` for `react-query`

* Adding changelog snippet.

* Simplify QueryStringDecorators call (#15900)

* Bump semver from 5.7.1 to 5.7.2 in /graylog2-web-interface (#15928)

Bumps [semver](https://github.com/npm/node-semver) from 5.7.1 to 5.7.2.
- [Release notes](https://github.com/npm/node-semver/releases)
- [Changelog](https://github.com/npm/node-semver/blob/v5.7.2/CHANGELOG.md)
- [Commits](npm/node-semver@v5.7.1...v5.7.2)

---
updated-dependencies:
- dependency-name: semver
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Include link to replay an alert in alert mail notification template. (#15922)

* code cleanup

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: Maxwell <98284293+kodjo-anipah@users.noreply.github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Othello Maurer <othello@graylog.com>
Co-authored-by: Dan Torrey <dan@graylog.com>
Co-authored-by: Bernd Ahlers <bernd@users.noreply.github.com>
Co-authored-by: Patrick Mann <patrickmann@users.noreply.github.com>
Co-authored-by: Dennis Oelkers <dennis@graylog.com>
Co-authored-by: Jan Heise <jan.heise@graylog.com>
Co-authored-by: Ezequiel Lopez <105670363+zeeklop@users.noreply.github.com>
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: Gary Bot <garybot2@graylog.com>
Co-authored-by: luk-kaminski <lukasz.kaminski@graylog.com>
Co-authored-by: Marco Pfatschbacher <marco@graylog.com>
Co-authored-by: Linus Pahl <linus@graylog.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@mpfz0r mpfz0r mpfz0r approved these changes

@linuspahl linuspahl linuspahl approved these changes

Assignees

@mpfz0r mpfz0r

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Manually run index ranges cleanup through the GUI and api endpoint
3 participants
@dennisoelkers @mpfz0r @linuspahl