Support wildcard/regex for indices param in _remotestore/_restore #8922

linuxpi · 2023-07-27T12:46:45Z

Description

_remotestore/_restore will now support wildcards/regex for indices param.
Specifying body - { "indices": ["my-index-*"] } will match all indices whose name start with my-index- and trigger remote store restore.

Related Issues

Resolves #8480

Check List

New functionality includes testing.
- All tests pass
New functionality has been documented.
- New functionality has javadoc added
Commits are signed per the DCO using --signoff
Commit changes are listed out in CHANGELOG.md file (See: Changelog)

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

github-actions · 2023-07-27T13:10:14Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/21086/
CommitID: 85c230d
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

github-actions · 2023-07-27T18:59:54Z

Gradle Check (Jenkins) Run Completed with:

RESULT: UNSTABLE ❕
TEST FAILURES:

      1 org.opensearch.action.admin.cluster.node.tasks.ResourceAwareTasksTests.testTaskResourceTrackingDuringTaskCancellation

URL: https://build.ci.opensearch.org/job/gradle-check/21121/
CommitID: 61aa234
Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

codecov · 2023-07-27T19:01:26Z

Codecov Report

Attention: Patch coverage is 77.19298% with 13 lines in your changes are missing coverage. Please review.

Project coverage is 71.66%. Comparing base (b15cb0c) to head (488ab9c).
Report is 259 commits behind head on main.

Files	Patch %	Lines
...in/java/org/opensearch/common/util/IndexUtils.java	84.61%	4 Missing and 4 partials ⚠️
...arch/index/recovery/RemoteStoreRestoreService.java	0.00%	4 Missing ⚠️
...ava/org/opensearch/snapshots/SnapshotsService.java	0.00%	1 Missing ⚠️

Additional details and impacted files

@@             Coverage Diff              @@
##               main    #8922      +/-   ##
============================================
+ Coverage     71.42%   71.66%   +0.24%     
- Complexity    59978    61106    +1128     
============================================
  Files          4985     5052      +67     
  Lines        282275   287122    +4847     
  Branches      40946    41602     +656     
============================================
+ Hits         201603   205771    +4168     
- Misses        63999    64322     +323     
- Partials      16673    17029     +356

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

github-actions · 2023-08-01T11:45:59Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/21554/
CommitID: 0a5e37a
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

BhumikaSaini-Amazon

Thanks @linuxpi!
Please could you also enable the ITs in RemoteStoreIT that are currently awaiting fix on #8480? We need to pass those tests as part of this enhancement.

opensearch-trigger-bot · 2023-08-31T15:21:22Z

This PR is stalled because it has been open for 30 days with no activity. Remove stalled label or comment or this will be closed in 7 days.

linuxpi · 2024-05-02T10:20:23Z

@linuxpi I've cleaned up the conflict, and verify this is in a green state, before we merge this improvement please add a test case(s) that verifies the regex/wildcard scenario.

Integ tests for this already existed before my change but were skipped due to AwaitsFix annotation. I've remove this and now these tests are run as part of gradle check.

github-actions · 2024-05-02T11:07:54Z

❌ Gradle check result for 56d100a: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

linuxpi · 2024-05-02T11:37:56Z

Flaky Test - #10558

org.opensearch.cluster.allocation.ClusterRerouteIT.testDelayWithALargeAmountOfShards

peternied · 2024-05-02T11:52:49Z

@linuxpi Thanks for checking on the test failure, I've restarted the gradle-check workflow

github-actions · 2024-05-02T12:15:28Z

❌ Gradle check result for 56d100a: FAILURE

Known issue [BUG] org.opensearch.repositories.azure.AzureBlobStoreRepositoryTests.testContainerCreationAndDeletion / org.opensearch.repositories.azure.AzureBlobStoreRepositoryTests.testReadRange are flaky #10940

github-actions · 2024-05-02T16:15:39Z

❌ Gradle check result for 56d100a: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

Signed-off-by: Varun Bansal <bansvaru@amazon.com>

peternied · 2024-05-02T17:14:55Z

@linuxpi Please investigate the tests failures after the results come back:

* What went wrong:
Execution failed for task ':distribution:bwc:minor:buildBwcLinuxTar'.
> Building 2.14.0 didn't generate expected file /var/jenkins/workspace/gradle-check/search/distribution/bwc/minor/build/bwc/checkout-2.x/distribution/archives/linux-tar/build/distributions/opensearch-min-2.14.0-SNAPSHOT-linux-x64.tar.gz

Check will keep failing until you've rebased.

github-actions · 2024-05-02T17:41:57Z

❌ Gradle check result for e1f55c6: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

github-actions · 2024-05-03T04:51:10Z

❌ Gradle check result for e1f55c6: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

github-actions · 2024-05-03T08:23:14Z

❕ Gradle check result for 488ab9c: UNSTABLE

TEST FAILURES:

      1 org.opensearch.cluster.MinimumClusterManagerNodesIT.testThreeNodesNoClusterManagerBlock

Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

opensearch-trigger-bot · 2024-05-03T08:42:41Z

The backport to 2.x failed:

The process '/usr/bin/git' failed with exit code 128

To backport manually, run these commands in your terminal:

# Navigate to the root of your repository
cd $(git rev-parse --show-toplevel)
# Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add ../.worktrees/OpenSearch/backport-2.x 2.x
# Navigate to the new working tree
pushd ../.worktrees/OpenSearch/backport-2.x
# Create a new branch
git switch --create backport/backport-8922-to-2.x
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x --mainline 1 29ebd80fdd6771ccc2023bf432977c38c76bfd55
# Push it to GitHub
git push --set-upstream origin backport/backport-8922-to-2.x
# Go back to the original working tree
popd
# Delete the working tree
git worktree remove ../.worktrees/OpenSearch/backport-2.x

Then, create a pull request where the base branch is 2.x and the compare/head branch is backport/backport-8922-to-2.x.

…ensearch-project#8922) * Support wildcard/regex for indices param in _remotestore/_restore Signed-off-by: bansvaru <bansvaru@amazon.com> (cherry picked from commit 29ebd80)

linuxpi · 2024-05-03T10:03:14Z

backport pr #13527

…ensearch-project#8922) * Support wildcard/regex for indices param in _remotestore/_restore Signed-off-by: Varun Bansal <bansvaru@amazon.com> (cherry picked from commit 29ebd80)

) (#13527) * Support wildcard/regex for indices param in _remotestore/_restore Signed-off-by: Varun Bansal <bansvaru@amazon.com> (cherry picked from commit 29ebd80)

…ensearch-project#8922) * Support wildcard/regex for indices param in _remotestore/_restore Signed-off-by: bansvaru <bansvaru@amazon.com>

linuxpi force-pushed the restore-api-index-wildcard branch from 85c230d to 61aa234 Compare July 27, 2023 18:17

linuxpi force-pushed the restore-api-index-wildcard branch from 61aa234 to 0a5e37a Compare August 1, 2023 10:33

BhumikaSaini-Amazon suggested changes Aug 1, 2023

View reviewed changes

opensearch-trigger-bot bot added the stalled Issues that have stalled label Aug 31, 2023

linuxpi force-pushed the restore-api-index-wildcard branch from 0a5e37a to 92f112f Compare September 6, 2023 20:40

linuxpi marked this pull request as ready for review September 6, 2023 20:46

linuxpi requested review from reta, anasalkouz, andrross, Bukhtawar, CEHENKLE, dblock, gbbafna, setiah, kartg, kotwanikunal, mch2, nknize, owaiskazi19, peternied, Rishikesh1159, ryanbogan, saratvemulapalli, shwetathareja and dreamer-89 as code owners September 6, 2023 20:46

sachinpkale approved these changes May 2, 2024

View reviewed changes

linuxpi added v2.15.0 Issues and PRs related to version 2.15.0 and removed v2.14.0 labels May 2, 2024

peternied approved these changes May 2, 2024

View reviewed changes

Empty Commit

e1f55c6

Signed-off-by: Varun Bansal <bansvaru@amazon.com>

Merge branch 'main' into restore-api-index-wildcard

488ab9c

sachinpkale approved these changes May 3, 2024

View reviewed changes

sachinpkale merged commit 29ebd80 into opensearch-project:main May 3, 2024
28 checks passed

opensearch-trigger-bot bot added the backport-failed label May 3, 2024

linuxpi deleted the restore-api-index-wildcard branch May 3, 2024 10:02

This was referenced May 3, 2024

[BUG] Builds fail due to SnapshotUtils.filterIndices() not being resolved #13530

Closed

[BUG] Builds fail due to SnapshotUtils.filterIndices() not being resolved opensearch-project/security#4312

Closed

rishabh6788 mentioned this pull request May 5, 2024

[BUG] _restore api throwing 500 error on security enabled 3.0.0 cluster #13548

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Support wildcard/regex for indices param in _remotestore/_restore #8922

Support wildcard/regex for indices param in _remotestore/_restore #8922

linuxpi commented Jul 27, 2023 •

edited

github-actions bot commented Jul 27, 2023

github-actions bot commented Jul 27, 2023

codecov bot commented Jul 27, 2023 •

edited

github-actions bot commented Aug 1, 2023

BhumikaSaini-Amazon left a comment

opensearch-trigger-bot bot commented Aug 31, 2023

linuxpi commented May 2, 2024

github-actions bot commented May 2, 2024

linuxpi commented May 2, 2024

peternied commented May 2, 2024 •

edited

github-actions bot commented May 2, 2024 •

edited by peternied

github-actions bot commented May 2, 2024

peternied commented May 2, 2024

github-actions bot commented May 2, 2024

github-actions bot commented May 3, 2024

github-actions bot commented May 3, 2024

opensearch-trigger-bot bot commented May 3, 2024

linuxpi commented May 3, 2024

Support wildcard/regex for indices param in _remotestore/_restore #8922

Support wildcard/regex for indices param in _remotestore/_restore #8922

Conversation

linuxpi commented Jul 27, 2023 • edited

Description

Related Issues

Check List

github-actions bot commented Jul 27, 2023

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Jul 27, 2023

Gradle Check (Jenkins) Run Completed with:

codecov bot commented Jul 27, 2023 • edited

Codecov Report

github-actions bot commented Aug 1, 2023

Gradle Check (Jenkins) Run Completed with:

BhumikaSaini-Amazon left a comment

Choose a reason for hiding this comment

opensearch-trigger-bot bot commented Aug 31, 2023

linuxpi commented May 2, 2024

github-actions bot commented May 2, 2024

linuxpi commented May 2, 2024

peternied commented May 2, 2024 • edited

github-actions bot commented May 2, 2024 • edited by peternied

github-actions bot commented May 2, 2024

peternied commented May 2, 2024

github-actions bot commented May 2, 2024

github-actions bot commented May 3, 2024

github-actions bot commented May 3, 2024

opensearch-trigger-bot bot commented May 3, 2024

linuxpi commented May 3, 2024

linuxpi commented Jul 27, 2023 •

edited

codecov bot commented Jul 27, 2023 •

edited

peternied commented May 2, 2024 •

edited

github-actions bot commented May 2, 2024 •

edited by peternied