Add remote store main page updates. Add shallow snapshots #5078

Naarcha-AWS · 2023-09-22T16:36:10Z

Checklist

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license and subject to the Developers Certificate of Origin.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

Signed-off-by: Naarcha-AWS <naarcha@amazon.com>

_tuning-your-cluster/availability-and-recovery/remote-store/index.md

Signed-off-by: Naarcha-AWS <naarcha@amazon.com>

_tuning-your-cluster/availability-and-recovery/remote-store/index.md

sachinpkale · 2023-09-22T17:30:23Z

Overall changes for remote store main page LGTM.

_tuning-your-cluster/availability-and-recovery/remote-store/index.md

Signed-off-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com>

…dex.md Signed-off-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com>

cwillum

Looks good.

_tuning-your-cluster/availability-and-recovery/remote-store/index.md

_tuning-your-cluster/availability-and-recovery/remote-store/snapshot-interoperability.md

Co-authored-by: Chris Moore <107723039+cwillum@users.noreply.github.com> Signed-off-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com>

natebower

@Naarcha-AWS Please see my comments and changes and let me know if you have any questions. Thanks!

natebower · 2023-09-22T18:48:53Z

_tuning-your-cluster/availability-and-recovery/remote-store/index.md

-  ```json
-  POST my_index/_refresh
-  ```
+After segments are created on the primary shard as part of the refresh, flush, and merge flow, the segments are uploaded to remote segment store and the replica shards source a copy from the same remote segment store. This frees up the primary shard from having to perform a data copying operation.


"prevents" instead of "frees up"?

natebower · 2023-09-22T18:49:26Z

_tuning-your-cluster/availability-and-recovery/remote-store/index.md


-## Enable the feature flag
+Remote-backed storage is a cluster level setting. It can only be enabled when bootstrapping to the cluster. After bootstrapping completes, the remote-backed storage cannot be enabled or disabled. This provides durability at the cluster level.


Suggested change

Remote-backed storage is a cluster level setting. It can only be enabled when bootstrapping to the cluster. After bootstrapping completes, the remote-backed storage cannot be enabled or disabled. This provides durability at the cluster level.

Remote-backed storage is a cluster-level setting. It can only be enabled when bootstrapping to the cluster. After bootstrapping completes, the remote-backed storage cannot be enabled or disabled. This provides durability at the cluster level.

_tuning-your-cluster/availability-and-recovery/remote-store/index.md

_tuning-your-cluster/availability-and-recovery/remote-store/snapshot-interoperability.md

natebower · 2023-09-22T18:56:31Z

_tuning-your-cluster/availability-and-recovery/remote-store/snapshot-interoperability.md

+
+# Shallow snapshots
+
+Shallow copy snapshots allow you to reference data from an entire remote-backed segment instead of storing all of the data from the segment in a snapshot. This makes accessing segment data faster than normal snapshots, because segment data is not stored in the snapshot repository.


In the last sentence, I would either remove "than normal snapshots" or change to "than when using normal snapshots".

natebower · 2023-09-22T18:57:29Z

_tuning-your-cluster/availability-and-recovery/remote-store/snapshot-interoperability.md

+
+- Shallow copy snapshots only work for remote-backed indexes.
+- All nodes in the cluster must use OpenSearch 2.10 or later to take advantage of shallow copy snapshots.
+- There is no difference in file size between standard (regular, normal, primary or replica???) shards and shallow copy snapshot shards because no segment data is stored in the snapshot itself.


I'm assuming either the entire parenthetical or the question marks within it should be removed?

This was a suggestion from Chris I misunderstood. Adjusting.

…dex.md Co-authored-by: Chris Moore <107723039+cwillum@users.noreply.github.com> Signed-off-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com>

_tuning-your-cluster/availability-and-recovery/remote-store/index.md

_tuning-your-cluster/availability-and-recovery/remote-store/snapshot-interoperability.md

Co-authored-by: Nathan Bower <nbower@amazon.com> Signed-off-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com>

harishbhakuni · 2023-09-25T20:14:28Z

_tuning-your-cluster/availability-and-recovery/remote-store/snapshot-interoperability.md

+
+Use the [Cluster Settings API]({{site.url}}{{site.baseurl}}/api-reference/cluster-api/cluster-settings/) to enable the `remote_store_index_shallow_copy` repository setting, as shown in the following example:
+
+```bash


@Naarcha-AWS we don't need to update cluster settings api, we need to call the PUT _snapshot/ api to enable this

Could you post the exact call with request body here?

something like this:

curl -X PUT "localhost:9200/_snapshot/snap_repo?pretty" -H 'Content-Type: application/json' -d' { "type": "s3", "settings": { "bucket": "test-bucket", "base_path": "daily-snaps", "remote_store_index_shallow_copy": true } } '

harishbhakuni · 2023-09-25T20:23:53Z

_tuning-your-cluster/availability-and-recovery/remote-store/snapshot-interoperability.md

+
+- Shallow copy snapshots only work for remote-backed indexes.
+- All nodes in the cluster must use OpenSearch 2.10 or later to take advantage of shallow copy snapshots.
+- There is no difference in file size between standard shards and shallow copy snapshot shards because no segment data is stored in the snapshot itself.


Also, this line need to be updated. so, we show the incremental file count and size in bytes between the last snapshot and current snapshot in snapshot status API (https://opensearch.org/docs/latest/api-reference/snapshots/get-snapshot-status/#snapshot-file-stats). in case of shallow copy snapshot incremental file count and size in bytes will be zero.

_tuning-your-cluster/availability-and-recovery/remote-store/snapshot-interoperability.md

harishbhakuni · 2023-09-25T20:35:23Z

_tuning-your-cluster/availability-and-recovery/remote-store/snapshot-interoperability.md

+
+# Shallow snapshots
+
+Shallow copy snapshots allow you to reference data from an entire remote-backed segment instead of storing all of the data from the segment in a snapshot. This makes accessing segment data faster than using normal snapshots because segment data is not stored in the snapshot repository.


Shallow copy snapshots allows you to reference data directly from remote store repository instead of storing all of the segment data again in snapshot repository. These snapshots gets created faster than normal snapshots because segment data is not stored in the snapshot repository.

…-project#5078) * Add remote store main page updates. Add shallow snapshots Signed-off-by: Naarcha-AWS <naarcha@amazon.com> * Add next steps section Signed-off-by: Naarcha-AWS <naarcha@amazon.com> * Remove old content. Fix link. Fix typo. Signed-off-by: Naarcha-AWS <naarcha@amazon.com> * Fix link Signed-off-by: Naarcha-AWS <naarcha@amazon.com> * Apply suggestions from code review Signed-off-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com> * Update _tuning-your-cluster/availability-and-recovery/remote-store/index.md Signed-off-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com> * Apply suggestions from code review Co-authored-by: Chris Moore <107723039+cwillum@users.noreply.github.com> Signed-off-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com> * Update _tuning-your-cluster/availability-and-recovery/remote-store/index.md Co-authored-by: Chris Moore <107723039+cwillum@users.noreply.github.com> Signed-off-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com> * Apply suggestions from code review Co-authored-by: Nathan Bower <nbower@amazon.com> Signed-off-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com> --------- Signed-off-by: Naarcha-AWS <naarcha@amazon.com> Signed-off-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com> Co-authored-by: Chris Moore <107723039+cwillum@users.noreply.github.com> Co-authored-by: Nathan Bower <nbower@amazon.com>

* Add remote store main page updates. Add shallow snapshots Signed-off-by: Naarcha-AWS <naarcha@amazon.com> * Add next steps section Signed-off-by: Naarcha-AWS <naarcha@amazon.com> * Remove old content. Fix link. Fix typo. Signed-off-by: Naarcha-AWS <naarcha@amazon.com> * Fix link Signed-off-by: Naarcha-AWS <naarcha@amazon.com> * Apply suggestions from code review Signed-off-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com> * Update _tuning-your-cluster/availability-and-recovery/remote-store/index.md Signed-off-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com> * Apply suggestions from code review Co-authored-by: Chris Moore <107723039+cwillum@users.noreply.github.com> Signed-off-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com> * Update _tuning-your-cluster/availability-and-recovery/remote-store/index.md Co-authored-by: Chris Moore <107723039+cwillum@users.noreply.github.com> Signed-off-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com> * Apply suggestions from code review Co-authored-by: Nathan Bower <nbower@amazon.com> Signed-off-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com> --------- Signed-off-by: Naarcha-AWS <naarcha@amazon.com> Signed-off-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com> Co-authored-by: Chris Moore <107723039+cwillum@users.noreply.github.com> Co-authored-by: Nathan Bower <nbower@amazon.com>

Add remote store main page updates. Add shallow snapshots

89cf4bf

Signed-off-by: Naarcha-AWS <naarcha@amazon.com>

Naarcha-AWS added 3 - Tech review PR: Tech review in progress release-notes PR: Include this PR in the automated release notes v2.10.0 labels Sep 22, 2023

Naarcha-AWS self-assigned this Sep 22, 2023

Naarcha-AWS requested review from cwillum, hdhalter, kolchfa-aws, vagimeli, ananzh, seanneumann, AMoo-Miki and natebower as code owners September 22, 2023 16:36

Add next steps section

a3841d2

Signed-off-by: Naarcha-AWS <naarcha@amazon.com>

sachinpkale reviewed Sep 22, 2023

View reviewed changes

Naarcha-AWS added 2 commits September 22, 2023 12:19

Remove old content. Fix link. Fix typo.

1212904

Signed-off-by: Naarcha-AWS <naarcha@amazon.com>

Fix link

9d5622f

Signed-off-by: Naarcha-AWS <naarcha@amazon.com>

sachinpkale reviewed Sep 22, 2023

View reviewed changes

_tuning-your-cluster/availability-and-recovery/remote-store/index.md Outdated Show resolved Hide resolved

_tuning-your-cluster/availability-and-recovery/remote-store/index.md Outdated Show resolved Hide resolved

sachinpkale reviewed Sep 22, 2023

View reviewed changes

_tuning-your-cluster/availability-and-recovery/remote-store/index.md Outdated Show resolved Hide resolved

_tuning-your-cluster/availability-and-recovery/remote-store/index.md Outdated Show resolved Hide resolved

Naarcha-AWS commented Sep 22, 2023

View reviewed changes

_tuning-your-cluster/availability-and-recovery/remote-store/index.md Outdated Show resolved Hide resolved

Naarcha-AWS commented Sep 22, 2023

View reviewed changes

_tuning-your-cluster/availability-and-recovery/remote-store/index.md Outdated Show resolved Hide resolved

Naarcha-AWS commented Sep 22, 2023

View reviewed changes

_tuning-your-cluster/availability-and-recovery/remote-store/index.md Outdated Show resolved Hide resolved

Naarcha-AWS commented Sep 22, 2023

View reviewed changes

_tuning-your-cluster/availability-and-recovery/remote-store/index.md Outdated Show resolved Hide resolved

Apply suggestions from code review

78244ec

Signed-off-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com>

Naarcha-AWS added 4 - Doc review PR: Doc review in progress and removed 3 - Tech review PR: Tech review in progress labels Sep 22, 2023

Update _tuning-your-cluster/availability-and-recovery/remote-store/in…

c789bc1

…dex.md Signed-off-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com>

sachinpkale approved these changes Sep 22, 2023

View reviewed changes

cwillum approved these changes Sep 22, 2023

View reviewed changes

Apply suggestions from code review

a4cc717

Co-authored-by: Chris Moore <107723039+cwillum@users.noreply.github.com> Signed-off-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com>

natebower reviewed Sep 22, 2023

View reviewed changes

Update _tuning-your-cluster/availability-and-recovery/remote-store/in…

9641cb4

…dex.md Co-authored-by: Chris Moore <107723039+cwillum@users.noreply.github.com> Signed-off-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com>

Naarcha-AWS commented Sep 22, 2023

View reviewed changes

_tuning-your-cluster/availability-and-recovery/remote-store/index.md Outdated Show resolved Hide resolved

Naarcha-AWS commented Sep 22, 2023

View reviewed changes

_tuning-your-cluster/availability-and-recovery/remote-store/snapshot-interoperability.md Outdated Show resolved Hide resolved

Apply suggestions from code review

afb5b28

Co-authored-by: Nathan Bower <nbower@amazon.com> Signed-off-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com>

Naarcha-AWS merged commit 9479429 into main Sep 22, 2023
4 checks passed

harishbhakuni reviewed Sep 25, 2023

View reviewed changes

_tuning-your-cluster/availability-and-recovery/remote-store/snapshot-interoperability.md Show resolved Hide resolved

harishbhakuni reviewed Sep 25, 2023

View reviewed changes

Naarcha-AWS deleted the remote-store-main-page branch March 28, 2024 23:23

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add remote store main page updates. Add shallow snapshots #5078

Add remote store main page updates. Add shallow snapshots #5078

Naarcha-AWS commented Sep 22, 2023

sachinpkale commented Sep 22, 2023

cwillum left a comment

natebower left a comment

natebower Sep 22, 2023

natebower Sep 22, 2023

natebower Sep 22, 2023

natebower Sep 22, 2023

Naarcha-AWS Sep 22, 2023

harishbhakuni Sep 25, 2023

Naarcha-AWS Sep 25, 2023

harishbhakuni Sep 25, 2023

harishbhakuni Sep 25, 2023

harishbhakuni Sep 25, 2023 •

edited

Loading


		## Enable the feature flag
		Remote-backed storage is a cluster level setting. It can only be enabled when bootstrapping to the cluster. After bootstrapping completes, the remote-backed storage cannot be enabled or disabled. This provides durability at the cluster level.

	Remote-backed storage is a cluster level setting. It can only be enabled when bootstrapping to the cluster. After bootstrapping completes, the remote-backed storage cannot be enabled or disabled. This provides durability at the cluster level.
	Remote-backed storage is a cluster-level setting. It can only be enabled when bootstrapping to the cluster. After bootstrapping completes, the remote-backed storage cannot be enabled or disabled. This provides durability at the cluster level.


		# Shallow snapshots

		Shallow copy snapshots allow you to reference data from an entire remote-backed segment instead of storing all of the data from the segment in a snapshot. This makes accessing segment data faster than normal snapshots, because segment data is not stored in the snapshot repository.


		Use the [Cluster Settings API]({{site.url}}{{site.baseurl}}/api-reference/cluster-api/cluster-settings/) to enable the `remote_store_index_shallow_copy` repository setting, as shown in the following example:

		```bash

Add remote store main page updates. Add shallow snapshots #5078

Add remote store main page updates. Add shallow snapshots #5078

Conversation

Naarcha-AWS commented Sep 22, 2023

Checklist

sachinpkale commented Sep 22, 2023

cwillum left a comment

Choose a reason for hiding this comment

natebower left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

harishbhakuni Sep 25, 2023 • edited Loading

Choose a reason for hiding this comment

harishbhakuni Sep 25, 2023 •

edited

Loading