Improve Snapshot/Backup capabilities for Air Gapped deployments via direct downloads or allowing individuals to push snapshots to repositories such as GitLab, Nexus or Artifactory

[sheldonmcclung]

Description

The below image shows the docs with the current snapshot repository options:

For an environment that is air gapped, there are not many options available as far as registries to backup the elastic data.

I propose that allowing direct downloading of the data onto a system that could handle it OR allowing admins to push the download to a url that would accept the input it would be a great addition.

The shared file system repository does not work well for me as this is an ECK stack so there is no guarantee that the data will exist on a node that contains the mounted folder that would be used for backup.

I will grant that deploying MinIO to emulate an S3 bucket is an option, but I would rather not have to support another tool just to do Elastic backups.

I believe that additional options for snapshot repositories would be well appreciated!

[sheldonmcclung]

This was also posted in the Elastic forums for discussion before I put in a feature request: https://discuss.elastic.co/t/improve-snapshot-backup-capabilities-for-air-gapped-deployments-via-direct-downloads-or-allowing-individuals-to-push-snapshots-to-repositories-such-as-gitlab-nexus-or-artifactory/340599/1

[elasticsearchmachine]

Pinging @elastic/es-distributed (Team:Distributed)

[DaveCTurner]

I think there might be a misconception here. Deduplication is a key feature of snapshots: each snapshot will re-use existing data in the repository rather than copying it all over the wire each time. That means it doesn't really make sense to directly download the data somewhere external: Elasticsearch needs to interact with the existing data in order to avoid unnecessary snapshotting. Moreover I don't think it would be easy to restore from that kind of snapshot: it certainly wouldn't be reasonable to upload the (possibly PBs of) data back into Elasticsearch just to retrieve a single index.

Pushing the data out will work better, but again it's more than just sending one enormous POST request to a HTTP endpoint, there needs to be some protocol for determining what data does/doesn't need copying each time, and then the data nodes all take part in the pushing process concurrently. And then we'd expect to be able to access the data at fairly fine granularity in order to do efficient restores too. There's lots of different protocols you could choose for this; Elasticsearch already supports several common options, but it's relatively easy to write a plugin to support a new one if you need something bespoke.

But however you do it you're going to need some nontrivial server to implement the other end of the protocol. In other words, if you're not using MinIO you're going to have to use something equivalent.

[DaveCTurner]

No response, so I'm closing this.