snapshot should work when cluster is in read_only mode.

[webmstr]

I was trying to make a full, consistent backup before an upgrade. Snapshots are at a moment of time, which doesn't work if clients are still updating your indexes.

I tried putting the cluster into read_only mode by setting cluster.blocks.read_only: true, but running a snapshot returned this error:

{"error":"ClusterBlockException[blocked by: [FORBIDDEN/6/cluster read-only (api)];]","status":403}

Please consider allowing snapshots to provide a consistent backup by running when in read-only mode.

[clintongormley]

@webmstr Snapshots are still moment in time while updates are happening. You don't need to lock anything. A snapshot will only backup the state of the index at the point that the backup starts, it won't take any later changes into account.

[webmstr]

As I mentioned, snapshots - as currently implemented - are an unreasonable method of performing a consistent backup prior to an upgrade. This enhancement would have allowed that option.

Without the enhancement, snapshots should not be used before an upgrade, because the indexes may have been changed while the snapshot was running. As such, the upgrade documentation should be changed to not propose the use of snapshots as backups, and a "full" backup procedure should be documented in its place.

[clintongormley]

Out of interest, why don't you just stop writing to your cluster? Reopening for discussion.

[clintongormley]

@imotov what are your thoughts?

[webmstr]

I could turn off logstash, but that's just one potential client. Someone could be curl'ing, or using an ES plugin (like head), etc. If you need a consistent backup, you have to disconnect and lock out the clients from the server side.

[imotov]

@clintongormley see #5876 I think this one is similar.

[clintongormley]

@imotov thanks, so setting index.blocks.write to true on all indices would be a reasonable workaround, at least until #5855 is resolved.

[saahn]

@clintongormley Actually, I discovered that the index.blocks.write attribute only prevents writes to existing indices. If a client tries to create a new index, that request succeeds, which brings us back to the same problem. My workaround was to shutdown the proxy node though which our clients access our ES cluster.
I am running into the same issue as @webmstr , but for different reason: I cannot create a consistent backup for a restore to a secondary datacenter because each snapshot takes ~1 hour to complete and we cannot afford to block writes from our clients for such a long period of time.
I am still trying to root cause why snapshots are taking so long; the time required for snapshot completion increases with each snapshot. However, when i restore the same data to a new cluster, snapshotting that data to a new S3 bucket takes less than a minute.

EDIT: I may have a theory on why the snapshots were taking so long... i was taking a snapshot every two hours, and the s3 bucket has a LOT of snapshots now (49). I'm thinking that the calls the ES aws plugin makes to the S3 endpoint slow down over time as the number of snapshots increase.

Or may be it's just the number of snapshots that's causing the slowness...i.e. regardless of whether the backend repository is S3 or fs? I guess I should have an additional cron job that deletes older snaphots. Is there a good rule of thumb on the number of snapshots to retain?

[colings86]

@imotov we discussed this issue but were unclear on what the differences are between the index.blocks.* options are and why the snapshot fails with read_only set to false?

[imotov]

@colings86 there is an ongoing effort to resolve this issue in #9203

[imotov]

After discussing this with @tlrx it looks like the best way to address this issue is by moving snapshot and restore cluster state elements from cluster metadata to a custom cluster element where it seems to belong (since information about currently running snapshot and restore hardly qualifies as metadata).