feat(storage): implement backup and restore

[jacobmarble]

Closes #15603
Closes #15604
Closes #15605
Closes #15606

Implements online backup and offline restore.

To test functionality it was necessary to combine #15558 and #16352 into one PR based on recent master/HEAD. Then:

• wrote some data
• read it back
• backed it up with influx backup
• terminated influxd
• restored it with influxd restore to existing database directory
• started influxd, read it back, terminated influxd
• restored it with influxd restore to empty database directory
• started influxd, read it back, terminated influxd

Check before merging:

• "restored it with influxd restore to empty database directory" fails
• CHANGELOG.md updated with a link to the PR (not the Issue)
• Well-formatted commit messages
• Rebased/mergeable
• Tests pass
• Documentation updated or issue created (provide link to issue/pr)

[jacobmarble]

@tmgordeeva this is weird. Restore to existing directory works properly, but restore to empty dir looks like this:

$ influx query -o my-org 'from(bucket: "my-bucket") |> range(start: -3h)'
See 'influx query -h' for help
Error: Unauthorized access.
See 'influx query -h' for help

I think you can ignore the extra "See 'influx query -h' for help", it's the access that's the issue. I checked the bolt file MD5 sum before and after restore, and it looks identical in both scenarios. I've reproduced this error twice. Please take a look Monday.

[tmgordeeva]

@jacobmarble I think the problem is the credentials aren't part of this backup/restore process. I got the same, but restored it to working by copying the credentials back in.

[tmgordeeva]

@jacobmarble building idpe fails against this with:

etcd/kvservice.go:34:21: cannot use cache (type *KVCache) as type "github.com/influxdata/influxdb/kv".Store in argument to "github.com/influxdata/idpe/kv".NewClient:
	*KVCache does not implement "github.com/influxdata/influxdb/kv".Store (missing Backup method)
etcd/kvservice.go:56:21: cannot use store (type *KVStore) as type "github.com/influxdata/influxdb/kv".Store in argument to "github.com/influxdata/idpe/kv".NewClient:
	*KVStore does not implement "github.com/influxdata/influxdb/kv".Store (missing Backup method)

Created https://github.com/influxdata/idpe/pull/5776 to hopefully fix that.

[tmgordeeva]

@jacobmarble latest commit should fix the bolt being copied to credentials issue. Problem with my testing was I was still using the token explicitly in queries which I didn't realize overrides whatever's in credentials, retested without it to verify and verified credentials files match before and after.

[sebito91]

I think this is great, a couple of small nits included but one major question:

Why do we perform os.MkdirAll with 0777 permissions? That seems way too open for my liking, could we make do with 0755 instead? Do we truly need these files to be read/write/execute enable for other?

[jacobmarble]

I think this is great, a couple of small nits included but one major question:

Why do we perform os.MkdirAll with 0777 permissions? That seems way too open for my liking, could we make do with 0755 instead? Do we truly need these files to be read/write/execute enable for other?

I scanned the code to figure out what we currently do. In the storage engine, 0777 and 0666 is very common, and in #5171 @benbjohnson argues that the user of InfluxDB can umask if the defaults are 0777 and 0666. Therefore, I made the change in that direction.

[sebito91]

Thanks for that @jacobmarble, appreciate the history on this. I'm fine with the changes as long as you are OK with that return err comment.

[sebito91]

LGTM!

done