Update restore functionality to run in online mode, consume Enterprise backup files. #9207

aanthony1243 · 2017-12-07T20:53:43Z

Added a flag -online that will run a database restore on a live DB instance. A side effect of this change is that the existing DB's in the destination system will be unaffected, while the imported database must not exist in the system.

Also added a -enterprise flag that will enable restore to consume enterprise data files. This mode always runs as an online update.

Existing restore functionality is preserved.

Required for all non-trivial PRs

Rebased/mergable
Tests pass
CHANGELOG.md updated

Required only if applicable

You can erase any checkboxes below this note if they are not applicable to your Pull Request.

Update man page when modifying a command
InfluxData Documentation: issue filed or pull request submitted <link to issue or pull request>

aanthony1243 · 2017-12-07T20:58:53Z

pkg/tar/file_unix.go

@@ -0,0 +1,20 @@
+// +build !windows


copied from plutonium.

aanthony1243 · 2017-12-07T20:59:04Z

pkg/tar/file_windows.go

@@ -0,0 +1,19 @@
+package tar


copied from plutonium

aanthony1243 · 2017-12-07T21:00:22Z

pkg/tar/stream.go

+	"time"
+)
+
+func StreamFunc(w io.Writer, dir, relativePath string, writeFunc func(f os.FileInfo, shardRelativePath, fullPath string, tw *tar.Writer) error) error {


copied and adapted from plutonium to be more flexible for the OSS use-case where we filter files before they go to the archive. the original API is below, and is functionally equivalent to the original.

Does this need to be exported? Also, needs docs.

StreamFunc is an odd name for a function to call. Maybe have Stream take an optional func arg that defaults to the TarFile if nil?

👍 needs to be exported, esp since it now replaces Stream(...)

joelegasse · 2017-12-07T21:27:00Z

cmd/influxd/restore/data.proto

+
+message EnterpriseData {
+    required bytes Data = 1;
+}


missing newline

joelegasse · 2017-12-07T21:30:21Z

cmd/influxd/restore/restore.go

+	if cmd.enterprise || cmd.online {
+		if cmd.enterprise {
+			if filepath.Ext(cmd.backupFilesPath) != ".manifest" {
+				return fmt.Errorf("when using -enterprise, must provide path to a manifest file")


This is not right. Enterprise backups can be taken incrementally, in which case a directory containing the manifests should be used.

I had my reasons for this. maybe we can discuss in person.

joelegasse · 2017-12-07T21:32:10Z

cmd/influxd/restore/restore.go

-	if cmd.metadir != "" {
-		if err := cmd.unpackMeta(); err != nil {
-			return err
+	if cmd.enterprise || cmd.online {


This should be broken up into two method calls instead of single if statement with two large blocks

joelegasse · 2017-12-07T21:34:40Z

cmd/influxd/restore/restore.go

+	if cmd.enterprise || cmd.online {
+		// validate the arguments
+		if cmd.sourceDatabase == "" {
+			return fmt.Errorf("-db is a required parameter")


We should also support a complete restore, no? Why is this limited to restoring a single database at a time?

joelegasse · 2017-12-07T21:42:21Z

cmd/influxd/restore/restore.go

+func (cmd *Command) updateMetaLive() error {
+
+	var metaBytes []byte
+	if cmd.enterprise {


I'm seeing if cmd.enterprise scattered around far too much for my liking. Can you abstract away the behavior into a type/interface, and only make that decision once?

I'm not sure I agree with this point. I did tidy up the logic a bit so it's not "scattered around" so much.

It's still adds large blocks of code that a split by a single else. This needs to either be abstracted away as a common interface, or just split the codepaths (call different methods).

joelegasse · 2017-12-07T22:39:34Z

services/meta/data.go

+		return shardIDMap, errors.New("database already exists")
+	}
+
+	for _, rpImport := range dbImport.RetentionPolicies {


I would strongly recommend updating this to create new info objects throughout this code, rather than mutating what's already here. The metadata structs are full of slices, so you may be inadvertently modifying shared data.

solved using dbInfo.clone() which does a deep copy.

joelegasse · 2017-12-07T22:41:49Z

services/snapshotter/service.go

+	d := json.NewDecoder(conn)
+
+	if err := d.Decode(&r); err != nil {
+		return r, []byte{}, err


nil should be used for an empty slice

joelegasse · 2017-12-07T22:42:53Z

services/snapshotter/service.go

+
+		remainder := d.Buffered()
+
+		//// no time to investigate.  It seems the JSON encoder puts one extra byte on the stream,


What is going on here?

OK, I just read through... and it looks like at one point the protocol is [byte].[byte].[JSON-object].[binary-data]... That seems far too fragile to be useful or maintainable.

joelegasse · 2017-12-07T22:49:13Z

services/snapshotter/service.go

+		if err != nil && err != io.EOF {
+			return r, bits, err
+		}
+		return r, bits[1:], nil


Why is one byte being stripped off?

the JSON encoder seems to put an extra byte on the stream on the client side. I'll add a comment.

joelegasse · 2017-12-07T22:50:03Z

services/snapshotter/service.go

+		// it is a bit random but sometimes the Json decoder will consume all the bytes and sometimes
+		// it will leave a few behind.
+		if err != io.EOF && n < int(r.UploadSize+1) {
+			n, err = conn.Read(bits[n:])


In general, prefer ioutil.ReadAll or io.ReadFull to ensure draining of readers or reading sufficient data.

jwilder · 2017-12-12T16:18:29Z

pkg/tar/stream.go

+	"time"
+)
+
+func StreamFunc(w io.Writer, dir, relativePath string, writeFunc func(f os.FileInfo, shardRelativePath, fullPath string, tw *tar.Writer) error) error {


Does this need to be exported? Also, needs docs.

jwilder · 2017-12-12T16:18:40Z

pkg/tar/stream.go

+	return StreamFunc(w, dir, relativePath, TarFile)
+}
+
+func SinceFilterTarFile(since time.Time) func(f os.FileInfo, shardRelativePath, fullPath string, tw *tar.Writer) error {


Needs docs.

jwilder · 2017-12-12T16:18:55Z

pkg/tar/stream.go

+	}
+}
+
+func TarFile(f os.FileInfo, shardRelativePath, fullPath string, tw *tar.Writer) error {


Does this need to be exported?

Also, has a stutter tar.TarFile.

will doc new Stream function, but yes this should be exported. The design is that you can write a custom filter for Stream. For example, see timeStampFilterTarFile in engine.go, which is too specific to be in pkg/tar. A custom function may, and in many cases should, use TarFile internally. Will rename it to remove stutter.

jwilder · 2017-12-12T16:27:02Z

pkg/tar/stream.go

+	"time"
+)
+
+func StreamFunc(w io.Writer, dir, relativePath string, writeFunc func(f os.FileInfo, shardRelativePath, fullPath string, tw *tar.Writer) error) error {


StreamFunc is an odd name for a function to call. Maybe have Stream take an optional func arg that defaults to the TarFile if nil?

joelegasse

Still have some structural/organizational issues in the restore command that need to be cleaned up, along with a confusing naming scheme.

The tar package needs to be standardized around forward slash paths so that backups will be compatible across platforms.

Missing godoc comments, and a couple other quality issues.

Some potential issues with the importing of metadata. We need to be explicit in the restore, and not blindly copy everything and replace a few fields.

I still really don't like the "write a JSON object then an arbitrary stream" format that was added here. I think we need to do some actual planning. Either the request should be a single JSON object, or we should use protocol buffers as we do in other places to better handle the binary data.

joelegasse · 2017-12-12T14:43:03Z

pkg/tar/stream.go

@@ -0,0 +1,151 @@
+package tar


Needs godoc comments on all exported functions.

not sure what commit this is. Latest HEAD has docs on every function in this file.

joelegasse · 2017-12-12T14:44:56Z

services/meta/data.go

@@ -783,6 +784,91 @@ func (data *Data) hasAdminUser() bool {
 	return false
 }

+// ImportData imports selected data into the current metadata.
+// opts provides options to possibly rename the Database or Retention Policy, or change the Replication Factor.


Comment doesn't line-up with args. There is no opts parameter, and the list of parameters doesn't mention anything about replication factor. The comment should also describe what the returned map is.

joelegasse · 2017-12-14T18:39:53Z

services/meta/data.go

+		return fmt.Errorf("imported metadata does not have datbase named %s", backupDBName)
+	}
+
+	dbImport := dbPtr.clone()


I'm actually going to insist that a new DatabaseInfo object get created and populated manually. There are other fields (continuous queries, default RP) that are being copied without concern, and any new fields would get copied without consideration of what that means for an import.

joelegasse · 2017-12-15T14:08:31Z

services/snapshotter/client.go

+		return nil, fmt.Errorf("updating metadata on influxd service failed: err=%v, n=%d", err, n)
+	}
+
+	resp := b.Bytes()


In this scenario, prefer ioutil.ReadAll rather than creating a bytes.Buffer, using ReadFrom, and then only calling Bytes().

joelegasse · 2017-12-15T14:09:32Z

services/snapshotter/client.go

+
+func (c *Client) decodeUintPair(bits []byte) (uint64, uint64, error) {
+	if len(bits) != 16 {
+		return 0, 0, fmt.Errorf("array must have exactly 16 bytes")


errors.New, there is no formatting occurring here. Also, the error message says "array", when it is talking about a slice.

joelegasse · 2017-12-15T14:47:11Z

cmd/influxd/restore/restore.go

+				if cmd.backupRetention == "" || cmd.backupRetention == file.Policy {
+					if cmd.shard == "" || cmd.shard == strconv.FormatUint(file.ShardID, 10) {
+						cmd.StdoutLogger.Printf("Restoring shard %d live from backup %s\n", file.ShardID, file.FileName)
+						f, err := os.Open(filepath.Join(cmd.backupFilesPath, file.FileName))


👎 It is not closed in the case of an error

joelegasse · 2017-12-15T14:47:34Z

cmd/influxd/restore/restore.go

+			if err != nil {
+				return err
+			}
+			f, err := os.Open(fn)


👎 handle the error case as well

joelegasse · 2017-12-15T14:48:24Z

cmd/influxd/restore/restore.go

+				}
+			}
+		}
+	} else {


Split this off somehow. Two giant blocks of code split by a single else makes it difficult to maintain.

joelegasse · 2017-12-15T14:51:50Z

cmd/influxd/restore/restore.go

+		if err != nil {
+			return err
+		}
+		if cmd.manifest.Platform == "OSS" {


If that's the case, this needs to be called something else, then. "enterprise" is not a state of mind, it is a different version of the software. The open-source database does not produce "enterprise" backups, it's just an updated backup format on-disk. Calling it "enterprise" but allowing it come from the open-source database is very confusing.

joelegasse · 2017-12-15T14:52:46Z

cmd/influxd/restore/restore.go

+	// should get us ["db","rp", "00001", "00"]
+	pathParts := strings.Split(filepath.Base(tarFile), ".")
+	if len(pathParts) != 4 {
+		return fmt.Errorf("backup tarfile name incorrect length")


It's not the length that is incorrect, it's the format.

aanthony1243 · 2018-01-02T20:27:01Z

@joelegasse should have covered all your change requests. I rolled in a small feature #3019 to backup that collects a file listing. Initially I was using it to drive some tests, though your change requests designed that out. Also added is support for incremental manifest file collections.

joelegasse

I'll remove myself as a reviewer at this point. Previous issues have not been addressed, and more changes are being added in.

joelegasse · 2018-01-04T17:48:54Z

cmd/influxd/backup/backup.go

@@ -57,6 +57,8 @@ type Command struct {
 	enterprise         bool
 	manifest           backup_util.Manifest
 	enterpriseFileBase string
+
+	BackupFiles []string


Needs a comment. What does this do, why is it exported, etc.

joelegasse · 2018-01-04T17:50:17Z

CHANGELOG.md

@@ -7,7 +7,9 @@
 ### Features

 - [#8495](https://github.com/influxdata/influxdb/pull/8495): Improve CLI connection warnings
- [#9146](https://github.com/influxdata/influxdb/issues/9146): Backup can produce data in the same format as the enterprise backup/restore tool.
+- [#3019](https://github.com/influxdata/influxdb/issues/3019): Backup utility prints a list of backup files. 


This needs to stop in the future... We can't just keep adding more and more issues to a PR. If there's a dependency, get it merged first. Adding extra issues and extra code to the same PR makes review longer and more difficult to tell what changes are related to which issue.

joelegasse · 2018-01-04T17:51:05Z

cmd/influxd/backup/backup.go

@@ -237,6 +244,9 @@ func (cmd *Command) backupShard(db, rp, sid string) error {

 	// TODO: verify shard backup data


Is this still a TODO?

rbetts · 2018-01-05T17:04:50Z

This PR has been open for a long time and has been read and iterated against by reviewers and by Adam. I understand that there are some lingering differences but at this point I've decided to allow this code to master.

The most recent addition to the PR is largely my fault - I asked Adam to simultaneously hold this work back from master until TSI landed and to go ahead and implement the remaining functionality to support restore of incremental backups to OSS.

Adam is going to put together docs for the supported changes. I'll arrange a review of those with PM.

Thank you everyone for working together on this - I know it has been a challenging experience.

final approval by ryan

Moved protobuf code to backup_util Fix for import-all-dbs mode

…ackup file paths at conclusion

… location

aanthony1243 requested review from jwilder and joelegasse December 7, 2017 20:53

ghost assigned aanthony1243 Dec 7, 2017

ghost added the review label Dec 7, 2017

aanthony1243 commented Dec 7, 2017

View reviewed changes

pkg/tar/file_unix.go

@@ -0,0 +1,20 @@

// +build !windows

Copy link

Contributor Author

aanthony1243 Dec 7, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

copied from plutonium.

aanthony1243 commented Dec 7, 2017

View reviewed changes

pkg/tar/file_windows.go

@@ -0,0 +1,19 @@

package tar

Copy link

Contributor Author

aanthony1243 Dec 7, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

copied from plutonium

aanthony1243 commented Dec 7, 2017

View reviewed changes

aanthony1243 changed the title ~~Aa 8880 liverestore~~ Update restore functionality to run in online mode, consume Enterprise backup files. Dec 7, 2017

joelegasse suggested changes Dec 7, 2017

View reviewed changes

aanthony1243 force-pushed the aa_8880_liverestore branch from 4824853 to 068f74f Compare December 11, 2017 22:33

jwilder suggested changes Dec 12, 2017

View reviewed changes

jwilder approved these changes Dec 12, 2017

View reviewed changes

joelegasse previously requested changes Dec 15, 2017

View reviewed changes

aanthony1243 force-pushed the aa_8880_liverestore branch from 09175cf to 17fa782 Compare December 18, 2017 16:07

aanthony1243 force-pushed the aa_8880_liverestore branch from 5822c06 to 230f14f Compare January 2, 2018 15:19

aanthony1243 force-pushed the aa_8880_liverestore branch 6 times, most recently from ba5254e to a059547 Compare January 5, 2018 04:47

joelegasse reviewed Jan 5, 2018

View reviewed changes

rbetts self-requested a review January 5, 2018 16:59

rbetts approved these changes Jan 5, 2018

View reviewed changes

aanthony1243 force-pushed the aa_8880_liverestore branch from a059547 to c7225dc Compare January 8, 2018 15:40

aanthony1243 force-pushed the aa_8880_liverestore branch 2 times, most recently from 49e99ea to 478ceab Compare January 8, 2018 20:26

Adam added 16 commits January 10, 2018 13:05

Live Restore + Enterprise data format compatability

b117002

Extended ImportData to import all DB's if no db name given

7b7b933

Moved protobuf code to backup_util Fix for import-all-dbs mode

Fix syntax error and double DB creation in importOneDB

b5ce4be

Update to use protobuf in all enterprise data cases

3ed1974

One last PR change, ready to re-review

597f0c6

set default retention policy in whole-db case

916fe6a

Removed Platform key from manifest, as it's no longer needed

9343d1c

Added a new enterprise data test, and backup command now prints the b…

08ac3a7

…ackup file paths at conclusion

Added whole-system backup test

740f1f8

Update to test to do cross-testing with enterprise version

530bde5

incremental enterprise backup format support

9eeda0f

Removed unneeded proto files from restore

3d32a81

fix test since we now always use a backup dir and not a manifest file…

ecc6557

… location

Add 3019 to changelog

df51ada

Fix for windows filepaths in uploadShard client

455fd8e

debugging master regression

d033efc

aanthony1243 force-pushed the aa_8880_liverestore branch from 478ceab to d033efc Compare January 10, 2018 18:06

aanthony1243 merged commit 938db68 into master Jan 10, 2018

ghost removed the review label Jan 10, 2018

aanthony1243 deleted the aa_8880_liverestore branch January 10, 2018 19:10


		remainder := d.Buffered()

		//// no time to investigate. It seems the JSON encoder puts one extra byte on the stream,

		@@ -237,6 +244,9 @@ func (cmd *Command) backupShard(db, rp, sid string) error {

		// TODO: verify shard backup data

Update restore functionality to run in online mode, consume Enterprise backup files. #9207

Update restore functionality to run in online mode, consume Enterprise backup files. #9207

Conversation

aanthony1243 commented Dec 7, 2017 • edited Loading

Required for all non-trivial PRs

Required only if applicable

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

joelegasse left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

aanthony1243 commented Jan 2, 2018

joelegasse left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

rbetts commented Jan 5, 2018

aanthony1243 commented Dec 7, 2017 •

edited

Loading