backupccl: add full cluster restore

[pbardea]

This commit adds the ability to entirely restore a full cluster backup.
This includes replacing the contents of some system tables to match the
data in the backup. Full cluster restore can be used using the RESTORE FROM ... syntax and is expected to be run on a new cluster with no user
data.

Release note (enterprise change): Add full cluster restore feature which
restores all the information contained in a full cluster backup. This
includes all of the user data as well as relevant data from system
tables. It is expected to be run on a new cluster with no user data.

[cockroach-teamcity]

This change is 

[dt]

Reviewed 8 of 10 files at r1, 6 of 8 files at r3, 5 of 6 files at r4.
Reviewable status: complete! 0 of 0 LGTMs obtained (waiting on @dt and @pbardea)

---

pkg/ccl/backupccl/backup.go, line 97 at r1 (raw file):

// Databases which are present in the cluster by default.
var defaultUserDBs = map[string]struct{}{sessiondata.DefaultDatabaseName: {}, sessiondata.PgDatabaseName: {}}

this seems like it belongs in restore.go ?

---

pkg/ccl/backupccl/restore.go, line 407 at r2 (raw file):

		numberOfIncrements := maxDescIDInBackup - uint32(maxDefaultID)
		for i := uint32(0); i <= numberOfIncrements; i++ {
			_, err = sql.GenerateUniqueDescID(ctx, p.ExecCfg().DB)

Huh, why the loop? why can't we just set it to maxDescIDInBackup?

---

pkg/ccl/backupccl/restore.go, line 431 at r2 (raw file):

			}
			if found && descriptorCoverage == tree.AllDescriptors {
				if foundID > 52 {

is there a name for this constant we could use?

---

pkg/ccl/backupccl/restore.go, line 1707 at r4 (raw file):

		if t := desc.Table(hlc.Timestamp{}); t != nil {
			// Don't count the default tables or system tables.
			if t.ParentID != sqlbase.SystemDB.ID {

i'd love to move this logic into a IsDefaultCreatedTable or something method in sqlbase so that new tables get included correctly.

[pbardea]

Reviewable status: complete! 0 of 0 LGTMs obtained (waiting on @dt)

---

pkg/ccl/backupccl/backup.go, line 97 at r1 (raw file):

Previously, dt (David Taylor) wrote…

this seems like it belongs in restore.go ?

Moved it over to sql/descriptor.go so that it can be used to identify user created tables in general.

---

pkg/ccl/backupccl/restore.go, line 407 at r2 (raw file):

Previously, dt (David Taylor) wrote…

Huh, why the loop? why can't we just set it to maxDescIDInBackup?

It looks like the key it writes to only expects increments. Specifically when this key is incremented the comments say: "If the key exists but was set using Put or CPut an error will be returned."

---

pkg/ccl/backupccl/restore.go, line 431 at r2 (raw file):

Previously, dt (David Taylor) wrote…

is there a name for this constant we could use?

Yep, moved this to the sql package along with helpers to identify user created tables.

---

pkg/ccl/backupccl/restore.go, line 1707 at r4 (raw file):

Previously, dt (David Taylor) wrote…

i'd love to move this logic into a IsDefaultCreatedTable or something method in sqlbase so that new tables get included correctly.

Moved this over to sql/descriptor.go along with the list of expected user tables.

[dt]

Reviewable status: complete! 0 of 0 LGTMs obtained (waiting on @dt and @pbardea)

---

pkg/ccl/backupccl/restore.go, line 407 at r2 (raw file):

Previously, pbardea (Paul Bardea) wrote…

It looks like the key it writes to only expects increments. Specifically when this key is incremented the comments say: "If the key exists but was set using Put or CPut an error will be returned."

huh, interesting. let's put a TODO and follow up with KV to see if we can change this later.

[pbardea]

bors r+

[craig]

Build succeeded

• GitHub CI (Cockroach)