importccl: Support import resume for all input formats

[miretskiy]

Add support for resuming imports for all inputs formats.
Add a unit test to verify resume behavior.

Release note (cli change): resume paused import

[cockroach-teamcity]

This change is 

[miretskiy]

Please note, top 2 commits are being reviewed in #42476

[miretskiy]

Master picked up dependent changes; ready for review.

[spaskob]

thank you will look at it after lunch!

…

On Tue, Dec 10, 2019 at 10:00 AM Yevgeniy Miretskiy < ***@***.***> wrote: Master picked up dependent changes; ready for review. — You are receiving this because your review was requested. Reply to this email directly, view it on GitHub <#43053?email_source=notifications&email_token=AANL633RDACWIXJIMTL3PO3QX6VIZA5CNFSM4JYJIZX2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEGPRBZY#issuecomment-564072679>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AANL63YDFTQCUD7WMVFDE43QX6VIZANCNFSM4JYJIZXQ> .

[spaskob]

I think you are missing read_import_mysqlout.go

Reviewable status: complete! 0 of 0 LGTMs obtained (waiting on @dt, @miretskiy, and @spaskob)

---

pkg/ccl/importccl/import_processor_test.go, line 222 at r1 (raw file):

nit: keep it a 1-liner as well

---

pkg/ccl/importccl/import_processor_test.go, line 803 at r1 (raw file):

descr, &descr.PrimaryIndex, colMap, []tree.Datum{tree.NewDInt(tree.DInt(id))}, primaryIndexKeyPrefix)

long line

[miretskiy]

read_import_mysqlout.go
Good catch. Added; test updated.

Reviewable status: complete! 0 of 0 LGTMs obtained (waiting on @dt and @spaskob)

---

pkg/ccl/importccl/import_processor_test.go, line 803 at r1 (raw file):

Previously, spaskob (Spas Bojanov) wrote…

descr, &descr.PrimaryIndex, colMap, []tree.Datum{tree.NewDInt(tree.DInt(id))}, primaryIndexKeyPrefix)

long line

Done.

[spaskob]

All Done

Reviewable status: complete! 0 of 0 LGTMs obtained (waiting on @dt, @miretskiy, and @spaskob)

---

pkg/ccl/importccl/import_processor_test.go, line 245 at r2 (raw file):

			ExternalStorage: externalStorageFactory,
			BulkAdder: func(
				_ context.Context, _ *client.DB, _ hlc.Timestamp, _ storagebase.BulkAdderOptions) (storagebase.BulkAdder, error) {

long line

---

pkg/ccl/importccl/import_processor_test.go, line 312 at r2 (raw file):

			}

			processor.Run(context.TODO())

context.Background()

---

pkg/ccl/importccl/import_processor_test.go, line 333 at r2 (raw file):

			ExternalStorage: externalStorageFactory,
			BulkAdder: func(
				_ context.Context, _ *client.DB, _ hlc.Timestamp, _ storagebase.BulkAdderOptions) (storagebase.BulkAdder, error) {

long line

---

pkg/ccl/importccl/import_processor_test.go, line 344 at r2 (raw file):

	// In this test, we'll specify various resume positions for different input formats.
	// We expect that the rows before resume position will be skipped.
	// NB: We assume that the (external) test files are sorted and contain at least batchSize rows.

long line

---

pkg/ccl/importccl/import_processor_test.go, line 368 at r2 (raw file):

				// Setup progress consumer.
				go func() {
					// Consume progress reports. Since we expect every batch to be flushed (BulkAdderFlushesEveryBatch),

long lines

---

pkg/ccl/importccl/import_processor_test.go, line 379 at r2 (raw file):

				}()

				_, err := runImport(context.TODO(), flowCtx, spec, progCh)

context.Background()

[spaskob]

Reviewable status: complete! 0 of 0 LGTMs obtained (waiting on @dt, @miretskiy, and @spaskob)

[miretskiy]

bors r+

[craig]

Build succeeded

• GitHub CI (Cockroach)