Support write TiCDC log to S3. #826

3pointer · 2020-08-05T13:21:01Z

What problem does this PR solve?

part of #768, support incremental log and write log data to s3 directly.

What is changed and how it works?

Define cdclog with three kind of events.
- row changed event: translate to protobuf then write data into cdclog
- ddl event: translate to protobuf then write data into ddlog
- resolved ts event: remain global resolved ts to log.meta
Since s3 doesn't allow append-like operation. there are three different treatments for different events
- row changed event: flush data to cdclog, if flushed data size is less than 5M. directly write to s3 with putObject request. if flushed data size is greater than 5M, use multi-upload request to upload data as one chunk in s3. if total upload chunks size is greater than 100M, then complete and rotate it. use the last row event's commit ts as the suffix of each cdclog name.
- ddl event: consider ddl is not frequent in most cases. when ddllog size less than 10M, I choose to read content, append new ddl event then write back to ddllog.
- resolved ts event: update in every EmitCheckpoinTs inplace.

Check List

Tests

Manual test (add detailed scripts or steps below)

Side effects

Possible performance regression
Increased code complexity
Breaking backward compatibility

Related changes

Need to cherry-pick to the release branch
Need to update the documentation
Need to update the tidb-cdc/cdc-ansible

Release note

CLAassistant · 2020-08-05T13:21:06Z

All committers have signed the CLA.

emit row change event emit ddl event

This reverts commit 16baa26.

cdc/model/sink.go

proto/LogProtocol.proto

amyangfei · 2020-08-07T08:20:14Z

cdc/processor.go

@@ -1024,6 +1024,11 @@ func runProcessor(
 		cancel()
 		return nil, errors.Trace(err)
 	}
+	err = sink.Initialize(ctx, nil)


Seems initialize does nothing in processor, is it better to remove it?

amyangfei · 2020-08-07T08:27:53Z

cdc/sink/cdclog/file.go

+				}
+				data, err := row.ToProtoBuf().Marshal()
+				if err != nil {
+					errCh <- err


Do we need to return when an error happens?

amyangfei · 2020-08-07T08:32:58Z

cdc/sink/cdclog/file.go

+				if err != nil {
+					errCh <- err
+				}
+				rowDatas = append(rowDatas, append(data, '\n')...)


is \n used as row delimiter, what happens when data contains \n

cdc/sink/cdclog/file.go

cdc/sink/cdclog/s3.go

cdc/sink/cdclog/file.go

cdc/sink/cdclog/s3.go

zier-one · 2020-08-12T11:27:44Z

cdc/entry/mounter.go

@@ -442,6 +442,7 @@ func (m *mounterImpl) mountRowKVEntry(tableInfo *model.TableInfo, row *rowKVEntr
 		Table: &model.TableName{
 			Schema:    schemaName,
 			Table:     tableName,
+			TableID:   row.PhysicalTableID,
 			Partition: partitionID,


we can remove the Partition now

How about remove the Partition in another PR?

3pointer · 2020-08-13T02:34:38Z

/run-integration-tests

zier-one · 2020-08-13T02:43:50Z

/lgtm

amyangfei · 2020-08-13T02:55:20Z

cdc/sink/cdclog/file.go

+				if err != nil {
+					return err
+				}
+				file, err := os.OpenFile(filepath.Join(tableDir, fileName), os.O_CREATE|os.O_WRONLY|os.O_APPEND, defaultFileMode)


The new file has the same file name with the rotated file?

No, when batch events flushed, it will create new file name here, but not every new file name will be used.

eh... maybe we should use the first event' commit ts as the filename

amyangfei

Also need test for file sink, we can do it in another PR

3pointer · 2020-08-13T04:01:52Z

/run-integration-tests

3pointer · 2020-08-13T04:26:19Z

/run-integration-tests

3pointer · 2020-08-13T05:33:29Z

/run-integration-tests

amyangfei · 2020-08-13T05:58:05Z

/merge

ti-srebot · 2020-08-13T05:58:15Z

/run-all-tests

ti-srebot · 2020-08-13T06:14:16Z

@3pointer merge failed.

3pointer · 2020-08-13T06:18:41Z

/run-all-tests

3pointer added 4 commits August 5, 2020 21:22

support

a6a7c10

emit row change event emit ddl event

debug: need revert

187f6a3

fix panic && finish test on minio

0782c60

Revert "debug: need revert"

0449058

This reverts commit 16baa26.

3pointer force-pushed the log_sink branch from 087551f to 9c3f73f Compare August 5, 2020 13:23

3pointer requested a review from zier-one August 5, 2020 13:24

3pointer force-pushed the log_sink branch from 41394c7 to 9c323ba Compare August 6, 2020 04:09

3pointer changed the title ~~Log sink~~ Support write TiCDC log to S3. Aug 6, 2020

fmt code && make check pass

8396faa

3pointer force-pushed the log_sink branch from 9c323ba to 8396faa Compare August 6, 2020 05:32

3pointer requested a review from amyangfei August 6, 2020 06:07

3pointer added 4 commits August 6, 2020 14:43

Merge branch 'master' into log_sink

991fda2

log: add file sink

072262b

Merge branch 'file_sink' into log_sink

4fb091d

fix log meta lost

57594b5

3pointer added component/sink Sink component. status/WIP labels Aug 7, 2020

3pointer added 4 commits August 7, 2020 12:05

use approximate row entry size

b453cee

format code

aad35fc

fix log meta bug

f74702b

fill ddl tableID

f985070

amyangfei reviewed Aug 7, 2020

View reviewed changes

cdc/model/sink.go Outdated Show resolved Hide resolved

proto/LogProtocol.proto Outdated Show resolved Hide resolved

proto/LogProtocol.proto Outdated Show resolved Hide resolved

amyangfei reviewed Aug 7, 2020

View reviewed changes

address comment

3563558

amyangfei reviewed Aug 7, 2020

View reviewed changes

cdc/sink/cdclog/s3.go Outdated Show resolved Hide resolved

cdc/sink/cdclog/s3.go Outdated Show resolved Hide resolved

cdc/sink/cdclog/file.go Outdated Show resolved Hide resolved

cdc/sink/cdclog/s3.go Show resolved Hide resolved

3pointer added 4 commits August 7, 2020 18:19

sink:remove lock in s3

c8095bc

use record encoder as row delimiter

3519565

use filepath instead of path

c5d3fd1

fix check

23b2a94

ti-srebot added the status/LGT1 Indicates that a PR has LGTM 1. label Aug 12, 2020

zier-one reviewed Aug 12, 2020

View reviewed changes

3pointer removed the status/WIP label Aug 13, 2020

ti-srebot removed the status/LGT1 Indicates that a PR has LGTM 1. label Aug 13, 2020

ti-srebot approved these changes Aug 13, 2020

View reviewed changes

ti-srebot added the status/LGT2 Indicates that a PR has LGTM 2. label Aug 13, 2020

amyangfei reviewed Aug 13, 2020

View reviewed changes

3pointer added 3 commits August 13, 2020 11:42

sink: rename file name when rotate

f77dcfb

add file sink test

39c1f7d

Merge branch 'master' into log_sink

06411f0

3pointer force-pushed the log_sink branch from 9f72844 to 06411f0 Compare August 13, 2020 04:01

fix file ci

e11e2d6

3pointer added 2 commits August 13, 2020 13:32

fix ci

e0e98cc

Merge branch 'master' into log_sink

759c6fd

amyangfei approved these changes Aug 13, 2020

View reviewed changes

ti-srebot added status/LGT3 The PR has already had 3 LGTM. and removed status/LGT2 Indicates that a PR has LGTM 2. labels Aug 13, 2020

ti-srebot added the status/can-merge Indicates a PR has been approved by a committer. label Aug 13, 2020

3pointer merged commit 6753924 into pingcap:master Aug 13, 2020

3pointer mentioned this pull request Aug 17, 2020

add s3 api to adapt cdc log (#456) pingcap/br#463

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Support write TiCDC log to S3. #826

Support write TiCDC log to S3. #826

3pointer commented Aug 5, 2020 •

edited

Loading

CLAassistant commented Aug 5, 2020 •

edited

Loading

amyangfei Aug 7, 2020

amyangfei Aug 7, 2020

amyangfei Aug 7, 2020

zier-one Aug 12, 2020

3pointer Aug 13, 2020

zier-one Aug 13, 2020

3pointer commented Aug 13, 2020

zier-one commented Aug 13, 2020

amyangfei Aug 13, 2020

3pointer Aug 13, 2020 •

edited

Loading

3pointer Aug 13, 2020

amyangfei left a comment

3pointer commented Aug 13, 2020

3pointer commented Aug 13, 2020

3pointer commented Aug 13, 2020

amyangfei commented Aug 13, 2020

ti-srebot commented Aug 13, 2020

ti-srebot commented Aug 13, 2020

3pointer commented Aug 13, 2020

Support write TiCDC log to S3. #826

Support write TiCDC log to S3. #826

Conversation

3pointer commented Aug 5, 2020 • edited Loading

What problem does this PR solve?

What is changed and how it works?

Check List

Release note

CLAassistant commented Aug 5, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

3pointer commented Aug 13, 2020

zier-one commented Aug 13, 2020

Choose a reason for hiding this comment

3pointer Aug 13, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

amyangfei left a comment

Choose a reason for hiding this comment

3pointer commented Aug 13, 2020

3pointer commented Aug 13, 2020

3pointer commented Aug 13, 2020

amyangfei commented Aug 13, 2020

ti-srebot commented Aug 13, 2020

ti-srebot commented Aug 13, 2020

3pointer commented Aug 13, 2020

3pointer commented Aug 5, 2020 •

edited

Loading

CLAassistant commented Aug 5, 2020 •

edited

Loading

3pointer Aug 13, 2020 •

edited

Loading