*: use new row-format in tidb #12634

lysu · 2019-10-11T14:21:45Z

What problem does this PR solve?

using https://github.com/pingcap/tidb/blob/master/docs/design/2018-07-19-row-format.md format in TiDB

follow up #7597 but get optimize for rowcodec from unionstore

tikv pr in tikv/tikv#6132

What is changed and how it works?

TiDB:
- make insert/update/replace encode new data as new-format(try to make encoder be reuse)
- make point-get/batch-point-get decode new-format data into chunk directly(try to reuse decoder, but need more try..)
- make batchcheck/ddl's get operation can decode with new-format(DecodeRowWithMapNew)
- didn't modify cop relate logic, that has be resolved via arrow encode for chunk *: add a flag EncodeType for response to use Arrow/Default format to decode. #12536
mocktikv:
- make mocktikv read new-format from storage and back to old row-format to mocktikv's executor(mocktikv's executor still didn't using Chunk)
unionstore: https://github.com/lysu/unistore/tree/new-row-format
- make unionstore accept new-format update input(no need transform to old-format any more)
- return new format for kv-get/ batch-get
misc:
- didn't change datum encode to keep statistic works(we don't follow design doc to change varint to int) rowcodec: encode row int/uint as varint/varuint when back to old row ngaut/unistore#237
- keep binlog use old row format

WIP... need more tunning to make performance can be improved(now testing with unionstore).

Check List

Tests

Unit test
Integration test(only integrate with unionstore or mocktikv, tikv wait development)
Manual test

Upgrade from old mocktikv or old unionstore to new format one, old table keep works

Performance test WIP

Code changes

Has exported function/method change
Has exported variable/fields change
Has interface methods change
Has persistent data change

Side effects

Breaking backward compatibility
Need take care upgrade step

Related changes

n/a

Release note

use new-row format in tidb

This change is

codecov · 2019-11-18T08:28:36Z

Codecov Report

❗ No coverage uploaded for pull request base (master@85afb82). Click here to learn what that means.
The diff coverage is n/a.

@@             Coverage Diff             @@
##             master     #12634   +/-   ##
===========================================
  Coverage          ?   80.6571%           
===========================================
  Files             ?        483           
  Lines             ?     124325           
  Branches          ?          0           
===========================================
  Hits              ?     100277           
  Misses            ?      16121           
  Partials          ?       7927

lysu · 2019-11-18T13:53:02Z

read performance by https://github.com/lysu/sysbench/tree/dev-add-garbage, using --garbage_columns=170 table's point-get.

before:

sysbench --mysql-host=0.0.0.0 --mysql-port=4000 --mysql-user=root --threads=10 --tables=10 --table_size=30000 --time=60 --garbage_columns=170  oltp_point_select.lua run   
sysbench 1.1.0-1327e79 (using bundled LuaJIT 2.1.0-beta3)

Running the test with following options:
Number of threads: 10
Initializing random number generator from current time


Initializing worker threads...

Threads started!

SQL statistics:
    queries performed:
        read:                            673543
        write:                           0
        other:                           0
        total:                           673543
    transactions:                        673543 (11225.39 per sec.)
    queries:                             673543 (11225.39 per sec.)
    ignored errors:                      0      (0.00 per sec.)
    reconnects:                          0      (0.00 per sec.)

Throughput:
    events/s (eps):                      11225.3897
    time elapsed:                        60.0017s
    total number of events:              673543

Latency (ms):
         min:                                    0.15
         avg:                                    0.89
         max:                                   24.81
         95th percentile:                        2.14
         sum:                               599309.85

Threads fairness:
    events (avg/stddev):           67354.3000/84.41
    execution time (avg/stddev):   59.9310/0.00

after:

sysbench --mysql-host=0.0.0.0 --mysql-port=4000 --mysql-user=root --threads=10 --tables=10 --table_size=30000 --time=60 --garbage_columns=170  oltp_point_select.lua run   
sysbench 1.1.0-1327e79 (using bundled LuaJIT 2.1.0-beta3)

Running the test with following options:
Number of threads: 10
Initializing random number generator from current time


Initializing worker threads...

Threads started!

SQL statistics:
    queries performed:
        read:                            1112101
        write:                           0
        other:                           0
        total:                           1112101
    transactions:                        1112101 (18534.51 per sec.)
    queries:                             1112101 (18534.51 per sec.)
    ignored errors:                      0      (0.00 per sec.)
    reconnects:                          0      (0.00 per sec.)

Throughput:
    events/s (eps):                      18534.5106
    time elapsed:                        60.0016s
    total number of events:              1112101

Latency (ms):
         min:                                    0.11
         avg:                                    0.54
         max:                                   22.16
         95th percentile:                        1.06
         sum:                               599051.41

Threads fairness:
    events (avg/stddev):           111210.1000/92.84
    execution time (avg/stddev):   59.9051/0.00

/cc @coocood @jackysp

coocood · 2019-11-19T02:26:01Z

@lysu
how about select_random_ranges.lua workload?

lysu · 2019-12-17T09:10:21Z

/run-all-tests

zhangjinpeng87 · 2019-12-17T09:37:48Z

Amazing

lysu · 2019-12-24T05:05:15Z

/run-all-tests tikv=pr/6323

lysu · 2019-12-24T05:20:51Z

/run-integration-common-test tikv=pr/6323

breezewish · 2019-12-24T07:07:42Z

/run-all-tests tikv=pr/6323

lysu · 2019-12-24T08:04:28Z

/run-all-tests tikv=pr/6323

lysu · 2019-12-24T09:44:29Z

/run-all-tests tikv=pr/6323

sessionctx/variable/session.go

util/codec/codec.go

tablecodec/tablecodec.go

zhangjinpeng87 · 2019-12-25T03:52:59Z

executor/builder.go

 func (b *executorBuilder) buildBatchPointGet(plan *plannercore.BatchPointGetPlan) Executor {
 	startTS, err := b.getStartTS()
 	if err != nil {
 		b.err = err
 		return nil
 	}
+	decoder := newRowDecoder(b.ctx, plan.Schema(), plan.TblInfo)


Can we reuse the Decoder for the same table?

our PointGetExecutor is pointget + projection , so plan.Schema() will get different results for select a from t where id = 1 and select a, b from t where id = 1(len1 vs len2), in previous commits, I have try to reuse decoder in cached plan level, but after discuss with others, it introduce some complex code, maybe we can renew at here but do it later?

tablecodec/tablecodec.go

util/rowDecoder/decoder_test.go

lysu · 2019-12-25T08:33:39Z

/run-all-tests

lysu · 2019-12-25T08:38:01Z

/run-all-tests tikv=pr/6323

lysu · 2019-12-25T09:07:32Z

/run-all-tests tikv=pr/6132

lysu · 2020-01-02T03:13:26Z

/run-all-tests

lysu · 2020-01-02T03:13:52Z

/bench

sre-bot · 2020-01-02T04:39:43Z

Benchmark Report

Run Sysbench Performance Test on VMs

@@                               Benchmark Diff                               @@
================================================================================
--- tidb: b1c08ee2765ea6b370555dc172bb3b998a8cc6c0
+++ tidb: d4eae20aa781e13f6872556c50607606cfa70bf2
--- tikv: 267a5ebf7edb229cc53f5a8faa71975225bc6d8e
+++ tikv: 0de6ef5af5b75cf523ae0c54d9b433f62a407f06
pd: ea974c9488b8d304a72916d8cc50962f5ab99c5d
================================================================================
oltp_update_index:
    * QPS: 6404.59 ± 0.64% (std=26.52) delta: 0.25% (p=0.477)
    * Latency p50: 19.98 ± 0.64% (std=0.08) delta: -0.26%
    * Latency p99: 37.23 ± 0.90% (std=0.34) delta: -0.89%
            
oltp_insert:
    * QPS: 4769.73 ± 0.18% (std=6.40) delta: -0.04% (p=0.775)
    * Latency p50: 26.83 ± 0.19% (std=0.04) delta: 0.04%
    * Latency p99: 46.84 ± 2.24% (std=0.70) delta: -0.58%
            
oltp_read_write:
    * QPS: 16117.26 ± 0.02% (std=2.28) delta: 0.36% (p=0.276)
    * Latency p50: 159.14 ± 0.02% (std=0.02) delta: -0.41%
    * Latency p99: 301.53 ± 1.20% (std=2.55) delta: -2.42%
            
oltp_point_select:
    * QPS: 43379.19 ± 0.15% (std=38.50) delta: 1.49% (p=0.022)
    * Latency p50: 2.95 ± 0.00% (std=0.00) delta: -1.50%
    * Latency p99: 9.73 ± 0.00% (std=0.00) delta: 1.78%
            
oltp_update_non_index:
    * QPS: 4714.59 ± 0.02% (std=0.65) delta: -0.01% (p=0.906)
    * Latency p50: 27.15 ± 0.02% (std=0.00) delta: 0.00%
    * Latency p99: 41.10 ± 0.00% (std=0.00) delta: 0.00%

glorv · 2020-01-02T07:04:08Z

/run-unit-test

lysu · 2020-01-02T07:04:56Z

/bench +tpcc

workload:
  prepare: true

jackysp

LGTM

coocood · 2020-01-02T11:00:49Z

LGTM

coocood · 2020-01-02T11:02:22Z

/merge

sre-bot · 2020-01-02T11:02:25Z

Your auto merge job has been accepted, waiting for 14321

sre-bot · 2020-01-02T14:31:11Z

/run-all-tests

lysu added type/enhancement component/server status/WIP labels Oct 11, 2019

lysu changed the title *: use new-row format in tidb *: use new row-format in tidb Oct 11, 2019

lysu mentioned this pull request Nov 27, 2019

rowcodec: make rowcodec can be used for mocktikv & unistore #13774

Merged

lysu requested a review from zhangjinpeng87 December 5, 2019 03:36

lysu marked this pull request as ready for review December 24, 2019 09:42

lysu removed the status/WIP label Dec 24, 2019

lysu added the status/all tests passed label Dec 24, 2019

lysu requested review from jackysp and coocood December 25, 2019 02:51

coocood reviewed Dec 25, 2019

View reviewed changes

sessionctx/variable/session.go Outdated Show resolved Hide resolved

coocood reviewed Dec 25, 2019

View reviewed changes

util/codec/codec.go Outdated Show resolved Hide resolved

coocood reviewed Dec 25, 2019

View reviewed changes

tablecodec/tablecodec.go Outdated Show resolved Hide resolved

zhangjinpeng87 reviewed Dec 25, 2019

View reviewed changes

jackysp reviewed Dec 25, 2019

View reviewed changes

tablecodec/tablecodec.go Outdated Show resolved Hide resolved

jackysp reviewed Dec 25, 2019

View reviewed changes

util/rowDecoder/decoder_test.go Outdated Show resolved Hide resolved

lysu added 5 commits January 2, 2020 11:11

*: use new row-format

e9a890e

fix test after rebase

e0accab

add switch to new/old row format

81bf445

address comments

7b61493

sc should not be nil in running path

d4eae20

lysu added 2 commits January 2, 2020 13:10

Merge branch 'master' into dev-using-new-rf

9e54334

Merge branch 'master' into dev-using-new-rf

1f99f6c

jackysp reviewed Jan 2, 2020

View reviewed changes

jackysp added the status/LGT1 Indicates that a PR has LGTM 1. label Jan 2, 2020

coocood approved these changes Jan 2, 2020

View reviewed changes

sre-bot added the status/can-merge Indicates a PR has been approved by a committer. label Jan 2, 2020

Merge branch 'master' into dev-using-new-rf

7d27580

sre-bot merged commit 0bab73a into pingcap:master Jan 2, 2020

lysu mentioned this pull request Jan 3, 2020

support tidb's new rowformat input ngaut/unistore#317

Merged

lysu deleted the dev-using-new-rf branch January 3, 2020 07:21

birdstorm mentioned this pull request Jan 3, 2020

[BUG] Support new row format of TiDB/TiKV pingcap/tispark#1355

Closed

zier-one mentioned this pull request Jan 16, 2020

decode: support new format of row data pingcap/tiflow#241

Merged

kennytm mentioned this pull request Feb 3, 2020

*: use new row-format in tidb pingcap/tidb-lightning#267

Closed

lysu restored the dev-using-new-rf branch March 16, 2020 09:17

lysu mentioned this pull request Apr 1, 2020

reference/configuration: add tidb_row_format_version pingcap/docs-cn#2605

Merged

4 tasks

sre-bot mentioned this pull request Apr 15, 2020

reference/configuration: add tidb_row_format_version (#2605) pingcap/docs-cn#2770

Merged

4 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

*: use new row-format in tidb #12634

*: use new row-format in tidb #12634

lysu commented Oct 11, 2019 •

edited

codecov bot commented Nov 18, 2019 •

edited

lysu commented Nov 18, 2019 •

edited

coocood commented Nov 19, 2019

lysu commented Dec 17, 2019

zhangjinpeng87 commented Dec 17, 2019

lysu commented Dec 24, 2019

lysu commented Dec 24, 2019

breezewish commented Dec 24, 2019

lysu commented Dec 24, 2019

lysu commented Dec 24, 2019

zhangjinpeng87 Dec 25, 2019

lysu Dec 25, 2019 •

edited

lysu commented Dec 25, 2019

lysu commented Dec 25, 2019

lysu commented Dec 25, 2019

lysu commented Jan 2, 2020

lysu commented Jan 2, 2020

sre-bot commented Jan 2, 2020

glorv commented Jan 2, 2020

lysu commented Jan 2, 2020

jackysp left a comment

coocood commented Jan 2, 2020

coocood commented Jan 2, 2020

sre-bot commented Jan 2, 2020

sre-bot commented Jan 2, 2020

*: use new row-format in tidb #12634

*: use new row-format in tidb #12634

Conversation

lysu commented Oct 11, 2019 • edited

What problem does this PR solve?

What is changed and how it works?

Check List

codecov bot commented Nov 18, 2019 • edited

Codecov Report

lysu commented Nov 18, 2019 • edited

coocood commented Nov 19, 2019

lysu commented Dec 17, 2019

zhangjinpeng87 commented Dec 17, 2019

lysu commented Dec 24, 2019

lysu commented Dec 24, 2019

breezewish commented Dec 24, 2019

lysu commented Dec 24, 2019

lysu commented Dec 24, 2019

zhangjinpeng87 Dec 25, 2019

Choose a reason for hiding this comment

lysu Dec 25, 2019 • edited

Choose a reason for hiding this comment

lysu commented Dec 25, 2019

lysu commented Dec 25, 2019

lysu commented Dec 25, 2019

lysu commented Jan 2, 2020

lysu commented Jan 2, 2020

sre-bot commented Jan 2, 2020

Benchmark Report

glorv commented Jan 2, 2020

lysu commented Jan 2, 2020

jackysp left a comment

Choose a reason for hiding this comment

coocood commented Jan 2, 2020

coocood commented Jan 2, 2020

sre-bot commented Jan 2, 2020

sre-bot commented Jan 2, 2020

lysu commented Oct 11, 2019 •

edited

codecov bot commented Nov 18, 2019 •

edited

lysu commented Nov 18, 2019 •

edited

lysu Dec 25, 2019 •

edited