backend/local: optimize local writer concurrency and memory usage

[glorv]

What problem does this PR solve?

Optimize local writer performance and memory usage.

What is changed and how it works?

Optimize local writer by:

• Open an index LocalWriter for each chunk to avoid the bottle neck at write index kvs
• Replace the async ApplendRows with sync operation, to optimize the memory usage and simplify the logic
• decrease the encode/deliverLoop chan to decrease memory usage
• use a memory buffer to restore the temp kv pairs before write them to SST file
• Batch ingest SST files into pebble to avoid the sync overhead in pebble ingest operation

Bench result:
The benchmark tests were run on a 40core machine with an NVME disk. and based on the following three data and table schema:

• DataSet1. 14k warehouse tpcc data
• DataSet2. 1k warehouse order_line table with 3 indexes. Thus each row generates 4 kvs.

  PRIMARY KEY (`ol_w_id`,`ol_d_id`,`ol_o_id`,`ol_number`),
  KEY `idx_d_i` (`ol_d_id`, `ol_i_id`),
  KEY `idx_d_w_supply` (`ol_d_id`, `ol_w_id`, `ol_supply_w_id`)
  

• DataSet3. 1k warehouse order_line table with 3 indexes. Thus each row generates 8 kvs.

  PRIMARY KEY (`ol_w_id`,`ol_d_id`,`ol_o_id`,`ol_number`),
  KEY `idx_d_i` (`ol_d_id`,`ol_i_id`),
  KEY `idx_d_w_supply` (`ol_d_id`,`ol_w_id`,`ol_supply_w_id`),
  KEY `idx_o_amount` (`ol_o_id`,`ol_amount`),
  KEY `idx_d_supply` (`ol_d_id`,`ol_supply_w_id`),
  KEY `idx_o_d_i` (`ol_o_id`,`ol_d_id`,`ol_i_id`),
  KEY `idx_i_id` (`ol_i_id`)
  

Bench Result:

DataSet	Branch	Peak Memory	Cost Time
DataSet1	master	34GB	2h25m
DataSet1	opt-local-writer	29GB	1h46m
DataSet2	master	30GB	24m57s
DataSet2	opt-local-writer	30GB	9m32s
DataSet3	master	64GB	1h22m
DataSet3	opt-local-writer	33GB	33m

Check List

Tests

• Unit test
• Integration test
• Manual test (add detailed scripts or steps below)
• No code

Side effects

Related changes

[glorv]

/run-all-tests

[kennytm]

it is very strange to use context.WithValue to pass in these settings. since we control the interface of OpenEngine we could add an argument specifying the type of engine (index engine, data engine sorted, data engine unsorted).

[glorv]

/run-all-tests

[glorv]

closed in favor of pingcap/br#753