-
Notifications
You must be signed in to change notification settings - Fork 1
7. Multi Step Migration
Gang Liao edited this page Mar 10, 2021
·
5 revisions
In multi-step migration, a schema change is registered with the system ahead of time, and the system copies data into the new schema in a background process. Reads are served from the old schema, while writes go to both schemas. The implementation of multi-step migration is based on PostgreSQL with zero changes. This post will show you how to set up the raw PostgreSQL and how to run the TPC-C benchmark on top of that.
This tutorial assumes you have a current version of Docker installed on your machine. If you do not have Docker installed, choose your preferred operating system below to download Docker:
After you installed Docker, you can issue a command to pull our docker image and start a container:
# Pulls Docker image and run it on your local machine.
$ docker run --rm -it -d --name bullfrog gangliao/bullfrog:multi_step_migration bash
# Enters the container.
$ docker exec -u postgres -it bullfrog bash
# Output:
#
# postgres@968c58c4b04d:/$-
Deploy a DB instance:
# Deploy the database $ cd /home/postgres/BullFrog && ./deploy.sh
-
Run a TPC-C Benchmark (without schema migration) where data is already loaded into the database.
$ ./oltpbenchmark -b tpcc -c config/pgtpcc_config.xml --execute=true -s 1 -o tpcc --port 5433
-
Run a TPC-C Benchmark (with projection migration).
# ./oltpbenchmark -b tpcc -c config/pgtpcc_multistep_proj.xml --create=true --load=true --port=5433 # psql -h localhost -p 5433 tpcc -f clean_new_tables.sql # pg_ctl -D $PGDATA restart $ ./oltpbenchmark -b tpcc -c config/pgtpcc_multistep_proj.xml --execute=true -s 1 -o tpcc --port 5433 --bgthread=proj
Experiment Results
- Note: We have simplified the experimental environment and dataset size, so the results may be different from the paper. Since Docker for Mac runs in a LinuxKit VM, to get the best performance, please run the experiment on bare metal machines.
time(sec), throughput(req/sec), avg_lat(ms), min_lat(ms), 25th_lat(ms), median_lat(ms), 75th_lat(ms), 90th_lat(ms), 95th_lat(ms), 99th_lat(ms), max_lat(ms), tp (req/s) scaled
0,29.000,61.010,4.673,5.837,6.731,13.193,18.549,29.741,1521.178,1521.178,0.016
1,25.000,6.346,4.585,5.157,5.994,6.825,8.937,9.816,10.793,10.793,0.158
2,31.000,8.102,4.658,5.826,6.452,11.031,13.943,14.791,17.106,17.106,0.123
3,28.000,6.565,4.266,5.699,6.146,6.653,8.621,8.641,15.831,15.831,0.152
4,33.000,5.874,4.461,5.311,5.846,6.428,6.789,7.053,7.461,7.461,0.170
5,33.000,5.870,4.350,5.535,5.881,6.192,6.824,7.009,7.151,7.151,0.170
6,35.000,5.617,3.926,4.704,5.373,6.265,7.211,9.299,9.648,9.648,0.178
7,30.000,5.534,3.904,5.110,5.534,6.256,6.659,6.798,6.897,6.897,0.181
8,30.000,5.548,3.755,4.917,5.448,6.011,6.764,7.236,10.674,10.674,0.180
9,35.000,5.263,3.886,4.858,5.404,5.741,6.117,6.183,6.240,6.240,0.190
10,29.000,5.342,4.284,4.994,5.367,5.777,6.211,6.230,6.321,6.321,0.187
11,31.000,5.486,4.197,4.965,5.450,5.853,6.460,6.970,7.011,7.011,0.182
12,27.000,4.955,3.552,4.341,5.015,5.557,5.923,5.941,6.142,6.142,0.202
13,25.000,5.289,4.062,5.032,5.381,5.565,6.076,6.181,6.562,6.562,0.189
14,26.000,4.949,3.918,4.604,5.023,5.387,5.466,5.502,5.688,5.688,0.202
15,25.000,5.152,3.558,4.808,5.226,5.584,5.986,6.070,6.175,6.175,0.194
16,24.000,5.290,3.989,5.026,5.373,5.745,5.929,6.010,6.046,6.046,0.189
17,34.000,5.149,3.601,4.734,5.250,5.537,6.084,6.258,6.383,6.383,0.194
18,30.000,5.037,3.619,4.706,5.104,5.451,5.752,5.891,6.954,6.954,0.199
19,24.000,4.916,3.664,4.453,5.104,5.425,5.553,5.737,5.829,5.829,0.203
20,22.000,5.181,4.330,4.644,5.254,5.673,5.810,5.838,6.565,6.565,0.193
21,22.000,5.057,3.964,4.651,5.027,5.645,5.880,5.903,5.997,5.997,0.198
22,39.000,4.855,3.274,4.531,4.886,5.160,5.512,5.568,5.769,5.769,0.206
23,28.000,5.659,3.896,4.810,5.395,6.021,7.227,9.168,11.979,11.979,0.177
24,24.000,5.772,4.035,5.072,5.606,6.526,7.186,7.334,10.369,10.369,0.173
25,29.000,6.034,4.506,5.518,5.704,5.808,8.214,9.163,11.089,11.089,0.166
26,28.000,5.384,4.327,4.960,5.424,5.862,5.968,6.129,6.420,6.420,0.186
27,26.000,5.550,4.109,4.839,5.506,6.160,6.597,6.789,8.079,8.079,0.180
28,35.000,5.358,4.215,4.902,5.309,5.696,6.284,6.390,7.822,7.822,0.187
29,34.000,5.024,3.130,4.523,4.954,5.589,6.108,6.581,8.056,8.056,0.199
30,27.000,5.278,4.524,4.905,5.322,5.617,5.713,5.729,5.965,5.965,0.189
31,34.000,5.112,4.114,4.775,5.117,5.636,5.714,5.750,5.824,5.824,0.196
32,31.000,5.399,3.538,4.942,5.457,5.978,6.256,6.633,7.225,7.225,0.185
33,25.000,5.294,4.119,4.778,5.317,5.863,6.293,6.320,6.462,6.462,0.189
34,40.000,5.116,3.566,4.493,5.119,5.854,6.127,6.299,6.326,6.326,0.195
35,31.000,5.093,3.854,4.643,4.891,5.558,6.014,6.460,6.975,6.975,0.196
36,31.000,5.091,3.822,4.677,4.931,5.564,5.981,6.231,6.453,6.453,0.196
37,23.000,5.246,3.919,4.754,5.416,5.657,5.770,5.888,6.475,6.475,0.191
38,34.000,5.196,3.949,4.776,5.289,5.633,5.906,5.989,6.123,6.123,0.192
39,30.000,5.222,4.091,4.917,5.247,5.582,5.787,5.886,6.074,6.074,0.191
40,702.000,735.114,13.961,497.717,865.406,974.572,1061.288,1089.200,1111.580,1148.561,0.001
41,699.000,1526.345,1098.778,1336.479,1535.443,1711.824,1839.570,1872.561,1900.479,1907.083,0.001
42,701.000,2101.749,1885.498,2022.445,2114.554,2202.961,2251.823,2265.882,2282.779,2302.657,0.000
43,699.000,2442.997,2258.746,2357.246,2427.199,2536.878,2595.943,2624.419,2644.739,2658.998,0.000
44,701.000,2878.104,2626.830,2735.099,2866.980,2989.735,3111.465,3150.730,3177.142,3205.949,0.000
45,699.000,3581.316,3170.652,3331.126,3597.682,3821.980,3917.702,3950.740,3998.902,4044.248,0.000
46,701.000,4534.611,3974.044,4322.307,4541.282,4761.086,4865.645,4905.693,4971.559,5053.531,0.000
47,699.000,5337.989,4964.346,5181.509,5345.927,5484.555,5611.357,5656.276,5710.879,5751.211,0.000
48,700.000,6108.255,5706.703,5945.737,6101.441,6312.957,6394.282,6436.720,6465.448,6490.282,0.000
49,700.000,6833.705,6461.116,6669.292,6888.010,6988.254,7045.749,7080.865,7107.189,7117.742,0.000
50,700.000,7473.809,7095.547,7273.726,7481.619,7646.574,7770.443,7816.606,7895.764,7949.786,0.000
...-
You may want to shut down the database for any reason:
$ cd /home/postgres/BullFrog && ./shutdown.sh # Output: # ... # pg_ctl: no server running
-
You may stop a running container when you finish the tutorial:
$ docker stop bullfrog