-
Notifications
You must be signed in to change notification settings - Fork 1
6. Eager Migration
Gang Liao edited this page Mar 10, 2021
·
19 revisions
In eager migration, the system immediately physically moves all data stored under the old schema into tables in the new schema prior to becoming available to client requests over the new schema.
The implementation of eager migration is based on PostgreSQL with zero changes. This post will show you how to set up the raw PostgreSQL and how to run the TPC-C benchmark on top of that.
Using the following as a guide, we will walk you through the setup of BullFrog. To make it smoother, we record terminal sessions and share them on the web.
TODO
This tutorial assumes you have a current version of Docker installed on your machine. If you do not have Docker installed, choose your preferred operating system below to download Docker:
After you installed Docker, you can issue a command to pull our docker image and start a container:
# Pulls Docker image and run it on your local machine.
$ docker run --rm -it -d --name bullfrog gangliao/bullfrog:eager_migration bash
# Enters the container.
$ docker exec -u postgres -it bullfrog bash
# Output:
#
# postgres@968c58c4b04d:/$-
Deploy a DB instance:
# Deploy the database $ cd /home/postgres/BullFrog && ./deploy.sh
-
Run a TPC-C Benchmark (without migration) where data is already loaded into the database.
$ ./oltpbenchmark -b tpcc -c config/pgtpcc_config.xml --execute=true -s 1 -o tpcc --port 5433
-
Schema migration phase and Post-migration phase.
# You must use `\timing` and migration commands to calculate the approximate downtime time, # and use it to set the corresponding configuration file: `config/pgtpcc_base_proj.xml`. # # For example, table split migration: # # $ psql -h localhost -p 5433 tpcc # $ \timing # $ insert into customer_proj1( # c_w_id, c_d_id, c_id, c_discount, c_credit, c_last, c_first, # c_balance, c_ytd_payment, c_payment_cnt, c_delivery_cnt, c_data) # (select # c_w_id, c_d_id, c_id, c_discount, c_credit, c_last, c_first, # c_balance, c_ytd_payment, c_payment_cnt, c_delivery_cnt, c_data # from customer) on conflict (c_w_id,c_d_id,c_id) do nothing; # # $ insert into customer_proj2( # c_w_id, c_d_id, c_id, c_last, c_first, # c_street_1, c_city, c_state, c_zip) # (select # c_w_id, c_d_id, c_id, c_last, c_first, # c_street_1, c_city, c_state, c_zip # from customer) on conflict (c_w_id,c_d_id,c_id) do nothing; $ ./oltpbenchmark -b tpcc -c config/pgtpcc_base_proj.xml --execute=true -s 1 -o tpcc --port 5433 --migration=6
Experiment Results
- Note: We have simplified the experimental environment and dataset size, so the results may be different from the paper. Since Docker for Mac runs in a LinuxKit VM, to get the best performance, please run the experiment on bare metal machines.
time(sec), throughput(req/sec), avg_lat(ms), min_lat(ms), 25th_lat(ms), median_lat(ms), 75th_lat(ms), 90th_lat(ms), 95th_lat(ms), 99th_lat(ms), max_lat(ms), tp (req/s) scaled
0,29.000,61.010,4.673,5.837,6.731,13.193,18.549,29.741,1521.178,1521.178,0.016
1,25.000,6.346,4.585,5.157,5.994,6.825,8.937,9.816,10.793,10.793,0.158
2,31.000,8.102,4.658,5.826,6.452,11.031,13.943,14.791,17.106,17.106,0.123
3,28.000,6.565,4.266,5.699,6.146,6.653,8.621,8.641,15.831,15.831,0.152
4,33.000,5.874,4.461,5.311,5.846,6.428,6.789,7.053,7.461,7.461,0.170
5,33.000,5.870,4.350,5.535,5.881,6.192,6.824,7.009,7.151,7.151,0.170
6,35.000,5.617,3.926,4.704,5.373,6.265,7.211,9.299,9.648,9.648,0.178
7,30.000,5.534,3.904,5.110,5.534,6.256,6.659,6.798,6.897,6.897,0.181
8,30.000,5.548,3.755,4.917,5.448,6.011,6.764,7.236,10.674,10.674,0.180
9,35.000,5.263,3.886,4.858,5.404,5.741,6.117,6.183,6.240,6.240,0.190
10,29.000,5.342,4.284,4.994,5.367,5.777,6.211,6.230,6.321,6.321,0.187
11,31.000,5.486,4.197,4.965,5.450,5.853,6.460,6.970,7.011,7.011,0.182
12,27.000,4.955,3.552,4.341,5.015,5.557,5.923,5.941,6.142,6.142,0.202
13,25.000,5.289,4.062,5.032,5.381,5.565,6.076,6.181,6.562,6.562,0.189
14,26.000,4.949,3.918,4.604,5.023,5.387,5.466,5.502,5.688,5.688,0.202
15,25.000,5.152,3.558,4.808,5.226,5.584,5.986,6.070,6.175,6.175,0.194
16,24.000,5.290,3.989,5.026,5.373,5.745,5.929,6.010,6.046,6.046,0.189
17,34.000,5.149,3.601,4.734,5.250,5.537,6.084,6.258,6.383,6.383,0.194
18,30.000,5.037,3.619,4.706,5.104,5.451,5.752,5.891,6.954,6.954,0.199
19,24.000,4.916,3.664,4.453,5.104,5.425,5.553,5.737,5.829,5.829,0.203
20,22.000,5.181,4.330,4.644,5.254,5.673,5.810,5.838,6.565,6.565,0.193
21,22.000,5.057,3.964,4.651,5.027,5.645,5.880,5.903,5.997,5.997,0.198
22,39.000,4.855,3.274,4.531,4.886,5.160,5.512,5.568,5.769,5.769,0.206
23,28.000,5.659,3.896,4.810,5.395,6.021,7.227,9.168,11.979,11.979,0.177
24,24.000,5.772,4.035,5.072,5.606,6.526,7.186,7.334,10.369,10.369,0.173
25,29.000,6.034,4.506,5.518,5.704,5.808,8.214,9.163,11.089,11.089,0.166
26,28.000,5.384,4.327,4.960,5.424,5.862,5.968,6.129,6.420,6.420,0.186
27,26.000,5.550,4.109,4.839,5.506,6.160,6.597,6.789,8.079,8.079,0.180
28,35.000,5.358,4.215,4.902,5.309,5.696,6.284,6.390,7.822,7.822,0.187
29,34.000,5.024,3.130,4.523,4.954,5.589,6.108,6.581,8.056,8.056,0.199
30,27.000,5.278,4.524,4.905,5.322,5.617,5.713,5.729,5.965,5.965,0.189
31,34.000,5.112,4.114,4.775,5.117,5.636,5.714,5.750,5.824,5.824,0.196
32,31.000,5.399,3.538,4.942,5.457,5.978,6.256,6.633,7.225,7.225,0.185
33,25.000,5.294,4.119,4.778,5.317,5.863,6.293,6.320,6.462,6.462,0.189
34,40.000,5.116,3.566,4.493,5.119,5.854,6.127,6.299,6.326,6.326,0.195
35,31.000,5.093,3.854,4.643,4.891,5.558,6.014,6.460,6.975,6.975,0.196
36,31.000,5.091,3.822,4.677,4.931,5.564,5.981,6.231,6.453,6.453,0.196
37,23.000,5.246,3.919,4.754,5.416,5.657,5.770,5.888,6.475,6.475,0.191
38,34.000,5.196,3.949,4.776,5.289,5.633,5.906,5.989,6.123,6.123,0.192
39,30.000,5.222,4.091,4.917,5.247,5.582,5.787,5.886,6.074,6.074,0.191
40,702.000,735.114,13.961,497.717,865.406,974.572,1061.288,1089.200,1111.580,1148.561,0.001
41,699.000,1526.345,1098.778,1336.479,1535.443,1711.824,1839.570,1872.561,1900.479,1907.083,0.001
42,701.000,2101.749,1885.498,2022.445,2114.554,2202.961,2251.823,2265.882,2282.779,2302.657,0.000
43,699.000,2442.997,2258.746,2357.246,2427.199,2536.878,2595.943,2624.419,2644.739,2658.998,0.000
44,701.000,2878.104,2626.830,2735.099,2866.980,2989.735,3111.465,3150.730,3177.142,3205.949,0.000
45,699.000,3581.316,3170.652,3331.126,3597.682,3821.980,3917.702,3950.740,3998.902,4044.248,0.000
46,701.000,4534.611,3974.044,4322.307,4541.282,4761.086,4865.645,4905.693,4971.559,5053.531,0.000
47,699.000,5337.989,4964.346,5181.509,5345.927,5484.555,5611.357,5656.276,5710.879,5751.211,0.000
48,700.000,6108.255,5706.703,5945.737,6101.441,6312.957,6394.282,6436.720,6465.448,6490.282,0.000
49,700.000,6833.705,6461.116,6669.292,6888.010,6988.254,7045.749,7080.865,7107.189,7117.742,0.000
50,700.000,7473.809,7095.547,7273.726,7481.619,7646.574,7770.443,7816.606,7895.764,7949.786,0.000
...-
You may want to shut down the database for any reason:
$ cd /home/postgres/BullFrog && ./shutdown.sh # Output: # ... # pg_ctl: no server running
-
You may stop a running container when you finish the tutorial:
$ docker stop bullfrog