2. Aggregate Migration

Quick Start

Using the following as a guide, we will walk you through the setup of BullFrog. To make it smoother, we record terminal sessions and share them on the web.

TODO

I. Docker Image

This tutorial assumes you have a current version of Docker installed on your machine. If you do not have Docker installed, choose your preferred operating system below to download Docker:

• Download Docker Desktop for Mac
• Download Docker Desktop for Windows
• Install Docker Engine on Linux

After you installed Docker, you can issue a command to pull our docker image and start a container:

# Pulls Docker image and run it on your local machine.
$ docker run --rm -it -d  --name bullfrog gangliao/bullfrog:agg_migration bash

# Output:
#
# Unable to find image 'gangliao/bullfrog:agg_migration' locally    
# latest: Pulling from gangliao/bullfrog
# ......: Pull complete
# ......: Pull complete
# ......: Pull complete
# Digest: sha256:......
# Status: Downloaded newer image for gangliao/bullfrog:agg_migration  

# Enters the container.
$ docker exec -u postgres -it bullfrog bash

# Output:
#
# postgres@968c58c4b04d:/$

II. BullFrog Experiments

1. Rebooting a DB instance:

   # Deploys the postgres backend
   $ cd /home/postgres/BullFrog && ./deploy.sh
   
   # Output:
   #
   # +++ case $- in
   # +++ return
   # ++ rm -rf /home/postgres/BullFrog/build/data
   # ++ initdb -D /home/postgres/BullFrog/build/data
   # The files belonging to this database system will be owned by user "postgres".
   # This user must also own the server process.
   # 
   # The database cluster will be initialized with locale "C".
   # The default database encoding has accordingly been set to "SQL_ASCII".
   # The default text search configuration will be set to "english".
   # 
   # Data page checksums are disabled.
   # 
   # creating directory /home/postgres/BullFrog/build/data ... ok
   # creating subdirectories ... ok
   # selecting default max_connections ... 100
   # selecting default shared_buffers ... 128MB
   # selecting dynamic shared memory implementation ... posix
   # creating configuration files ... ok
   # running bootstrap script ... Shared Global Bitmap created!
   # ok
   # performing post-bootstrap initialization ... ok
   # syncing data to disk ... ok
   # 
   # WARNING: enabling "trust" authentication for local connections
   # You can change this by editing pg_hba.conf or using the option -A, or
   # --auth-local and --auth-host, the next time you run initdb.
   # 
   # Success. You can now start the database server using:
   # 
   #     pg_ctl -D /home/postgres/BullFrog/build/data -l logfile start
   # 
   # ++ cp /home/postgres/postgresql.conf /home/postgres/BullFrog/build/data/
   # ++ pg_ctl -D /home/postgres/BullFrog/build/data -o '-F -p 5433' start

2. Run a TPC-C Benchmark where data is already loaded into the database.

   • Please re-execute all commands in step 2 if the benchmark fails to run.

   $ cd /home/postgres/BullFrog-Oltpbench
   
   # Clean tuples in new tables with new schemas
   $ psql -h localhost -p 5433 tpcc -f /home/postgres/BullFrog-Oltpbench/clean_new_tables.sql
   
   # Output:
   #
   # DROP TABLE
   # CREATE TABLE
   # DROP TABLE
   # ...
   
   # Clean shared memory via restarting the database
   $ pg_ctl -D $PGDATA restart 
   
   # Output:
   #
   # waiting for the server to shut down...
   # 2021-02-15 05:22:27.763 UTC [1208] LOG:  background worker "logical replication launcher" (PID 1214) exited with exit code 1
   # 2021-02-15 05:22:27.964 UTC [1210] LOG:  shutting down
   # 2021-02-15 05:22:28.557 UTC [1208] LOG:  database system is shut down
   #  done
   # server stopped
   # waiting for server to start.....2021-02-15 05:22:29.579 UTC [1275] LOG:  listening on IPv4 address "127.0.0.1", port 5433
   # 2021-02-15 05:22:29.579 UTC [1275] LOG:  could not bind IPv6 address "::1": Cannot assign requested address
   # 2021-02-15 05:22:29.579 UTC [1275] HINT:  Is another postmaster already running on port 5433? If not, wait a few seconds and retry.
   # 2021-02-15 05:22:29.579 UTC [1275] LOG:  listening on Unix socket "/tmp/.s.PGSQL.5433"
   # Shared Global Bitmap created!
   # 2021-02-15 05:22:29.863 UTC [1276] LOG:  database system was shut down at 2021-02-15 05:22:28 UTC
   # 2021-02-15 05:22:29.865 UTC [1275] LOG:  database system is ready to accept connections
   #  done
   # server started
   
   # run benchmark
   $ ./oltpbenchmark -b tpcc -c config/pgtpcc_lazy_agg.xml  --execute=true -s 1 -o lazy_agg --port=5433 --bgthread=agg
   
   # Output:
   # ...
   # 05:37:19,860 (ThreadBench.java:473) INFO  - TERMINATE :: Waiting for all terminals to finish...
   # 05:37:20,010 (ThreadBench.java:534) INFO  - Attempting to stop worker threads and collect measurements
   # 05:37:20,012 (ThreadBench.java:255) INFO  - Starting WatchDogThread
   # 05:37:20,059 (DBWorkload.java:886) INFO  - ======================================================================
   # 05:37:20,059 (DBWorkload.java:887) INFO  - Rate limited reqs/s: Results(nanoSeconds=60000173489, measuredRequests=25774) = 429.56542458540093 requests/sec
   # 05:37:20,071 (DBWorkload.java:708) INFO  - Upload Results URL: com.oltpbenchmark.util.ResultUploader@4690b489
   # 05:37:20,073 (DBWorkload.java:741) INFO  - Output Raw data into file: results/lazy_agg.csv
   # 05:37:20,597 (DBWorkload.java:760) INFO  - Output summary data into file: results/lazy_agg.summary
   # 05:37:20,625 (DBWorkload.java:767) INFO  - Output DBMS parameters into file: results/lazy_agg.params
   # 05:37:20,647 (DBWorkload.java:774) INFO  - Output DBMS metrics into file: results/lazy_agg.metrics
   # 05:37:20,720 (DBWorkload.java:781) INFO  - Output experiment config into file: results/lazy_agg.expconfig
   # 05:37:20,806 (DBWorkload.java:798) INFO  - Output throughput samples into file: results/lazy_agg.res
   # 05:37:20,806 (DBWorkload.java:801) INFO  - Grouped into Buckets of 1 seconds

Experiment Results

• Note: We have simplified the experimental environment and dataset size, so the results may be different from the paper. Since Docker for Mac runs in a LinuxKit VM, to get the best performance, please run the experiment on bare metal machines.

  time(sec), throughput(req/sec), avg_lat(ms), min_lat(ms), 25th_lat(ms), median_lat(ms), 75th_lat(ms), 90th_lat(ms), 95th_lat(ms), 99th_lat(ms), max_lat(ms), tp (req/s) scaled
  0,706.000,604.707,38.397,386.575,614.881,866.314,936.039,955.485,973.231,1019.166,0.002
  1,699.000,1306.444,957.788,1179.149,1290.722,1485.054,1571.452,1590.538,1618.129,1641.346,0.001
  2,700.000,1794.739,1606.000,1711.818,1799.713,1878.593,1944.953,1952.713,1970.791,1982.999,0.001
  3,701.000,2082.493,1945.550,2015.371,2081.222,2153.369,2184.001,2204.982,2223.718,2255.650,0.000
  4,700.000,2368.295,2204.121,2288.164,2383.718,2443.757,2475.950,2495.712,2532.740,2558.801,0.000
  5,700.000,2666.453,2523.024,2585.706,2672.411,2744.834,2753.667,2761.129,2778.356,2812.170,0.000
  6,700.000,2910.633,2759.950,2825.160,2894.810,2995.622,3049.608,3065.478,3092.609,3100.133,0.000
  7,700.000,3244.865,3081.229,3188.336,3245.358,3303.141,3350.784,3364.326,3380.812,3405.791,0.000
  8,700.000,3570.022,3368.187,3474.548,3556.151,3666.670,3735.112,3741.346,3758.226,3795.341,0.000
  9,699.000,3853.773,3742.145,3770.786,3815.900,3920.924,4019.114,4049.348,4081.640,4128.965,0.000
  10,701.000,4225.025,4066.742,4169.739,4233.320,4291.100,4313.382,4321.958,4337.348,4380.297,0.000
  11,700.000,4476.412,4319.456,4406.374,4485.085,4543.242,4596.526,4615.681,4642.886,4673.994,0.000
  12,699.000,4850.176,4634.015,4734.662,4849.103,4935.858,5042.479,5072.728,5102.400,5166.579,0.000
  13,700.000,5255.580,5087.132,5151.793,5264.432,5337.238,5387.281,5406.056,5425.608,5439.706,0.000
  14,701.000,5548.977,5408.030,5471.325,5533.413,5625.701,5674.641,5702.818,5725.521,5797.349,0.000
  15,700.000,5877.303,5713.550,5808.066,5902.311,5939.554,5981.891,5998.344,6013.861,6047.161,0.000
  16,699.000,6160.709,6001.222,6076.911,6175.495,6233.540,6263.451,6270.482,6284.167,6322.660,0.000
  17,701.000,6430.069,6273.578,6351.546,6442.974,6507.849,6542.764,6550.292,6563.209,6594.284,0.000
  18,699.000,6765.165,6537.249,6623.870,6740.428,6882.442,6998.713,7012.110,7033.431,7055.885,0.000
  19,700.000,7220.648,7017.074,7117.648,7200.361,7320.984,7398.826,7447.719,7472.587,7508.996,0.000
  20,700.000,7893.257,7443.594,7614.341,7962.098,8142.447,8195.930,8238.068,8286.179,8355.670,0.000
  21,700.000,8563.656,8265.070,8437.270,8555.370,8721.890,8778.514,8787.443,8810.966,8829.192,0.000
  22,700.000,8980.285,8792.839,8910.671,8979.200,9056.299,9120.645,9133.554,9157.795,9197.577,0.000
  23,700.000,9295.122,9148.029,9233.209,9299.492,9361.021,9400.504,9410.848,9437.850,9469.989,0.000
  24,701.000,9647.696,9425.605,9528.611,9641.645,9768.754,9830.916,9861.331,9879.398,9887.923,0.000
  25,699.000,10142.676,9867.778,9992.193,10150.004,10281.481,10360.377,10374.628,10400.839,10436.277,0.000
  26,700.000,10709.333,10388.865,10572.143,10762.311,10859.673,10901.954,10912.913,10934.449,10964.821,0.000
  27,700.000,11134.877,10923.425,11028.510,11145.307,11237.928,11284.146,11301.004,11321.450,11335.498,0.000
  28,701.000,11505.164,11307.240,11429.211,11509.850,11580.891,11635.397,11666.970,11690.608,11709.231,0.000
  29,699.000,11859.133,11678.996,11790.278,11867.790,11922.300,11993.131,12015.991,12035.648,12069.918,0.000
  30,700.000,12243.764,12026.661,12129.457,12245.277,12367.023,12416.737,12427.919,12451.469,12490.829,0.000
  31,701.000,12644.232,12432.400,12532.828,12654.714,12743.857,12793.316,12815.897,12846.126,12854.167,0.000
  32,700.000,13022.663,12830.739,12957.480,13014.651,13093.843,13165.857,13183.604,13200.486,13213.162,0.000
  33,699.000,13376.102,13188.096,13270.875,13371.738,13484.223,13531.373,13550.525,13575.961,13616.251,0.000
  34,700.000,13760.363,13563.893,13692.826,13769.482,13846.422,13885.945,13905.061,13930.640,14012.969,0.000
  35,675.000,14153.584,13918.714,14050.099,14170.890,14270.821,14292.759,14305.364,14320.601,14362.633,0.000
  36,523.000,14298.597,14278.146,14289.595,14292.982,14305.918,14313.260,14316.769,14351.586,14361.664,0.000
  37,538.000,14298.499,14283.297,14289.338,14294.433,14305.441,14312.547,14318.359,14350.558,14376.908,0.000
  38,513.000,14298.468,14277.847,14289.160,14292.756,14304.599,14314.870,14325.424,14345.484,14377.135,0.000
  39,500.000,14299.692,14279.739,14289.696,14295.147,14307.181,14314.264,14321.315,14359.855,14375.134,0.000
  40,518.000,14298.603,14280.535,14289.555,14293.492,14306.054,14313.331,14322.533,14351.456,14366.849,0.000
  41,527.000,14298.387,14280.682,14289.117,14293.139,14305.211,14312.738,14318.940,14349.782,14374.045,0.000
  42,541.000,14298.021,14280.376,14289.136,14295.094,14304.317,14312.397,14315.894,14352.568,14362.643,0.000
  43,529.000,14298.198,14279.611,14289.174,14292.481,14305.804,14312.483,14317.917,14354.628,14367.834,0.000
  44,498.000,14299.315,14279.779,14289.143,14295.831,14305.726,14314.246,14319.590,14357.639,14386.271,0.000
  45,341.000,14299.344,14283.046,14289.506,14295.326,14306.980,14313.255,14316.137,14352.062,14355.896,0.000

III. Stop Database & Container

1. You may want to shut down the database for any reason:

   $ cd /home/postgres/BullFrog && ./shutdown.sh
   
   # Output:
   #
   # ++ source /home/postgres/.bashrc
   # +++ case $- in
   # +++ return
   # ++ pg_ctl -D /home/postgres/BullFrog/build/data stop
   # waiting for server to shut down.... done
   # server stopped
   # ++ pg_ctl -D /home/postgres/BullFrog/build/data status
   # pg_ctl: no server running

2. You may stop a running container when you finish the tutorial:

   $ docker stop bullfrog