feat(postgres): install wal-e for continuous backups

[bacongobbler]

To test, build the image, modify the deis-database-rc manifest in the deis-dev chart then run it:

$ helm install deis-dev
$ kubectl --namespace=deis get po
$ kubectl --namespace=deis logs -f deis-database-ecqur # ensure database is up
$ kubectl --namespace=deis exec -it deis-database-ecqur gosu postgres psql
postgres=# create database foo;
postgres=# \q
$ kubectl --namespace=deis delete po deis-database-ecqur

After that, ensure the new database pod comes back up and has the foo database:

$ kubectl --namespace=deis exec -it deis-database-asdqr gosu postgres psql
postgres=# \l
                                            List of databases
               Name               |  Owner   | Encoding |  Collate   |   Ctype    |   Access privileges   
----------------------------------+----------+----------+------------+------------+-----------------------
 eJmZkwZc4CFuzg641lEoxuX8tTrnC9lr | postgres | UTF8     | en_US.utf8 | en_US.utf8 | 
 postgres                         | postgres | UTF8     | en_US.utf8 | en_US.utf8 | 
 template0                        | postgres | UTF8     | en_US.utf8 | en_US.utf8 | =c/postgres          +
                                  |          |          |            |            | postgres=CTc/postgres
 template1                        | postgres | UTF8     | en_US.utf8 | en_US.utf8 | =c/postgres          +
                                  |          |          |            |            | postgres=CTc/postgres
 foo                              | postgres | UTF8     | en_US.utf8 | en_US.utf8 | 
(5 rows)

closes #25
closes #31
closes #32

[bacongobbler]

I'm going to remove the tests while I figure out minio failures in another issue (incoming), but this is ready for review.

[bacongobbler]

removing tests for now, to be implemented in the future in #41 so we can get this out the door.

[bacongobbler]

pinging for reviews

[carmstrong]

It's in the pipeline but I didn't want it to block this so I opted for the manual instructions this time.

Manual testing is fine - I meant to use that script to validate the PR. I think we need more testing on the backup/recovery scenarios before this is merged. Just ensuring the database has recovered in a simple case doesn't fully exercise the numerous disaster scenarios we've seen in production.

[bacongobbler]

etcd's no longer a dependency which means half of the network flakiness issues are gone. It's just postgres and minio so we got that going for us, which is nice. :)

I agree that a better backup/recovery scenario is required for testing. What kind of scenario would you suggest us test? Keep in mind that the beta target is just "Testing: single node failure, recovery" so I just requested the bare minimum here to pass acceptance criteria.

[carmstrong]

Keep in mind that the beta target is just "Testing: single node failure, recovery" so I just requested the bare minimum here to pass acceptance criteria.

Right, but I don't think we've fully tested recovery. Testing single-node backup/recovery under load is what I'm interested in:

We should test by writing a script that inserts a ton of things into the DB, pull the plug, let the DB recover, and then see what the window of loss is.

[bacongobbler]

Sure. I'll write up a sql script to dump a bunch of stuff into a database, kill it off and see what happens :)

[bacongobbler]

manually tested and working with no data loss (woop!)

><> kd get po
NAME                  READY     STATUS    RESTARTS   AGE
deis-builder-d0nqp    1/1       Running   1          1m
deis-database-0ml44   1/1       Running   1          1m
deis-minio-haqai      1/1       Running   0          1m
deis-registry-l4slz   1/1       Running   0          1m
deis-router-t8xfh     1/1       Running   0          1m
deis-workflow-bjzar   1/1       Running   2          1m
><> deis init
Registered bacongobbler
Logged in as bacongobbler
Uploading id_rsa.pub to deis... done
><> for i in $(seq 1 100); do deis create foo-$i --no-remote &>/dev/null; done
><> kd delete po deis-database-0ml44
pods/deis-database-0ml44
><> kd get po
NAME                  READY     STATUS    RESTARTS   AGE
deis-builder-d0nqp    1/1       Running   1          2m
deis-database-0ml44   0/1       Error     1          2m
deis-database-ofv8s   0/1       Pending   0          4s
deis-minio-haqai      1/1       Running   0          2m
deis-registry-l4slz   1/1       Running   0          2m
deis-router-t8xfh     1/1       Running   0          2m
deis-workflow-bjzar   1/1       Running   2          2m
><> kd get po # notice the API is unresponsive due to its internal health check
NAME                  READY     STATUS    RESTARTS   AGE
deis-builder-d0nqp    1/1       Running   1          2m
deis-database-ofv8s   1/1       Running   0          47s
deis-minio-haqai      1/1       Running   0          2m
deis-registry-l4slz   1/1       Running   0          2m
deis-router-t8xfh     1/1       Running   0          2m
deis-workflow-bjzar   0/1       Running   3          2m
><> kd get po
NAME                  READY     STATUS    RESTARTS   AGE
deis-builder-d0nqp    1/1       Running   1          3m
deis-database-ofv8s   1/1       Running   0          1m
deis-minio-haqai      1/1       Running   0          3m
deis-registry-l4slz   1/1       Running   0          3m
deis-router-t8xfh     1/1       Running   0          3m
deis-workflow-bjzar   1/1       Running   3          3m
><> deis apps:list | wc -l
101

There is 101 lines because of the === Apps line that is displayed when you run that command.

Some changes that were necessary to get this working:

• had to add listen_addresses = '' to postgresql.conf after restoring it. docker-entrypoint.sh uses sed to replace this config with listen_addresses = '*' after initialization is complete, so workflow was not able to connect. All my local testing had passed because I was execing into the container after postgres rebooted, which wouldn't cause this error to occur.

Ready for another round of reviews

[bacongobbler]

So the problem @mboersma was seeing with the database being restarted was because the /bin/is_master script would check that the database is in recovery. When wal-e is used, on a restart with data stored inside minio, the database is considered always in recovery, causing the health check to fail.

I've gone ahead and changed the health check behaviour to only check pg_ctl status. For now the solution is to use deis/charts#117 while testing this, as /bin/is_master is no longer relevant. /bin/is_running is the new healthcheck command, and as such requires a chart change. More extensive health checking will have to be implemented in the future to ensure that the database is ready to accept incoming connections, and not just that postgres is running.

[bacongobbler]

After these changes I was still able to confirm that only < 0.1% of data loss had occurred. I CTRL+C'd a job after a total of 34,551 applications and 35 users were created with this script:

for i in $(seq 1 1000); do deis register http://deis.$DEIS_TEST_DOMAIN --username bacongobbler-$i --password secret --email me@bacongobbler.com; for j in $(seq 1 1000); do deis create foo-$i-$j --no-remote &>/dev/null; done; done

This was the final log on the controller side before CTRL+C'ing the job:

INFO foo-35-551: release foo-35-551-v1 created

After killing the database and restarting it, I then jumped into the controller:

><> kd exec -it deis-workflow-vs0e4 python manage.py shell
Python 3.5.1 (default, Dec  9 2015, 14:41:32) 
[GCC 5.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
(InteractiveConsole)
>>> from api.models import App 
>>> App.objects.count()   
34545

And showing from the client:

><> for i in $(seq 1 35); do deis login http://deis.$DEIS_TEST_DOMAIN --username bacongobbler-$i --password bacon; deis apps:list | head -n 1; done
Logged in as bacongobbler-1
=== Apps (100 of 34545)
Logged in as bacongobbler-2
=== Apps (100 of 1000)
Logged in as bacongobbler-3
=== Apps (100 of 1000)
Logged in as bacongobbler-4
=== Apps (100 of 1000)
Logged in as bacongobbler-5
=== Apps (100 of 1000)
Logged in as bacongobbler-6
=== Apps (100 of 1000)
Logged in as bacongobbler-7
=== Apps (100 of 1000)
Logged in as bacongobbler-8
=== Apps (100 of 1000)
Logged in as bacongobbler-9
=== Apps (100 of 1000)
Logged in as bacongobbler-10
=== Apps (100 of 1000)
Logged in as bacongobbler-11
=== Apps (100 of 1000)
Logged in as bacongobbler-12
=== Apps (100 of 1000)
Logged in as bacongobbler-13
=== Apps (100 of 1000)
Logged in as bacongobbler-14
=== Apps (100 of 1000)
Logged in as bacongobbler-15
=== Apps (100 of 994)
Logged in as bacongobbler-16
=== Apps (100 of 1000)
Logged in as bacongobbler-17
=== Apps (100 of 1000)
Logged in as bacongobbler-18
=== Apps (100 of 1000)
Logged in as bacongobbler-19
=== Apps (100 of 1000)
Logged in as bacongobbler-20
=== Apps (100 of 1000)
Logged in as bacongobbler-21
=== Apps (100 of 1000)
Logged in as bacongobbler-22
=== Apps (100 of 1000)
Logged in as bacongobbler-23
=== Apps (100 of 1000)
Logged in as bacongobbler-24
=== Apps (100 of 1000)
Logged in as bacongobbler-25
=== Apps (100 of 1000)
Logged in as bacongobbler-26
=== Apps (100 of 1000)
Logged in as bacongobbler-27
=== Apps (100 of 1000)
Logged in as bacongobbler-28
=== Apps (100 of 1000)
Logged in as bacongobbler-29
=== Apps (100 of 1000)
Logged in as bacongobbler-30
=== Apps (100 of 1000)
Logged in as bacongobbler-31
=== Apps (100 of 1000)
Logged in as bacongobbler-32
=== Apps (100 of 1000)
Logged in as bacongobbler-33
=== Apps (100 of 1000)
Logged in as bacongobbler-34
=== Apps (100 of 1000)
Logged in as bacongobbler-35
=== Apps (100 of 551)

Which looks like client number 15 lost 6 applications; a total of 0.02% data loss.

[bacongobbler]

Keep in mind that this is also an extreme example. Realistically we've only seen clusters with at most 500 containers running, which is only one or two MBs at maximum. This particular pg_dump file is 16MB in size, and the database was killed while undergoing some heavy processing. This likely won't happen in the wild, but as a worst case scenario I'd say this test went well above and beyond the expected.

[jchauncey]

I am getting the following error -

core@k8solo-01 ~ $ docker logs 98fd7503a778
The files belonging to this database system will be owned by user "postgres".
This user must also own the server process.

The database cluster will be initialized with locale "en_US.utf8".
The default database encoding has accordingly been set to "UTF8".
The default text search configuration will be set to "english".

Data page checksums are disabled.

fixing permissions on existing directory /var/lib/postgresql/data ... ok
creating subdirectories ... ok
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting dynamic shared memory implementation ... posix
creating configuration files ... ok
creating template1 database in /var/lib/postgresql/data/base/1 ... ok
initializing pg_authid ... ok
initializing dependencies ... ok
creating system views ... ok
loading system objects' descriptions ... ok
creating collations ... ok
creating conversions ... ok
creating dictionaries ... ok
setting privileges on built-in objects ... ok
creating information schema ... ok
loading PL/pgSQL server-side language ... ok
vacuuming database template1 ... ok
copying template1 to template0 ... ok
copying template1 to postgres ... ok

WARNING: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
syncing data to disk ... ok

Success. You can now start the database server using:

    postgres -D /var/lib/postgresql/data
or
    pg_ctl -D /var/lib/postgresql/data -l logfile start

waiting for server to start....LOG:  database system was shut down at 2016-02-25 21:09:36 UTC
LOG:  MultiXact member wraparound protections are now enabled
LOG:  database system is ready to accept connections
LOG:  autovacuum launcher started
 done
server started
CREATE DATABASE

CREATE ROLE


/docker-entrypoint.sh: running /docker-entrypoint-initdb.d/001_setup_envdir.sh

/docker-entrypoint.sh: running /docker-entrypoint-initdb.d/002_create_bucket.sh

/docker-entrypoint.sh: running /docker-entrypoint-initdb.d/003_restore_from_backup.sh
Rebooting postgres to enable archive mode
waiting for server to shut down...LOG:  received smart shutdown request
.LOG:  autovacuum launcher shutting down
LOG:  shutting down
LOG:  database system is shut down
 done
server stopped
waiting for server to start....LOG:  database system was shut down at 2016-02-25 21:09:38 UTC
LOG:  MultiXact member wraparound protections are now enabled
LOG:  database system is ready to accept connections
LOG:  autovacuum launcher started
 done
server started
No backups found. Performing an initial backup...
wal_e.main   INFO     MSG: starting WAL-E
        DETAIL: The subcommand is "backup-push".
        STRUCTURED: time=2016-02-25T21:09:40.517480-00 pid=107
wal_e.operator.backup INFO     MSG: start upload postgres version metadata
        DETAIL: Uploading to s3://dbwal/basebackups_005/base_000000010000000000000002_00000040/extended_version.txt.
        STRUCTURED: time=2016-02-25T21:09:40.838975-00 pid=107
wal_e.blobstore.s3.calling_format INFO     MSG: connecting to S3 with WALE_S3_ENDPOINT (http://10.100.159.241:9000)
        STRUCTURED: time=2016-02-25T21:09:40.839407-00 pid=107
botocore.credentials INFO     Found credentials in environment variables.
boto3.resources.action INFO     Calling s3:put_object with {'Body': <cStringIO.StringI object at 0x7f5f94d083e8>, u'Bucket': 'dbwal', 'Key': 'basebackups_005/base_000000010000000000000002_00000040/extended_version.txt'}
botocore.vendored.requests.packages.urllib3.connectionpool INFO     Starting new HTTP connection (1): 10.100.159.241
boto3.resources.action INFO     Calling s3:head_object with {u'Bucket': 'dbwal', u'Key': 'basebackups_005/base_000000010000000000000002_00000040/extended_version.txt'}
wal_e.operator.backup INFO     MSG: postgres version metadata upload complete
        STRUCTURED: time=2016-02-25T21:09:40.886791-00 pid=107
wal_e.worker.upload INFO     MSG: beginning volume compression
        DETAIL: Building volume 0.
        STRUCTURED: time=2016-02-25T21:09:40.949183-00 pid=107
wal_e.worker.upload INFO     MSG: begin uploading a base backup volume
        DETAIL: Uploading to "s3://dbwal/basebackups_005/base_000000010000000000000002_00000040/tar_partitions/part_00000000.tar.lzo".
        STRUCTURED: time=2016-02-25T21:09:41.288625-00 pid=107
wal_e.blobstore.s3.calling_format INFO     MSG: connecting to S3 with WALE_S3_ENDPOINT (http://10.100.159.241:9000)
        STRUCTURED: time=2016-02-25T21:09:41.389762-00 pid=107
boto3.resources.action INFO     Calling s3:put_object with {'Body': <open file '<fdopen>', mode 'r+b' at 0x7f5f944cc4b0>, u'Bucket': 'dbwal', 'Key': 'basebackups_005/base_000000010000000000000002_00000040/tar_partitions/part_00000000.tar.lzo'}
botocore.vendored.requests.packages.urllib3.connectionpool INFO     Starting new HTTP connection (1): 10.100.159.241
boto3.resources.action INFO     Calling s3:head_object with {u'Bucket': 'dbwal', u'Key': 'basebackups_005/base_000000010000000000000002_00000040/tar_partitions/part_00000000.tar.lzo'}
wal_e.worker.upload INFO     MSG: finish uploading a base backup volume
        DETAIL: Uploading to "s3://dbwal/basebackups_005/base_000000010000000000000002_00000040/tar_partitions/part_00000000.tar.lzo" complete at 23912.4KiB/s.
        STRUCTURED: time=2016-02-25T21:09:41.490857-00 pid=107
NOTICE:  WAL archiving is not enabled; you must ensure that all required WAL segments are copied through other means to complete the backup
wal_e.blobstore.s3.calling_format INFO     MSG: connecting to S3 with WALE_S3_ENDPOINT (http://10.100.159.241:9000)
        STRUCTURED: time=2016-02-25T21:09:41.524566-00 pid=107
boto3.resources.action INFO     Calling s3:put_object with {'Body': <cStringIO.StringO object at 0x7f5f94720ce0>, u'Bucket': 'dbwal', 'Key': 'basebackups_005/base_000000010000000000000002_00000040_backup_stop_sentinel.json'}
botocore.vendored.requests.packages.urllib3.connectionpool INFO     Starting new HTTP connection (1): 10.100.159.241
boto3.resources.action INFO     Calling s3:head_object with {u'Bucket': 'dbwal', u'Key': 'basebackups_005/base_000000010000000000000002_00000040_backup_stop_sentinel.json'}

/docker-entrypoint.sh: running /docker-entrypoint-initdb.d/004_setup_backup_restore.sh

LOG:  received fast shutdown request
LOG:  aborting any active transactions
LOG:  autovacuum launcher shutting down
LOG:  shutting down
waiting for server to shut down....LOG:  database system is shut down
 done
server stopped

PostgreSQL init process complete; ready for start up.

LOG:  database system was shut down at 2016-02-25 21:09:41 UTC
LOG:  MultiXact member wraparound protections are now enabled
LOG:  database system is ready to accept connections
LOG:  autovacuum launcher started

[bacongobbler]

that looks like the database is running. Are you running the database with deis/charts#117?

[bacongobbler]

@jchauncey I wasn't able to reproduce your issue of not being able to see the foo database on reboot with either vagrant or on AWS. Perhaps it's something specific to kube-solo?

><> k get no
NAME                                         LABELS                                                              STATUS
ip-172-20-0-194.us-west-2.compute.internal   kubernetes.io/hostname=ip-172-20-0-194.us-west-2.compute.internal   Ready
ip-172-20-0-195.us-west-2.compute.internal   kubernetes.io/hostname=ip-172-20-0-195.us-west-2.compute.internal   Ready
ip-172-20-0-196.us-west-2.compute.internal   kubernetes.io/hostname=ip-172-20-0-196.us-west-2.compute.internal   Ready
><> kd delete po deis-database-m7ill
><> kd get po
NAME                  READY     STATUS    RESTARTS   AGE
deis-builder-lgsfs    1/1       Running   10         16m
deis-database-86ey4   1/1       Running   0          1m
deis-minio-9dkxn      1/1       Running   0          9m
deis-registry-h8vxe   1/1       Running   0          9m
deis-router-savsw     1/1       Running   3          9m
deis-workflow-qezy9   1/1       Running   2          9m
><> kd exec -it deis-database-86ey4 gosu postgres psql
psql: FATAL:  the database system is starting up
><> kd exec -it deis-database-86ey4 gosu postgres psql
psql (9.4.6)
Type "help" for help.

postgres=# \l
                                            List of databases
               Name               |  Owner   | Encoding |  Collate   |   Ctype    |   Access privileges   
----------------------------------+----------+----------+------------+------------+-----------------------
 PQbai6uGDdNqvD8vSZbWpHOhro1GeugV | postgres | UTF8     | en_US.utf8 | en_US.utf8 | 
 foo                              | postgres | UTF8     | en_US.utf8 | en_US.utf8 | 
 postgres                         | postgres | UTF8     | en_US.utf8 | en_US.utf8 | 
 template0                        | postgres | UTF8     | en_US.utf8 | en_US.utf8 | =c/postgres          +
                                  |          |          |            |            | postgres=CTc/postgres
 template1                        | postgres | UTF8     | en_US.utf8 | en_US.utf8 | =c/postgres          +
                                  |          |          |            |            | postgres=CTc/postgres
(5 rows)

[jchauncey]

Ok I'll try testing it on my aws cluster.

[jchauncey]

Alright it seems to work just fine on AWS so its definitely something with running this on kube-solo.