Use s3 for backups #432

magikid · 2023-06-08T18:54:15Z

Relates to #431

The infrastructure is managed through terraform cloud in https://github.com/Friends-of-DeSoto/digitalocean-terraform

magikid · 2023-06-09T15:06:03Z

.env-example

+export AWS_ACCESS_KEY_ID=
+export AWS_SECRET_ACCESS_KEY=


Required to write or read the bucket

magikid · 2023-06-09T15:06:41Z

.env-example

@@ -28,6 +30,7 @@ export DB_BACKUP_DEPLOY_KEY=Tm90IGFjdHVhbGx5IGEgcHJpdmF0ZSBrZXkuLi4g8J+ZhAo=
 export DB_BACKUP_SUB_DIR=agimus
 export DB_BACKUP_GITHUB_USER=
 export DB_BACKUP_GITHUB_EMAIL=
+export S3_BUCKET_NAME=aggy-brain-backups


The bucket needs to already exist otherwise the backups will fail. This bucket does.

magikid · 2023-06-09T15:07:04Z

Dockerfile

@@ -5,7 +5,7 @@ FROM ubuntu:20.04
 RUN rm /bin/sh && ln -s /bin/bash /bin/sh \
    && apt-get update \
    && DEBIAN_FRONTEND=noninteractive apt-get install -y \
-        curl wget apt-utils python3 python3-pip make build-essential locales openssl git jq tzdata sudo lsb-release mysql-client \
+        curl wget apt-utils python3 python3-pip make build-essential locales openssl git jq tzdata sudo lsb-release mysql-client s3cmd \


This is a handy tool to interact with s3-like APIs.

magikid · 2023-06-09T15:07:29Z

Makefile

@@ -17,6 +17,10 @@ help: ## Displays this help dialog (to set repo/fork ownker REPO_OWNWER=[github-
 	@echo ""
 	@awk 'BEGIN {FS = ":.*##"; printf "\nUsage:\n  make \033[36m<target>\033[0m\n"} /^[a-zA-Z_0-9-]+:.*?##/ { printf "  \033[36m%-15s\033[0m %s\n", $$1, $$2 } /^##@/ { printf "\n\033[1m%s\033[0m\n", substr($$0, 5) } ' $(MAKEFILE_LIST)

+define guard


This will check that a particular environment variable is defined before allowing execution to pass.

magikid · 2023-06-09T15:08:09Z

Makefile

@@ -95,50 +99,30 @@ db-seed: ## Reload the database from a file at $DB_SEED_FILEPATH
 	@docker-compose exec -T app mysql -h$(DB_HOST) -u$(DB_USER) -p$(DB_PASS) <<< "create database FoD;"
 	@docker-compose exec -T app mysql -h$(DB_HOST) -u$(DB_USER) -p$(DB_PASS) $(DB_NAME) < $(DB_SEED_FILEPATH)

+DB_DUMP_S3_PREFIX=$(shell date +%Y-%m-%d)


Each backup is stored in a folder like bucket/2023-07-09/ which makes it easier to find the most recent bucket with sort.

magikid · 2023-06-09T15:09:07Z

cogs/backups.py

@@ -3,7 +3,6 @@
 class Backups(commands.Cog):
  def __init__(self, bot):
    self.bot = bot
-    self.commit_base_url = "https://github.com/Friends-of-DeSoto/database/commit/"


Since the point of this PR is to stop using the repo, no need to have this.

magikid · 2023-06-09T15:11:20Z

cogs/backups.py

    backup_hashes = run_make_backup()
    if not backup_hashes:
      await ctx.send(f"Something went wrong with the backup! No databases have been wiped out, I don't think.")
    else:
      embed = discord.Embed(
-        title=f"BACKUP {backup_hashes['new']} COMPLETE",
+        title=f"BACKUP {backup_hashes['backup_name']} COMPLETE",


The backup_name will be a string like s3://aggy-brain-backups/2023-06-09/bot-dump.sql-1686322965.sql.gz.

magikid · 2023-06-09T15:12:17Z

cogs/backups.py

+      embed.add_field(name="🌟 NEW BACKUP 🌟", value=f"`{backup_hashes['backup_name']}`", inline=False)
+      embed.add_field(name="Presigned URL (valid 15m) to new backup", value=f"{backup_hashes['url']}", inline=False)


There's a new make command to generate a pre-signed URL that allows anyone with the link to access it for 15 minutes so that whoever kicked off the backup can download it. That time limit can always be changed in the Makefile.

magikid · 2023-06-09T15:13:41Z

scripts/db-backup.sh

+set -euo pipefail
+IFS=$'\n\t'


Unofficial bash strict mode

magikid · 2023-06-09T15:15:16Z

scripts/db-backup.sh

+function setupDotEnv() {
+    if [ -f .env ]
+    then
+        set -o allexport; source .env; set +o allexport


A handy way in bash and zsh to load and export environment variables from a file. source

zmattingly

Looks good to me but would still like @mathew-fleisch to give it the official 👍

magikid · 2024-02-09T17:43:58Z

This slide right off my radar. Is this still something y'all are interested in? If not, I'll close out this PR and tear down the infra.

ping @zmattingly @mathew-fleisch @jp00p

zmattingly · 2024-02-09T18:15:47Z

This slide right off my radar. Is this still something y'all are interested in? If not, I'll close out this PR and tear down the infra.

ping @zmattingly @mathew-fleisch @jp00p

Looks good to me, but I'm also not really a devops guy which is why I was checking with Drax. I'll ping him on the Discord as well, but also wanted to ask 2 quick questions:

Do you have a AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY you'd be able to provide for @jp00p, or will we need a new AWS account for this? One of the appeals of Drax's approach was that it was free, but I'd be willing to maybe toss in a few bucks or whatever it would require to get the account for this set up (?).
Have you been able to test the full process here using your own bot on The Mirror Hood? I forget if you have one set up there.

Once we have it ready to go, you'll also want to bump the version in Chart.yaml (we're not strictly using semver really so a minor or even major bump would be fine), and then resolve that .env-example conflict obviously.

Thanks for the effort and time on this, much appreciated @magikid!

magikid · 2024-02-10T01:43:20Z

Yeah, I have the keys for jp00p but I've also invited all y'all to the team on DigitalOcean. I haven't tested it lately but I think I did because I see backups in the bucket from around the time I opened this. I'll try to find some time to test again.

This totally reworks how and where database backups are stored. Instead of storing database dumps in a git repo, store them in a DigitalOcean bucket.

magikid · 2024-06-11T20:41:11Z

Migrated to #532

magikid commented Jun 9, 2023

View reviewed changes

zmattingly requested a review from mathew-fleisch June 12, 2023 02:29

zmattingly approved these changes Jun 12, 2023

View reviewed changes

magikid added 8 commits May 30, 2024 20:39

Update db-backup and db-restore targets in Makefile

134d6e7

This totally reworks how and where database backups are stored. Instead of storing database dumps in a git repo, store them in a DigitalOcean bucket.

Make finding latest day easier

5731269

Update python files for new backups

3ea1e49

Autowire s3 key and secret with proper env var names

d53bf1c

Add host and bucket location to all s3cmd commands

0832452

Fix db backup command

d78d73d

Add config flags for s3cmd to config file

6e3980d

Cleanup tasks backup command

b9655ed

magikid force-pushed the use-s3-for-backups branch from be6be3d to b9655ed Compare May 31, 2024 00:40

magikid mentioned this pull request Jun 11, 2024

Use s3 for backups #532

Open

magikid closed this Jun 11, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Use s3 for backups #432

Use s3 for backups #432

magikid commented Jun 8, 2023 •

edited

Loading

magikid Jun 9, 2023

magikid Jun 9, 2023

magikid Jun 9, 2023

magikid Jun 9, 2023

magikid Jun 9, 2023

magikid Jun 9, 2023

magikid Jun 9, 2023

magikid Jun 9, 2023

magikid Jun 9, 2023

magikid Jun 9, 2023

zmattingly left a comment

magikid commented Feb 9, 2024

zmattingly commented Feb 9, 2024

magikid commented Feb 10, 2024

magikid commented Jun 11, 2024

		embed.add_field(name="🌟 NEW BACKUP 🌟", value=f"`{backup_hashes['backup_name']}`", inline=False)
		embed.add_field(name="Presigned URL (valid 15m) to new backup", value=f"{backup_hashes['url']}", inline=False)

Use s3 for backups #432

Use s3 for backups #432

Conversation

magikid commented Jun 8, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

zmattingly left a comment

Choose a reason for hiding this comment

magikid commented Feb 9, 2024

zmattingly commented Feb 9, 2024

magikid commented Feb 10, 2024

magikid commented Jun 11, 2024

magikid commented Jun 8, 2023 •

edited

Loading