Skip to content

Commit

Permalink
datastore backup: document in readme, add cloud storage lifecycle pol…
Browse files Browse the repository at this point in the history
…icy. fixes #233
  • Loading branch information
snarfed committed Jul 24, 2014
1 parent 31f9972 commit 7747a8d
Show file tree
Hide file tree
Showing 2 changed files with 27 additions and 11 deletions.
29 changes: 18 additions & 11 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -43,17 +43,24 @@ dependencies, make sure the patches are included!

Misc
---
I've made manual datastore backups, but we don't yet have automated backup set
up. Bridgy isn't the canonical source of any actual user data, so it's not as
important as for other services, but at minimum, the registered users and their
OAuth tokens would be bad to lose. We should probably figure out a real backup
system.

The most likely candidate is
[Datastore Admin](https://developers.google.com/appengine/docs/adminconsole/datastoreadmin),
which I've used for the manual backups. There's even an easy way to
[run them on a schedule](https://developers.google.com/appengine/articles/scheduled_backups).
Definitely a TODO.
The datastore is automatically backed up by a
[cron job](https://developers.google.com/appengine/articles/scheduled_backups)
that runs
[Datastore Admin backup](https://developers.google.com/appengine/docs/adminconsole/datastoreadmin#backup_and_restore)
and stores the results in
[Cloud Storage](https://developers.google.com/storage/docs/), in the
[brid-gy.appspot.com bucket](https://console.developers.google.com/project/apps~brid-gy/storage/brid-gy.appspot.com/).
It backs up all entities weekly, and all entities except `Response` and
`SyndicatedPost` daily, since they make up 92% of all entities by size and
they aren't as critical to keep.

We use this command to set a
[Cloud Storage lifecycle policy](https://developers.google.com/storage/docs/lifecycle)
on that bucket that deletes all files over 30 days old:

```
gsutil lifecycle set cloud_storage_lifecycle.json gs://brid-gy.appspot.com
```


Here are
Expand Down
9 changes: 9 additions & 0 deletions cloud_storage_lifecycle.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@
{
"rule":
[
{
"action": {"type": "Delete"},
"condition": {"age": 30}
}
]
}

0 comments on commit 7747a8d

Please sign in to comment.