Backup/Restore commands

[rahul286]

Moved to https://github.com/EasyEngine/easyengine/issues/1045

backup/restore commands (using duplicity)

1. local backup
2. s3 backup
3. remote ftp backup
4. remote rsync backup

Also.. crashplan (optional)

---

From @MiteshShah in issue #102

1. Backup

• ee site backup

If backup path is not specified use default backup path

Backup Structure

/path/to/backup/location/sitename/YYYY-MM-DD-HH-MM-SS.tar.gz

Backup Includes

1. Nginx Conf
2. Database
3. Webroot
4. Logs Files

2. Restore

• ee site restore

If no path is specified latest will be used.
Also ask if we should proceed with backup or without backup.
We will ask user to confirm if destination nginx-conf/webroot/database/logs-files already exist.

--force - do not ask during restore

[Mermouy]

Just discovered bsync, witch may be a nice and easy solution...
"It uses rsync for file transfers, GNU find to generate filelist snapshots, and ssh for remote transfers."
Have a look here:
https://github.com/dooblem/bsync
For a "real time" sync this may be interresting:
https://github.com/deliciousrobots/inotify-sync/blob/master/inotify-sync.py
Which could be used with any fuse-mounted dir (we can find fuse lib for almost any services (s3, hubic , ubuntuone, ssh, even iso...
The "easy" thing is that it take care about conflicting files itself what do you think?

[andremacola]

Dropbox would be nice.

[jamesgiroux]

Will there be some form of GUI or plugin for this on the server or will it all be command line driven? Love the idea of integrating crashplan. What about one-click restore? I know, I'm asking a lot. :-)

[rahul286]

@Transforme crashplan has its own UI.

For duplicity based options, we will provide some abstract commands for now.

[rahul286]

What a coincidence - Amazon S3 reduced backup prices today - http://aws.typepad.com/aws/2014/03/aws-price-reduction-42-ec2-s3-rds-elasticache-and-elastic-mapreduce.html

CrashPlan has downside of using Java and eating a lot of CPU.

Duplicity + S3 is a very good combo but was costly affair because of S3 cost.

[sebstam]

+1 for duplicity with s3! we managed to get this working ourselves on a vps making use of duply.

[EranSch]

Has anyone explored the implications of using Git for backups rather than traditional file copying methods? Each site could have a bare repo adjacent to it which could have some basic hooks and scripts to control it.

This would provide full transparency into changes between backups and allow for super-cool, easy, restoration. Not to mention it uses more Git... which is always a win :)

[QROkes]

Now I'm using duplicity to backing up my EE server and its a great tool.
This are the steps to do it:

sudo apt-get install python-boto python-gobject-2 python-pycryptopp librsync-dev python-deb python-paramiko
sudo apt-get install duplicity duply

#Create back up profile
duply create

#generate new gpg key pairs
openssl rand -base64 20
gpg --gen-key

#Edit and configure conf file
nano /home/dryweb/.duply//conf
GPG_PW='Qx+9csdfsdfsdfsSDFSDFJBJKSDFxNyxe8zM0='
GPG_KEY='E5864DB1'

TARGET='s3://s3-ap-southeast-1.amazonaws.com/'
TARGET_USER=''
TARGET_PASS='<your AWS secret key'

base directory to backup

SOURCE='/var/www/qrokes.com/htdocs/wp-content'
MAX_AGE=3M

sudo duply backup

crontab -e
MAILTO=mail@mydomain.com
0 2 * * 7 env HOME=/home/ duply backup_verify_purge --force

#restore backups
sudo duply restore /restored_files

[cm896]

I used to backup the server with the headless dropbox linux client, but the web interface once you hit a large number of objects is useless.
I have long since switched to using a headless copy.com linux client to move server backups offsite. The web interface is superiorIMO. I usually setup a couple of folders and cron jobs to make these file changes "noticeable" to the watched copy.com folder for synching.

DB job:
drops all databases as tar.gz files on a frequency depending on the site's traffic/updates, hourly in the most extreme, and usually at least once daily, then rotate these db backups based on age - usually keeping a full month+. (this quickly grows storage, 1GB db x 24 hourly backups x 31 days = 744GB of database backups)

Webroot job:
gz tar ball the whole web root usually daily - depends on asset update frequency - stored at least 31 days of changes. (This plus the db can recreate the whole site - not the server config though) Depending on the webroot assets, this can grow storage needs as well - see above when multiplied by frequency)

Live job:
rsync is used to make a local copy of the webroot. I setup a local folder that is rsynch'ed from the webroot with the copy.com account so that the changes are not bi-directional. (This is intentional but can be the inverse). There is a storage cost to this since it creates a local mirror of the webroot, but any time a file is changed in the web root, it's rysnc'ed at it's interval - hourly in most cases - and that change is then captured offsite almost immediately via copy.com's headless client. There is another advantage to the mirror'ed local webroot. When the client calls and mentions that they flushed an asset unintentionally, if it falls within that rsync frequency, it's super easy to restore. If not, I have to pull it from that last webroot tar ball. If the file is too fresh (uploaded after the last rsync), as in not having been captured in a tar ball or an hourly rsync, then the client has lost the file.

I would love to add another job, that would effectively capture all the server setup/configuration. I haven't started to write that bash script yet, but if there was a method to recreate the ee install on a new box in a disaster recovery setting, that would be awesome. I'd love to include all the ee, nginx, php, etc... .conf and configs in a manner which could be restored through ee to bring back the site/server.

All that being said, the free copy.com account is 15gb and is enough for most smaller, less frequently updated needy clients to store a month of revision off site. My data intensive clients have the 1TB business account and it's bursting at the seams. We play with adjusting frequency for the full webroot when combined assets are over 5gb, and rely more on the rsync frequency to capture changes between those gaps. Also as a precaution, the local rsync'ed copy.com folder is setup to not delete files pulled from the webroot, as a just in case measure.

I'd be super interested in hearing how others are accomplishing their disaster recovery tasks in an ee environment.

[ddelaey]

@rahul286
What's the status backup/restore commands?
What are the best ways to backup Wordpress and EasyEngine and the server?

Thanks, David

[mewcrazy]

+1

A function like ee site backup would be so awesome, simple and compatible with cronjobs :D

[MattyLB]

Only simple if you've already defined your backup
repository's authentication tokens and backup rules in a web interface. I
believe this is easier done in a separate web based ee management console
and set to automatically run on a cron schedule. Once this is set
initially, it wouldn't be a bad idea to have an ee server backup -- or ee
site backup -- command to initiate one on-demand tho. … just my opinion,
and it keeps the bloat out unless we opt for the premium features of it.

On Sunday, May 17, 2015, Dennis Bitsch notifications@github.com wrote:

+1

A function like ee site backup would be so awesome, simple and compatible
with cronjobs :D

—
Reply to this email directly or view it on GitHub
#145 (comment).

[ddelaey]

@rahul286
What's the status backup/restore (duplicate) commands?
Do you have recommendations to backup Wordpress and EasyEngine (and the server)?

Thanks, David

[ddelaey]

@rahul286
What's the status backup/restore (duplicate) commands?
Any recommendations to backup Wordpress and EasyEngine?

Thanks, David

[gagomap]

You can buy a small vps for backup. I use Duplicity to backup everyday.
Or if your vps support snapshot, why don't you use it ?

[MattyLB]

ddelaey - on Amazon AWS it's called snapshot. There should be other free
options depending on your VPS or server, but this should not be an
EasyEngine task in my opinion because there are so many different scenarios
and external storage methods that it can get out of hand. Look into your
server providers options.

On Thursday, December 3, 2015, ddelaey notifications@github.com wrote:

@rahul286 https://github.com/rahul286
What's the status backup/restore (duplicate) commands?
Any recommendations to backup Wordpress and EasyEngine?

Thanks, David

—
Reply to this email directly or view it on GitHub
#145 (comment)
.

[oviliz]

In my opinion, a backup function would be great.

[MattyLB]

The question becomes: "backup what and to where?" If it's a simple backup
of our customized EE configs to a user folder or external ftp source, then
that is great - and I would +1 that. Anything more involved, such as full
server backup to either S3, Google Drive, FTP or Dropbox and I believe
that's putting to much pressure on the EE team to maintain and misusing
their valuable resources. If course, this is my opinion but I believe it's
a valid one considering they offer EE to us for FREE and it's supposed to
be an "easy" tool that works well and is solid for production environments
(for FREE).

On Friday, December 4, 2015, Cristian O. Balan notifications@github.com
wrote:

In my opinion, a backup function would be great.

—
Reply to this email directly or view it on GitHub
#145 (comment)
.

[jamesgiroux]

I think there needs to be some form of backup functionality integrated if EE is going to work in production environments. Not being able to backup the entire server somewhere is kind of scary. Most vps providers have some way of doing this. Digital Ocean offers weekly backups for a fee as an example but redundancies are important do some other method whether baked in or officially supported would make a lot of sense. I agree though that EE settings at a minimum would be very helpful.

[w33zy]

While they are at it, why not add a migrate function? Why not add support for Postgre? Why not add support for NodeJS?

There has to be a point where all these new feature request goes beyond the scope of EE's intended purpose.

[MattyLB]

@w33zy - my point exactly. A demarcation line has to be drawn with EE's
core mission with the option of paid (supported) integrated add-ons for
those who feel EasyEngine needs to become SwissArmyKnifeEngine.

On Saturday, December 5, 2015, w33zy notifications@github.com wrote:

While they are at it, why not add a migrate function? Why not add support
for Postgre? Why not add support for NodeJS?

There has to be a point where all these new feature request goes beyond
the scope of EE's intended purpose.

—
Reply to this email directly or view it on GitHub
#145 (comment)
.

[christoferw]

@Swingline0 did you see this? https://github.com/bup/bup

[rahul286]

Moved to https://github.com/EasyEngine/easyengine/issues/1045