Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Make it easy to backup and restore e-mail, contacts, settings and metadata #811

Open
smtalk opened this issue Aug 8, 2014 · 8 comments
Open

Comments

@smtalk
Copy link

@smtalk smtalk commented Aug 8, 2014

People's e-mail contains large amounts of valuable personal information. Considering that part of Mailpile's mission is to make it easy for people to self-host and remove their private data from the corporate clouds, it is vitally important that we take steps to ensure this valuable data does not get lost due hardware malfunction, software bugs or human error. Mailpile needs to help people make secure backups of their mail, whether to removable physical media or to remote online storage.

(Updated for roadmap, original issue text is below)

Hello,

It would be very nice to have a command-line tool, which dumps everything account-related (address book, identities, contacts etc.) to an XML/JSON file for an easy restore, so that server-to-server account transfers could be easily done.

Thank you!

@BjarniRunar
Copy link
Member

@BjarniRunar BjarniRunar commented Aug 9, 2014

Making backups/restores/moves easy is importent.

However, Mailpile's data is all in normal files in the file system, so moving from server-to-server is just a matter of copying the files over. Backups are similarly simple.

In the case where the user lets Mailpile encrypt its on-disk data, we'll even be able to upload it to untrusted cloud sources more or less unmodified (and may be able to write code that does this automatically) - if we get the UI around this right, then all the user needs in order to restore is an offline copy of the symmetric key, which is small enough that it could be printed on a piece of paper, stored in a vault with other important dead-tree talismans and typed back in by hand on restore.

@BjarniRunar BjarniRunar changed the title A simple backup/restore of a user (with API/CLI) Make it easy to backup and restore e-mail, contacts, settings and metadata Nov 28, 2014
@BjarniRunar BjarniRunar added this to the Post 1.0 Roadmap milestone Nov 28, 2014
@BjarniRunar
Copy link
Member

@BjarniRunar BjarniRunar commented Oct 25, 2018

An update on the status of this, and the strategy currently being implemented:

Status: There is now a backup tool on the Mailpile settings page which generates a .zip the user can download and save manually. The backup contains the user's settings, tags, contacts and keys. It is possible to upload this .zip during the setup of a new Mailpile to import the data. Only the e-mail data (and metadata) is missing from this archive. The contents are encrypted and protected by the same password/passphrase as Mailpile itself.

Strategy: Javascript in the web UI will automatically store this backup .zip in Localstorage on the user's browser - once users are encouraged and assisted with accessing Mailpile from mobile devices, we will automatically have guaranteed a backup that lives on separate hardware from Mailpile itself. Some Javascript magic will be needed to help the user extract the backup for use with their new Mailpile.

The second phase of the backup strategy will be to upload e-mails and metadata (encrypted) to one of the user's IMAP accounts. We might also upload the .zip archive itself, but unless we know our users have generated very strong passphrases (how can we?), that isn't really safe.

@gellenburg
Copy link

@gellenburg gellenburg commented Feb 13, 2019

Hi Bjarni. I'm trying to follow the strategy listed here to backup my mail to mbox format: https://github.com/mailpile/Mailpile/wiki/Troubleshooting#backing-up-and-restoring-downloaded-mail

But doing the search is only returning 120 maximum messages at a time.

I have over 9,000 emails that I'd like to backup / export out of Mailpile.

I tried pointing Thunderbird to MailPiles local maildirs but that was a disaster. :-)

Any hints or tips on how I can export/ backup all the mail out of Mailpile in case my laptop dies? :-)

@BjarniRunar
Copy link
Member

@BjarniRunar BjarniRunar commented Feb 14, 2019

If you're just concerned with the laptop dying, why don't you just copy the Mailpile data (and GnuPG) folder in its entirety? If you think the search or export commands are broken, please file a separate issue to discuss that, I'd rather not derail this issue. The only comment I have there, is to point out that even if a search command only displays the first page of results, the export ... all command will include all results (page 2, 3, etc.). If your search returns too few results, change the search terms?

@gellenburg
Copy link

@gellenburg gellenburg commented Feb 14, 2019

I'll try it again. I followed the exact search query as specified on that web page.

@neurokrish
Copy link

@neurokrish neurokrish commented Feb 17, 2019

An update on the status of this, and the strategy currently being implemented:

Status: There is now a backup tool on the Mailpile settings page which generates a .zip the user can download and save manually. The backup contains the user's settings, tags, contacts and keys. It is possible to upload this .zip during the setup of a new Mailpile to import the data. Only the e-mail data (and metadata) is missing from this archive. The contents are encrypted and protected by the same password/passphrase as Mailpile itself.

Strategy: Javascript in the web UI will automatically store this backup .zip in Localstorage on the user's browser - once users are encouraged and assisted with accessing Mailpile from mobile devices, we will automatically have guaranteed a backup that lives on separate hardware from Mailpile itself. Some Javascript magic will be needed to help the user extract the backup for use with their new Mailpile.

The second phase of the backup strategy will be to upload e-mails and metadata (encrypted) to one of the user's IMAP accounts. We might also upload the .zip archive itself, but unless we know our users have generated very strong passphrases (how can we?), that isn't really safe.

Have you looked at rclone for saving backups to different destinations? This may be useful for many of them who already have cloud storage elsewhere. I think the program allows one to generate a strong random passphrase but unless you force the user to use this, may be you can't ensure that a strong passphrase is used (more here - https://rclone.org/crypt/)?

@BjarniRunar
Copy link
Member

@BjarniRunar BjarniRunar commented Feb 18, 2019

@neurokrish That's a good idea - I have not looked into rclone myself. My gut reaction is that the "core team" should focus on getting IMAP-based backups to work as planned, but hopefully someone will contribute something like rclone support as a plugin down the line. And the core team should take care to design the backup system so it's amenable to such things.

That's largely academic at this point though, since I'm not able to work much on Mailpile these days, for various reasons. So if someone volunteered to just implement rclone and skip my idea, that would be way better than nothing! I'd cheer them on. 😁

@neurokrish
Copy link

@neurokrish neurokrish commented Feb 19, 2019

I understand!

I want to also put this out there, just in case someone is interested. There are two programs, restic and rclone. The former is more focused on backups and the latter is more focused on providing a way to sync to various cloud destinations. In fact, both these authors have come together and integrated their work to use rclone as a backend to restic (https://restic.net/blog/2018-04-01/rclone-backend).

Perhaps someone can come forward to develop a plugin for Mailpile..

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
4 participants