1.1.0

A schedule per location, pinned restore points with notes, disaster recovery without the recovery kit, and much faster reading on network and cloud disks. After the update every location keeps the backup time it had.

Added

• A schedule per location: off, once a day, or every N hours from a given time, on the page under Backups or with occ backup:target:schedule.
• Pin a restore point so the retention policy leaves it alone, and give it a note you can change later (occ backup:pin, occ backup:note, or the restore page). Both are stored encrypted on the location, so a new server sees them.
• Disaster recovery without the recovery kit: on a new server, occ backup:target:add takes the key of the backups from the location with one of the old server's passphrases. The deletion delay of the Bin is now stored on the location too.
• A passphrase change that cannot reach an offline location reaches it at that location's next backup.
• occ backup:target:add --repair-config writes a missing config file on a location again from the recovery kit.
• A key made by mistake on a server you are restoring can make way for the key of your backups, as long as it was never used (--replace-unused-key). A used key is never replaced, and neither is a key from before 1.1.0.
• All folders get their original modification time back on a restore, from the first backup made with this version.
• Heavy background jobs (Memories, Recognize, Preview Generator, full text search, antivirus scan) pause while a backup runs, at most 6 hours in a row. On by default; turn it off under Backups or with occ backup:pause-jobs off.

Changed

• Reading files for a backup is much faster on disks where every request waits, such as network and cloud volumes: on a test server 27 MiB/s instead of 5 MiB/s.
• Locations are backed up one after the other; a waiting backup shows as queued.
• Choosing a time no longer starts a backup right away when that time has already passed today.
• Administrators are told when a schedule is turned off, instead of getting "no recent backup" notifications for a location without a schedule.
• A file or folder left out because of its name gets a notification of its own that asks to rename it.
• Long labels, notes and errors wrap in the list of backups instead of making it scroll sideways.
• Smaller fixes, see CHANGELOG.md.

Tested on Nextcloud 32 to 35 with MariaDB, PostgreSQL and SQLite, with an upgrade from 1.0.3, and with 14 disaster-recovery scenarios with and without the recovery kit.

Note: an older version of NG Backup can still read a location written by 1.1.0, but drops pins and notes when it writes to it.

1.0.3

Empty folders now come back, and a full disk during a restore no longer looks like a damaged backup. Nothing changes in your settings.

Fixed

• Empty folders are backed up and restored (#27), in occ backup:restore, occ backup:restore:full and the restore page. They come back from the first backup made with this version; older restore points do not contain them.
• A full disk, or other trouble writing to the restore directory, stops a restore with a clear message, also with --salvage.
• A file or folder whose name a restore point cannot hold (a backslash, a control character, not UTF-8) is left out and reported as unreadable. Before, such a name could make a whole restore point unreadable, or make the backup fail.
• After a restore that was killed and ran again, a backup no longer has to wait up to two hours for its locks.
• A database restore that fails no longer leaves tables behind that could stop an app's installation later.
• The Bin tab shows how far it is while it reads the removed restore points, and shows when it cannot read a location.
• Smaller fixes, see CHANGELOG.md.

Tested with three small full disaster recoveries on Nextcloud 34: as planned, with the restore killed halfway, and with a disk that runs full.

Known limitation: folders with files in them show the time of the restore, not their original modification time. The files keep theirs. On the roadmap.

1.0.2

Faster and more robust full restores, and a way to get back what can be read from a damaged backup. Nothing changes in your backups or settings.

Added

• --salvage for occ backup:restore and occ backup:restore:full: files that cannot be read are left out and listed, everything else is restored (exit code 2).
• A warning on the admin page and in occ backup:status when background jobs do not run on a system cron. Without one, a backup started with "Back up now" moves on while the NG Backup page stays open (#13).
• The trash of restore points has a tab of its own, "Bin".

Changed

• Full restores from S3 read ahead, and over SMB in larger reads. Reading the index over SMB takes fewer round trips.
• A full restore that stopped goes on where it was (--force): files already in place are left.
• A backup in its database phase shows how far it is.

Fixed

• A full restore over SMB could hang at 100% CPU when the connection dropped.
• A restore that stops because a file is gone from the location now points at --salvage.
• Smaller fixes, see CHANGELOG.md.

Tested with a full disaster recovery of about 1 TB over SMB, see the README.

Known issue: empty folders are not backed up or restored; to be fixed in 1.0.3 (#27).

1.0.1

Fixes for what was reported after 1.0.0 and small improvements. Nothing changes in your backups or settings.

Fixed

• SFTP locations with a key in OpenSSH format work again on Nextcloud 35 (#24).
• A full restore onto SQLite waits for another writer instead of stopping with "database is locked".
• occ backup:restore --to-directory restores a single file; a stopped restore no longer leaves half-written temporary files behind.
• The app store no longer shows the notes of every release candidate under 1.0.0.

Added and changed

• occ backup:user:restore takes a user id, and occ backup:user:list shows the exports on a location (#22).
• An optional label for "Back up now", an explanation of "removed by retention policy" in the trash, and a Docs tab on the admin page.
• occ shows times in the time zone of the schedule and names it.

See CHANGELOG.md for the details.

If you like NG Backup, don't forget to give the app a star in the app store and this repository a star on GitHub!

1.0.0

First stable release of NG Backup. NG Backup makes encrypted, incremental backups of your whole Nextcloud (files, database and configuration) to S3, SFTP, WebDAV, SMB and other storage, and restores them from the web interface or with occ.

Disaster recovery has been tested on real installations; see "Tested disaster recoveries" in the README. Read "Known limitations" before you rely on it, and test a restore.

The code is the same as 1.0.0-rc.6 (app version 0.9.12): updating from it changes nothing in your backups or settings.

See CHANGELOG.md for everything since the first release candidate.

1.0.0-rc.6

Sixth release candidate for 1.0.0; the app version is 0.9.12. Fixes from two disaster recovery
simulations and a user report. If no problems are reported for about a week, 1.0.0 will be this
same code.

Fixed

• The daily backup time was taken as UTC (Nextcloud runs PHP in UTC): 03:00 ran at 05:00 in
  Berlin (#19). The time is now saved with the administrator's time zone and shown with it. A time
  saved before is converted once to the zone of the server or an administrator, so it shows as it
  effectively ran (02:30 UTC becomes 04:30 Europe/Berlin in summer) and nothing moves.
• A full restore over SMB stopped after 17 hours with "Decryption failed": a read over a long
  line handed out wrong bytes (the backup itself was fine, the same files restored identically
  afterwards). A blob that does not decrypt as first read, of a file or of the database, is now
  read once more on a request of its own, and only a second failure stops the restore.
• A full restore onto MySQL/MariaDB stopped with "Incorrect string value" when it created a table
  (of an app that is not installed here) whose rows hold emoji or other 4-byte characters: the
  table got 3-byte utf8. Tables the restore creates now get this installation's character set and
  collation (utf8mb4). Found in a disaster recovery simulation onto MariaDB.

Changed

• The admin page uses more of the screen (sections up to 1400 pixels instead of 900), so the
  list of backup runs fits without scrolling sideways.
• README: a roadmap, the tested disaster recoveries, measurements for OVH S3, the Storage Box over
  SFTP and SMB, how to build the SMB extension when PECL is down, and what works without shell
  access. The guide lists shell access among what you need.
• The full-restore integration test checks a 4-byte character in a table the restore creates, and
  retries app installs from the app store.

1.0.0-rc.5

Fifth release candidate for 1.0.0; the app version is 0.9.11. A full restore of 1 TB over SFTP
stopped after 13 hours with "Blob not in index"; the backup itself was fine.

Changed

• README and app store description: NG Backup needs background jobs on system cron, with how to
  set that up (issues #13, #17).

Fixed

• A full restore no longer stops with "Blob not in index" after a pack could not be read for a
  moment. Reading the index again then got an empty listing (from a connection the location had
  dropped) and emptied the index. Reading the index again now only adds, an empty listing of a
  location that has an index is an error instead of an empty index, and a pack that could not be
  read is read once more before the restore gives up.
• After a full restore of a MySQL/MariaDB or SQLite backup onto PostgreSQL, new files, users or
  activities could not be added ("duplicate key", "File entry could not be inserted"): such
  backups record no ID counters, and they stayed at 1. Every counter of a restored table now goes
  after its data. Found in a disaster recovery simulation of 138 GB from MariaDB to PostgreSQL,
  where all 15,772 files and every count were identical.
• A full restore now runs in maintenance mode: a background job, a page visit or a user writing
  into the instance meanwhile could collide with restored rows ("duplicate key") or be lost. It
  is switched off when the restore completes; after an error it stays on, and the message gives
  the one line to switch it off and run the restore again. Maintenance mode that was on before
  the restore stays on.
• Adding a location whose repository config could not be read, while backup data is there,
  no longer creates a new repository over it: it stops with an explanation. A snapshot that
  cannot be taken over when a location is added is now logged instead of left out silently.
• An SFTP connection that a Hetzner Storage Box dropped without phpseclib noticing is now
  recognised and replaced: the check asks the server and counts its answer.
• The extra connection a full restore reads ahead on is replaced when it breaks (up to 20 times),
  instead of reading the rest of the restore the slow way.

1.0.0-rc.4

Fourth release candidate for 1.0.0; the app version is 0.9.10. 1.0.0-rc.3 (0.9.9) was not
released: its fixes are in here.

Changed

• A full restore over SFTP reads the packs ahead on a connection of its own, 32 requests in
  flight, into temporary files in the restore folder. From a Hetzner Storage Box to a server in
  Canada (122 ms) packs now come in at about 14 MiB/s instead of 2.8. More connections can be set
  with restore_sftp_connections (README). A part that cannot be read this way is read as
  before. Restores from other locations work as before.
• App store description: the line about Ensembia.

Fixed

• After a full restore stopped halfway, running it again with --force was refused when the
  database was already restored: Nextcloud then first wants occ upgrade. The guide and the
  error message now say so.
• A full restore of a backup made before 1.0.0-rc.2 with --skip-missing-tables failed on
  PostgreSQL with "relation ... does not exist": the left-out tables were still looked up to set
  their ID counters. It now skips them. A new test restores a backup made by 1.0.0-rc.1 on
  PostgreSQL and MariaDB.
• The app store showed old screenshots: it keeps a copy per URL, and the URLs always pointed at
  main. They now point at the release's tag.

1.0.0-rc.2

Second release candidate for 1.0.0; the app version is 0.9.8. Fixes from a full disaster recovery
test of a production instance (about 1 TB, Nextcloud All-in-One onto a plain Nextcloud with
PostgreSQL).

Added

• docs/DISASTER-RECOVERY.md: step-by-step guide for when your server is gone.
• occ backup:restore:full --install-apps installs (from the app store) or enables the apps the
  old server had enabled, before the restore starts.

Changed

• Backups record the structure of every database table. A full restore creates tables that do not
  exist on the new instance, for example those left behind by an app that was removed long ago or
  is switched off, instead of stopping at the first one.
• A full restore no longer refuses when apps are missing: it restores their data and lists the
  apps to install afterwards. nextcloud-aio is never required, so a backup of Nextcloud
  All-in-One can be restored onto a plain Nextcloud.
• A full restore onto an installation with a different table prefix (dbtableprefix) is refused
  before it starts, with what to change, instead of restoring into tables nobody uses.
• All problems are listed at once, with the app each missing table probably belongs to. For
  backups made before this version, --skip-missing-tables leaves out tables whose structure is
  unknown.
• A fuller app store description, ready for 1.0.0, and a note on paid restore help by Ensembia.
• README: switching the server to TCP BBR made backups over a long distance about 3 times faster.

Fixed

• A full restore failed on foreign keys (for example those of the Mail app), because tables are
  restored in name order. Foreign keys are now checked after all rows are loaded, before the
  restore commits.
• occ backup:target:add without a key pointed to backup:key:init, which makes a new key that
  cannot read existing backups. It now points to backup:key:import for a restore, and says that
  connecting to a location with backups can take a while. While it reads the index of a location
  with backups, it shows how far it is and about how long it still takes.
• A restore stopped with "Not found: packs/..." when the location was cleaned up while it ran,
  for example by the weekly cleanup of the original server during a recovery test on a second
  server: blobs that were moved into new packs were still looked for in the old ones. A restore
  now reloads the index and reads them from where they are now.
• Removing a location left its cached index in the database (tens of thousands of rows for a large
  backup). It is removed with the location now, and once for locations that are already gone.

1.0.0-rc.1

Release candidate for 1.0.0; the app version is 0.9.7. Only bug fixes until 1.0.0. Tested on a
production instance (about 1 TB, 150,000 files): full and incremental backups and a verify on
SFTP and SMB (Hetzner Storage Box) and S3 (OVH), a deep verify on SFTP, and partial restores from
SMB and S3 checked against checksums.

Added

• The admin page asks for feedback, with a link to a short "Experience report" form on GitHub:
  which location, setup and size you used, and how it went.

Changed

• New passphrases must be at least 16 characters (was 12). The location stores the key
  encrypted with each passphrase, so the passphrase is what really protects the backups.
  Existing passphrases keep working. The README and the how-to now say so, and why the
  recovery kit must still be kept off the server.
• Dates and times in the web interface follow the locale in the user's Nextcloud settings instead
  of the browser's language, so a 24-hour locale shows 16:00 rather than 4:00 PM.
• The list of backup runs (web interface and occ backup:status) shows the runs of the last 7
  days, those still running, and at least the 5 newest; a failed run stays visible for that week.
  occ backup:status --all and "Show older runs" in the web interface list all of them.
• occ backup:status and occ backup:verify show durations of a day or more with the days
  ("1d 12:10:13" instead of "12:10:13").

Fixed

• With system cron, a running backup moved forward only every other cron run: Nextcloud
  skipped the background job when cron came a few seconds before its 5-minute interval was
  over, so a backup was idle 6 of every 10 minutes. The job now runs at every cron run.
• Photos, video and other compressed data are no longer deflated only to throw the result away:
  a quick test on three small slices of each block skips it. Compressing was the slowest part of
  a backup (about 24 MiB/s).
• Checking a backup on an S3 location (occ backup:verify) took hours on a large repository:
  the index files were downloaded one at a time, at about two a second. They are now downloaded
  16 at a time: on a production instance (27,585 index files, 1 TB) the check went from about
  4 hours to 3.5 minutes.
• Writing the snapshot at the end of a long backup to S3 took a quarter of an hour without any
  progress shown: it read back the file list of every step one at a time. These reads are now
  done 16 at a time on S3 (96 reads: 82 seconds before, 2 now), blobs that follow each other in
  a pack are read at once on every location, the snapshot is written in a step of its own, and
  the run shows "commit" while it does so.