Migrating existing repo (on ssh-backed borg server) to S3-compatible storage with borg-2.x

[amarendra]

• Can existing ssh-backed borg 1.x repo be migrated to borg 2.x (with borgstore) S3-compatible repo with current repo data/history when this change happens?

I use borg with Vorta (to borgbase) so I suppose it will take some time anyway. As, first borg releases have to reach stable borg 2.x and then eventually Vorta shall need to pick that up and support it.

• Is there an easy way one can move such a repo right now? Is there any borg GUI that supports using borg 2.x?

I am just trying to plan/know when the move happens what all I shall have to do and whether I can change any config right now that makes the migration easier or possible later.

[stanleys12]

Yes, history can be carried over. Borg 2 cannot open a borg 1.x repo in place, but it has a command for exactly this: you create a new borg 2 repo (which can be on S3) and run borg transfer --from-borg1 to copy the archives from the old repo into it.

The documented steps are in the borg transfer help, section "Data migration / upgrade from Borg 1.x" (

borg/src/borg/archiver/transfer_cmd.py

Lines 351 to 365 in b84a5bf

	Data migration / upgrade from Borg 1.x
	++++++++++++++++++++++++++++++++++++++
	To migrate your Borg 1.x archives into a related, new Borg 2 repository, usage is quite similar
	to the above, but you need the ``--from-borg1`` option::
	borg --repo=DST_REPO repo-create --encryption=DST_ENC --other-repo=SRC_REPO --from-borg1
	# to continue using lz4 compression as you did in SRC_REPO:
	borg --repo=DST_REPO transfer --other-repo=SRC_REPO --from-borg1 \\
	--compression=lz4 --recompress=never
	# alternatively, to recompress everything to zstd,3:
	borg --repo=DST_REPO transfer --other-repo=SRC_REPO --from-borg1 \\
	--compression=zstd,3 --recompress=always

):

borg --repo=DST_REPO repo-create --encryption=DST_ENC --other-repo=SRC_REPO --from-borg1
borg --repo=DST_REPO transfer --other-repo=SRC_REPO --from-borg1 --compression=lz4 --recompress=never

Here DST_REPO is the new borg 2 repo and SRC_REPO is the old borg 1.x repo. --other-repo on repo-create makes the new repo reuse the key material of the old one, so chunking and chunk ids stay compatible and deduplication keeps working. You can also recompress during the transfer, for example --compression=zstd,3 --recompress=always.

Transfer is incremental. Archives already present in the destination are skipped, and only missing chunks are copied. An interrupted run is safe to repeat (

borg/src/borg/archiver/transfer_cmd.py

Lines 304 to 335 in b84a5bf

	Incremental transfer
	++++++++++++++++++++
	Transfer only copies what is missing in the DST_REPO, so you can run the same command
	repeatedly (e.g. from a cron job) to keep DST_REPO up to date - each run will only
	transfer what was added to SRC_REPO since the previous run.
	This works on two levels:
	- archive level: if the DST_REPO already has an archive with the same name and timestamp
	(or the same name and archive ID), that archive is skipped and borg prints
	"archive is already present in destination repo, skipping.".
	- chunk level: for archives that are copied, only the chunks not yet present in the
	DST_REPO are read from the SRC_REPO and stored - deduplication works against
	everything transferred before.
	The archive filter options can be used as usual, so multiple invocations with different
	filters are fine, e.g.::
	borg --repo=DST_REPO transfer --other-repo=SRC_REPO -a 'sh:daily-*'
	borg --repo=DST_REPO transfer --other-repo=SRC_REPO -a 'sh:myserver-*'
	Some things to keep in mind:
	- Interrupting a transfer is safe: an archive only becomes visible in the DST_REPO after
	it was copied completely, so an interrupted archive is simply transferred again by the
	next run. The chunks stored by the interrupted run are reused, so the retry is cheap -
	unless you run ``borg compact`` on the DST_REPO in between, which would remove the
	chunks that are not (yet) referenced by an archive.
	- Transfer only adds archives, it never deletes any. If you want old archives to expire
	in the DST_REPO, run ``borg prune`` (and ``borg compact``) against the DST_REPO
	separately.

). You can also filter with -a to move archives in batches.

borg 2 can talk to S3-compatible storage directly with a URL of the form s3:[profile|access_key_id:secret@][scheme://hostname[:port]]/bucket/path. If your provider has trouble with s3:, the docs suggest trying b2: instead (

borg/docs/usage/general/repository-urls.rst.inc

Lines 64 to 71 in b84a5bf

	**Remote repositories** accessed via S3:
	``(s3|b2):[(profile|(access_key_id:access_key_secret))@][scheme://hostname[:port]]/bucket/path`` — see the boto3 docs for more details about credentials.
	If you are connecting to AWS S3, ``[schema://hostname[:port]]`` is optional, but ``bucket`` and ``path`` are always required.
	`scheme` is usually `https` here, hostname and optional port refer to your S3/B2 server, if that is not Amazon's.
	Note: There is a known issue with some S3-compatible services, e.g., Backblaze B2. If you encounter problems, try using ``b2:`` instead of ``s3:`` in the URL.

).

As for what you can do now, there is nothing you need to change in your borg 1.x repo. Plan for having both repos side by side during the move, because the transfer reads from the old one and writes to the new one. Borg 2 is still a beta, and the docs describe it as a different on-disk format from 1.x, so keep the old repo until you are happy with the new one.

I can't say anything about Vorta or other GUIs supporting borg 2. That is a question for those projects.

(I looked this up in the code with help from an AI assistant; the links are to the lines I checked.)

[amarendra]

Thank you. This is encouraging.

I may try to move to borg2 sooner (via a script and a cron job without waiting for any GUI).

1. Transfer SRC_REPO archives to DST_REPO
2. Existing backup frequency and existing source set only to DST_REPO
3. And keep existing SRC_REPO around (and as it is/frozen) until borg2 becomes stable and mainstream

Or maybe still backup to existing SRC_REPO, but only:

• a reduced source set (excluding some less important files/folders from it)
• and much less frequent

Next I am going to look for experiences from people who might already started using borg2 other than just for testing :-)

[ThomasWaldmann]

borg2 is beta still, not for production.

you can use it for additional backups, but make sure you also have backups using borg 1.x.