Borg check freezes the low-powered server

[rolandu]

Have you checked borgbackup docs, FAQ, and open GitHub issues?

Yes

Is this a BUG / ISSUE report or a QUESTION?

Question

System information. For client/server mode post info for both machines.

Your borg version (borg -V).

• Server: 1.1.16
• Client: 1.2.8

Operating system (distribution) and version.

• Server: Debian 11, Linux 5.10.0-32-amd64
• Client: Mint 22, Linux 6.8.0

Hardware / network configuration, and filesystems used.

• Server: Mini PC, Intel Celeron N3000, 8 GB RAM, HDD, zfs
• Client: Laptop, Intel Core 7 Ultra, SSD, 64GB RAM, ext4

Client and Server are on the same local network / wifi.

To summarize: This is a high-end laptop trying to backup to a low-end "server" (i.e. NUC-clone mini-pc).

How much data is handled by borg?

~600 GB

Full borg commandline that lead to the problem (leave away excludes and passwords)

borgmatic check --force

borgmatic config:

checks:
    - name: repository

this runs:

borg check --repository-only --debug --show-rc --progress

Describe the problem you're observing.

On the client machine I can monitor the progress of the check, which is about 10% per hour.

Meanwhile I am observing lots of errors on the server machine relating to other processes being out of memory, returning error 500, etc. (I am running some services on the same server).

Even after cancelling the check client-side, the server is still unresponsive. Half an hour later, I was able to log in via ssh and kill the borg process on the server, which was hogging 20% of CPU and lots of RAM.

Questions:

• Could it be that my server is simply too underpowered to run the checks properly?
• Is there a way to limit the system resources the server's borg process uses, e.g. limit to 2GB RAM?
• Could running borg via sshfs instead of ssh reduce this problem, as I am not using the server's borg process?

Can you reproduce the problem? If so, describe how. If not, describe troubleshooting steps you took before opening the issue.

Yes, happens every time I try to perform a check.

Include any warning/errors/backtraces from the system logs

N/A; can provide CRUD benchmark if it's considered relevant.

[ThomasWaldmann]

borg serve needs some RAM because it loads the repo index into memory (and no, there is no way to "limit" that except using repositories with less chunks in them and thus having a smaller repo index). There is a formula in the docs to estimate the RAM usage.

but 8gb should usually be enough to run borg, so there might be some other issues:

• is zfs using lots of memory? you can limit the l2 arc memory usage, the default might not be the best if you don't have lots of RAM.
• other services taking a lot of these 8GB? maybe you can use some time slot at night or early morning to shut these down while you run borg operations, like backups, checks.
• if your RAM is upgradable: RAM is cheap. :-)
• if nothing else helps: add more swap (and maybe avoid swapping on zfs). it will be very slow if you use more memory than ram, but it might help with processes not getting killed.

You could also upgrade the borg running on the server (see debian backports), but I guess it won't help with the RAM usage, but give you some other improvements.

[ThomasWaldmann]

Running via sshfs is usually slower (more network traffic) and might have its own issues (like many network fs have), but yes, it would shift the borg memory usage from the server to the client.

I would first try the other options, see previous post.

[ThomasWaldmann]

@rolandu did you get it working better?

[rolandu]

I was not able to try it yet due to other commitments, I will report back soon.

[rolandu]

I have finally performed a test with sshfs. Unfortunately the result is the same: the server was completely overstressed by the load and did not function properly.

For comparison: I have two backup destinations, one is the low-powered home server and the other a cloud storage provider. The repositories are similar in size. The check on the cloud server takes four minutes (!) and no issues. On the low powered server, I turned it on in the evening, returned in the morning and came back to it 50% done and thousands of error messages from the other services due to low RAM/processor capacity / timeouts.

Discussing with ChatGPT, it seems that the high IO load of the check operation through zfs might be too much for my little server. I will probably either have to upgrade it or switch to ext4 for the drives.