Inconsistent permissions and smartremove.lck on server

[protist]

Describe the problem, feature or ask a question:

I backup via ssh to a server, and can see some odd permissions on some of the directories. Mostly the directories have drwxr-xr-x, but some are drwx------. From the dates, they possibly look like some snapshots that should have been smart removed? I also have a file named smartremove.lck that has been present for the last 24+ hours.

On the server:

$ ls -lF /path/to/backintime/host-name/root/1
total 76
drwxr-xr-x 3 root root 4096 Nov 30 20:40 20241130-202500-338/
drwx------ 2 root root 4096 Dec  9 07:48 20241208-164000-950/
drwx------ 2 root root 4096 Dec 26 09:35 20241225-181022-246/
drwx------ 3 root root 4096 Dec 28 18:53 20241228-183500-747/
drwxr-xr-x 3 root root 4096 Dec 31 17:47 20241231-171000-165/
drwxr-xr-x 3 root root 4096 Jan  5 11:31 20250105-110500-633/
drwxr-xr-x 3 root root 4096 Jan 11 16:05 20250111-152000-276/
drwxr-xr-x 3 root root 4096 Jan 19 12:51 20250119-123000-501/
drwxr-xr-x 3 root root 4096 Jan 26 16:20 20250126-160000-611/
drwx------ 3 root root 4096 Jan 30 12:19 20250130-115000-674/
drwxr-xr-x 3 root root 4096 Feb  6 09:41 20250206-091500-641/
drwxr-xr-x 3 root root 4096 Feb 16 14:55 20250216-142000-122/
drwxr-xr-x 3 root root 4096 Feb 23 19:06 20250223-181219-853/
drwxr-xr-x 3 root root 4096 Feb 26 19:49 20250226-193001-892/
drwxr-xr-x 3 root root 4096 Feb 28 08:17 20250228-075528-693/
drwxr-xr-x 3 root root 4096 Mar  4 09:14 20250304-084501-338/
drwxr-xr-x 3 root root 4096 Mar  6 11:55 20250306-100000-310/
drwx------ 3 root root 4096 Mar 10 13:51 20250310-132013-454/
drwxr-xr-x 3 root root 4096 Mar 10 18:20 20250310-140624-875/
lrwxrwxrwx 1 root root   19 Mar 10 18:20 last_snapshot -> 20250310-140624-875/
-rw-r--r-- 1 root root    0 Mar 10 13:51 smartremove.lck

When logging into the GUI on the client, backintime lists all but two of these backups. These are the two that are missing:

drwx------ 2 root root 4096 Dec  9 07:48 20241208-164000-950/
drwx------ 2 root root 4096 Dec 26 09:35 20241225-181022-246/

These are two of the five with the different permissions. I have several systems running backintime via ssh, and they all seem to have inconsistent permissions for the backups.

On the client:

$ backintime --diagnostics
{
    "backintime": {
        "name": "Back In Time",
        "version": "1.5.3",
        "running-as-root": false,
        "latest-config-version": 6,
        "local-config-file": "/home/UsernameReplaced/.config/backintime/config",
        "local-config-file-found": true,
        "global-config-file": "/etc/backintime/config",
        "global-config-file-found": false,
        "started-from": "/usr/share/backintime/common",
        "user-callback": "/home/UsernameReplaced/.config/backintime/user-callback",
        "keyring-supported": true
    },
    "host-setup": {
        "OS": {
            "/etc/os-release": "Arch Linux",
            "/etc/arch-release": "\n"
        },
        "platform": "Linux-6.12.17-1-lts-x86_64-with-glibc2.41",
        "system": "Linux #1 SMP PREEMPT_DYNAMIC Thu, 27 Feb 2025 14:12:30 +0000",
        "display-system": "tty",
        "locale": "en_AU, UTF-8",
        "PATH": "/usr/local/sbin:/usr/local/bin:/usr/bin:/usr/lib/jvm/default/bin:/usr/bin/site_perl:/usr/bin/vendor_perl:/usr/bin/core_perl",
        "RSYNC_OLD_ARGS": "(not set)",
        "RSYNC_PROTECT_ARGS": "(not set)"
    },
    "python-setup": {
        "python": "3.13.2 main Feb  5 2025 08:05:21 CPython GCC 14.2.1 20250128",
        "python-executable": "/usr/bin/python",
        "python-executable-symlink": true,
        "python-executable-resolved": "/usr/bin/python3.13",
        "sys.path": [
            "/usr/share/backintime/qt/plugins",
            "/usr/share/backintime/common/plugins",
            "/usr/share/backintime/plugins",
            "/usr/share/backintime/common",
            "/usr/lib/python313.zip",
            "/usr/lib/python3.13",
            "/usr/lib/python3.13/lib-dynload",
            "/usr/lib/python3.13/site-packages"
        ],
        "qt": {
            "Version": "PyQt 6.8.1 / Qt 6.8.2"
        }
    },
    "external-programs": {
        "rsync": {
            "version": "3.4.1",
            "protocol": "32.0",
            "capabilities": "file_bits: 64; inum_bits: 64; timestamp_bits: 64; long_int_bits: 64; socketpairs: True; symlinks: True; symtimes: True; hardlinks: True; hardlink_specials: True; hardlink_symlinks: True; IPv6: True; atimes: True; batchfiles: True; inplace: True; append: True; ACLs: True; xattrs: True; secluded_args: optional; iconv: True; prealloc: True; stop_at: True; crtimes: False",
            "optimizations": "SIMD_roll: True; asm_roll: False; openssl_crypto: True; asm_MD5: False",
            "checksum_list": "xxh128, xxh3, xxh64, md5, md4, sha1, none",
            "compress_list": "zstd, lz4, zlibx, zlib, none",
            "daemon_auth_list": "sha512, sha256, sha1, md5, md4"
        },
        "ssh": "OpenSSH_9.9p2, OpenSSL 3.4.1 11 Feb 2025",
        "sshfs": "3.7.3",
        "encfs": "1.9.5",
        "shell": "/usr/bin/bash",
        "shell-version": "GNU bash, version 5.2.37(1)-release (x86_64-pc-linux-gnu)"
    }
}

[buhtz]

Hello protist,

Thank you for taking the time to report the bug and providing the
details. I appreciate your high quality feedback, will investigate the issue,
and work on a solution to the best of my ability.

Do you run BIT in root mode or as a regular user?

I was not aware of smartremove.lck. My ToDo: Add this to maintain docu.

The permission thing: I don't have an idea yet how this can happen or how to reproduce it. I will check my own backups, too. It might be related to #988.

Smart remove (running in background) might be involved. If the snapshots with odd permissions are candidates for removal, something might have gone wrong while removing them. Mhm...

Not sure when we'll find the time to work on it. Please see the projects background information to get an idea about our workflow and priorities:

• Maintenance status
• The team
• Strategy outline / Roadmap

Best regards,
Christian

[protist]

I appreciate your high quality feedback

Ha, thank you. I'm not sure about that though. I feel like it was one of my briefer bug reports.

Do you run BIT in root mode or as a regular user?

root mode

I was not aware of smartremove.lck

I'm not sure if you know, but should this file be persisting over longer periods of time? I assumed it was just to prevent race conditions, so I would have thought it should disappear after the files were deleted, i.e. <1 hour? I wonder if my backups are in a strange situation because I have rebooted while smart removes are occurring. This would explain why smartremove.lck persists. In the meantime, should I delete this file?

Also, moving forward, should I just delete all the backups with drwx------? I'm happy to leave them a bit longer if it helps you troubleshoot, although presumably you need to be able to replicate the bug anyway. I'm also happy to wait and see if the other members of the team might have a different viewpoint, although you seem to be the main person responding to my bug reports anyway. 😄 Thanks again for all your hard work!

[buhtz]

I need to investigate first, what the intention of smartremove.lck is, before I can answer your question.

But I am assuming that you can remove it without causing harm. The same goes for the snapshots folders. To prevent situations like this in the future, you maybe can disable the "run in background" option.

I don't that me or someone else will find time in the near future to work on this. So you don't have to keep that snapshots for diagnosis reasons.

The "problem" with bugs like this is that the code structure and logging mechanic of BIT makes it very hard to diagnosis this. But I am working on this, too.

Regards,
Christian

[protist]

I need to investigate first

Thanks again @buhtz. Much appreciated. It does seems weird if a lock file is persisting across smart removes, and indeed reboots.

To prevent situations like this in the future, you maybe can disable the "run in background" option.

My memory was that if you have this off, then your client has to stay awake and alive for the duration of the smart remove? If so, that's not so ideal for laptops.

Thanks again for your clear response and continued efforts maintaining the code. I'll remove the strange backups, but leave smartremove.lck for now.

[buhtz]

Hello protist,
I did some investigation about that lock file and add the results to our maintenance docu (PR #2200).
In short I think it will be removed some day because it has no effect. It is never checked if it exists or not.

- Currently not clear. See issue #2055.
- Code location: `common/snapshots.py::Snapshots.smartRemove()` (introduced
  with commit 91d838fb in year 2025 by Germar)
- It seems that it is created only in SSH related modes when "Remove in
  background" is enabled. It is removed after the remove operation.
- Can not find a location in code for the existence of this file is
  checked. The assumption is, that it is part of an incomplete
  feature/behavior.
- Might be OK to remove it from code without replacement.

To your original problem about backups not being removed. I have no clear idea. I still think #988 is somehow involved here and the problem is quit complex. I don't see something else to do for your specific issue/ticket.

Because the problem is covered by #988 I would like to close your issue, if it is OK for you.

[protist]

Thanks @buhtz. Thanks for the information about smartremove.lck.

I'm totally happy to close it. You are the dev, so happy to be guided by what you think is best!

Having said that, regarding the permission issue, I'm not entirely sure if this is the same as #988. TBH I didn't read the entire 161 comments in that thread 😅, but that issue started with all previous snapshots being ignored? My issue is a bit different, because here they aren't just "ignored", but a permission issue, and also it appears to be occurring inconsistently. Also the solution in the other issue was to run --no-perms --no-group --no-owner, which seems like it's a backup issue (that occurs all the time). Mine is perhaps a smart remove issue?

I also just went back and checked the sizes of the "weird" backups, and some do indeed look like a stalled smart remove.

$ cd /path/to/backintime/host-name/root/1
$ sudo du -hs 20250319-174500-202 20250523-112000-874 20250531-122000-992 20250614-145000-363 20250619-163000-508 20250620-110500-299
4.0K	20250319-174500-202
4.0K	20250523-112000-874
469G	20250531-122000-992
17G	20250614-145000-363
4.0K	20250619-163000-508
13G	20250620-110500-299

I checked the three directories that were 4.0K, and they were empty.

EDIT: for the record the other three directories were the same size, so presumably not removed at all.

$ for i in 20250531-122000-992 20250614-145000-363 20250620-110500-299; do sudo du -hs $i; done
469G	20250531-122000-992
469G	20250614-145000-363
469G	20250620-110500-299