[Bug]: qBitManage can't move files it doesn't own even with proper group permissions

[datcactus]

Describe the Bug

Relevant starter info:
I run my docker services with individual user IDs and a shared group ID in order to keep permissions controlled. I set these with PUID/PGID. When a typical torrent workflow comes through it's not uncommon for file ownership to change whenever hardlinks are enabled, or if I'm just manually adding a torrent myself. qbitmanage runs with the same user and group as qbit to keep issues down, but permissions issues were observed with files that were owned by my standard 1000 user and the shared group 1215.

I noticed when I enabled QBT_REM_ORPHANED I received several error messages:

[Errno 1] Operation not permitted : Copying files instead.

When I exec into the container and perform manual mv, ln (hardlink), and other file relocation commands I have no issues at all.

In testing, I found that if I create a file from the container itself (so it's owned by the qbitmanage/qbit user) then it works as expected. However, if a file has a different owner even if it's part of the same group it will throw permission errors and copy the files instead of moving them as it should. There's no remove mechanism either, so every time qbitmanage runs this generates more copy commands which causes unnecessary disk activity.

Example of a file that will experience permission issues:

 cactus@hermes  /mnt/data/torrents/tv  stat cactus-test.txt
  File: cactus-test.txt
  Size: 66              Blocks: 1          IO Block: 131072 regular file
Device: 30h/48d Inode: 7982        Links: 1
Access: (0664/-rw-rw-r--)  Uid: ( 1000/  cactus)   Gid: ( 1215/dockermedia)
Access: 2023-07-07 18:11:47.703348471 -0400
Modify: 2023-07-07 18:11:47.704412165 -0400
Change: 2023-07-07 18:11:47.704412165 -0400
 Birth: -

orphan cleanup run:
| /data/torrents/tv/(redacted)-test.txt                                                              |
| Moving 1 Orphaned files to /data/torrents/orphaned_data/                                           |
| [Errno 1] Operation not permitted : Copying files instead.

Example of a working file from inside the container:

d75df54f68a4:/data/torrents/tv$ stat test-file.txt
  File: test-file.txt
  Size: 39              Blocks: 17         IO Block: 131072 regular file
Device: 0,48    Inode: 7659        Links: 1
Access: (0644/-rw-r--r--)  Uid: ( 1009/   hotio)   Gid: ( 1215/   hotio)
Access: 2023-07-07 18:38:56.216552267 -0400
Modify: 2023-07-07 18:38:56.244222709 -0400
Change: 2023-07-07 18:38:56.244235939 -0400
 Birth: -

orphan cleanup run:
[2023-07-07 18:39:01,036] [rem_orphaned:68]           [INFO]     | /data/torrents/tv/test-file.txt                                                                    |
[2023-07-07 18:39:01,036] [rem_orphaned:69]           [INFO]     | Moving 1 Orphaned files to /data/torrents/orphaned_data/                                           |
[2023-07-07 18:39:01,040] [rem_orphaned:89]           [INFO]     | Removing newly empty directories

The python process calling qbitmanage runs with the correct user IDs, so this should work as expected even if it's not owned by the user given how the shared group is configured.

Config

No response

Logs

https://gist.github.com/datcactus/d53853b83526923f13bdc8a840482cbb

Screenshots

No response

Installation

Docker

Version Number

4.0.2

What branch are you on?

master

[bakerboy448]

Docker compose as well and ls -lha of the problem file from in the container

[datcactus]

Docker compose as well and ls -lha of the problem file from in the container

Compose:
https://gist.github.com/datcactus/308bb312c618bdc13c32bd780ca41644

ls -lha of problem file (where 1000 is just my standard cactus user that created the file, and hotio is the mapped 1215 from the container's perspective):

cc23d92ef6fa:/$ ls -lha /data/torrents/tv/cactus-test.txt
-rw-rw-r-- 1 1000 hotio 48 Jul  7 20:26 /data/torrents/tv/cactus-test.txt

Directory perms since they're relevant when using mv (where hotio:hotio is 1009:1215):

cc23d92ef6fa:/data/torrents# tree -pugd -L 1
[drwxrwxr-x hotio    hotio   ]  .
├── [drwxrwxr-x hotio    hotio   ]  audiobooks
├── [drwxrwxr-x hotio    hotio   ]  books
├── [drwxrwxr-x hotio    hotio   ]  cross-seed
├── [drwxrwxr-x hotio    hotio   ]  isos
├── [drwxrwxr-x hotio    hotio   ]  movies
├── [drwxrwxr-x hotio    hotio   ]  music
├── [drwxrwxr-x hotio    hotio   ]  orphaned_data
└── [drwxrwxr-x hotio    hotio   ]  tv

[bakerboy448]

Looking at the code it seems likely this fails as the orphaned file logic attempts to set the last modified time on the file (to facilitate recycle bin cleanup) which requires owning the file.

[datcactus]

@bakerboy448 Interesting. I wonder if there's some way that os.utime works that's different than a standard touch.

From the container, a standard touch seems to work just fine to update times:

2b57880f8a72:/data/torrents/tv$ stat cactus-test.txt
  File: cactus-test.txt
  Size: 62              Blocks: 17         IO Block: 131072 regular file
Device: 0,51    Inode: 7667        Links: 1
Access: (0664/-rw-rw-r--)  Uid: ( 1000/ UNKNOWN)   Gid: ( 1215/   hotio)
Access: 2023-07-08 01:47:07.178065837 -0400
Modify: 2023-07-08 01:47:07.179044102 -0400
Change: 2023-07-08 01:47:07.179044102 -0400
 Birth: -
2b57880f8a72:/data/torrents/tv$ touch cactus-test.txt
2b57880f8a72:/data/torrents/tv$ stat cactus-test.txt
  File: cactus-test.txt
  Size: 62              Blocks: 17         IO Block: 131072 regular file
Device: 0,51    Inode: 7667        Links: 1
Access: (0664/-rw-rw-r--)  Uid: ( 1000/ UNKNOWN)   Gid: ( 1215/   hotio)
Access: 2023-07-08 01:48:01.627245932 -0400
Modify: 2023-07-08 01:48:01.627246132 -0400
Change: 2023-07-08 01:48:01.627265162 -0400
 Birth: -
2b57880f8a72:/data/torrents/tv$ id
uid=1009(hotio) gid=1215(hotio) groups=1215(hotio),100(users)

[bobokun]

Although it doesn't fix it, but I added a workaround, after copying the file it'll delete the original source so essentially "moving" the file by copying/deleting the original. Let me know if that works for you

[datcactus]

@bobokun I think as long as it can cleanup the file it's a decent workaround, but I'm mostly just confused about why a manual touch works just fine but the one being performed by os.utime has a strict permission requirement or some other underlying issue. Is there another way that the function could be constructed such that it doesn't have to directly perform that call on the file? Or, is this something that would be caused by having atime disabled on the underlying NFS share that feeds my media? A standard touch command works fine even with atime disabled so I'm not really sure how to debug this further.

There should be a way to get this to work with group permissions. While this doesn't present an issue for someone who just runs everything as root or 1000:1000 it's problematic in an environment where individual apps have their own user (or in this case qbit and qbitmanage run as the same qbit user) for their volume permissions and shared data access is managed via a shared group.

[rinseaid]

@bobokun, I face the same problem. The update in 4.0.3 does indeed work around the issue, but of course is slow and fills the log with errors. To provide further context, the user/group running the container is the same user/group that has ownership of the files (in terms of POSIX permissions), note, however that I am using extended ACLs.

Is there a reason to change the file modified timestamp? If I change the argument to 'false' on the move_file command in qbittorrent.py, it appears to work without any errors but I'm not sure if there would be any unintended consequence of doing so.

Edit: I am guessing this is for the clean up after x days logic.