You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
dvc checkout: permission errors with hardlinks and shared cache
Description
While working with hardlinks/symlinks and shared cache, DVC doesn't allow users different from owner to checkout (more precisely, create links between cache and their workspace) even if they belong to the same group as cache files.
Reproduce
0. Setup users, group and dvc cache path:
Users: max, egor, storage -> home folders are located in /home/{user}, mounted to the same volume
All users belong to group scoring
/home/storage and all its contents belong to group scoring
1. Create dvc cache dir with necessary permissions + setgid:
Check cache permissions find /home/storage/dvc_cache_test -type d -exec ls -al {} +:
/home/storage/dvc_cache_test:
total 12
drwxrwsr-x+ 3 max scoring 4096 Oct 28 12:35 .
drwxrws---+ 11 storage scoring 4096 Oct 28 10:59 ..
drwxrwsr-x+ 3 max scoring 4096 Oct 28 12:35 files
/home/storage/dvc_cache_test/files:
total 12
drwxrwsr-x+ 3 max scoring 4096 Oct 28 12:35 .
drwxrwsr-x+ 3 max scoring 4096 Oct 28 12:35 ..
drwxrwsr-x+ 3 max scoring 4096 Oct 28 13:01 md5
/home/storage/dvc_cache_test/files/md5:
total 12
drwxrwsr-x+ 3 max scoring 4096 Oct 28 13:01 .
drwxrwsr-x+ 3 max scoring 4096 Oct 28 12:35 ..
drwxrwsr-x+ 2 max scoring 4096 Oct 28 12:35 5f
/home/storage/dvc_cache_test/files/md5/5f:
total 12
drwxrwsr-x+ 2 max scoring 4096 Oct 28 12:35 .
drwxrwsr-x+ 3 max scoring 4096 Oct 28 13:01 ..
-r--r--r-- 2 max scoring 10 Oct 28 11:18 ebbef14389ebcfc3e501fa1091adcb
4. Change user and try to get data from cache
*Change user to egor, git clone, setup dvc cache as above
dvc checkout --verbose
2023-10-28 13:05:02,608 DEBUG: v3.27.0 (conda), CPython 3.12.0 on Linux-5.4.0-153-generic-x86_64-with-glibc2.31
2023-10-28 13:05:02,608 DEBUG: command: /home/egor/.conda/envs/dvc_test/bin/dvc checkout --verbose
Building workspace index |0.00 [00:00, ?entry/s]
Comparing indexes |2.00 [00:00, 1.45kentry/s]
2023-10-28 13:05:02,869 DEBUG: failed to create '/home/egor/dvc_test/test.txt' from '/home/storage/dvc_cache_test/files/md5/5f/ebbef14389ebcfc3e501fa1091adcb' - [Errno 95] no more link types left to try out: [Errno 1] Operation not permitted: '/home/storage/dvc_cache_test/files/md5/5f/ebbef14389ebcfc3e501fa1091adcb' -> '/home/egor/dvc_test/test.txt'
Traceback (most recent call last):
File "/home/egor/.conda/envs/dvc_test/lib/python3.12/site-packages/dvc_objects/fs/generic.py", line 273, in _try_links
_link(link, from_fs, from_path, to_fs, to_path)
File "/home/egor/.conda/envs/dvc_test/lib/python3.12/site-packages/dvc_objects/fs/generic.py", line 62, in _link
func(from_path, to_path)
File "/home/egor/.conda/envs/dvc_test/lib/python3.12/site-packages/dvc_objects/fs/base.py", line 387, in link
return self.fs.link(from_info, to_info)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/egor/.conda/envs/dvc_test/lib/python3.12/site-packages/dvc_objects/fs/local.py", line 168, in link
return system.hardlink(path1, path2)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/egor/.conda/envs/dvc_test/lib/python3.12/site-packages/dvc_objects/fs/system.py", line 32, in hardlink
os.link(src, link_name)
PermissionError: [Errno 1] Operation not permitted: '/home/storage/dvc_cache_test/files/md5/5f/ebbef14389ebcfc3e501fa1091adcb' -> '/home/egor/dvc_test/test.txt'
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/home/egor/.conda/envs/dvc_test/lib/python3.12/site-packages/dvc_objects/fs/generic.py", line 331, in transfer
_try_links(
File "/home/egor/.conda/envs/dvc_test/lib/python3.12/site-packages/dvc_objects/fs/generic.py", line 290, in _try_links
raise OSError(errno.ENOTSUP, "no more link types left to try out") from error
OSError: [Errno 95] no more link types left to try out
Applying changes |0.00 [00:00, ?file/s]
2023-10-28 13:05:02,870 DEBUG: Removing '/home/egor/dvc_test/test.txt'
2023-10-28 13:05:02,870 ERROR: Checkout failed for following targets:
test.txt
Is your cache up to date?
<https://error.dvc.org/missing-files>
Traceback (most recent call last):
File "/home/egor/.conda/envs/dvc_test/lib/python3.12/site-packages/dvc/cli/__init__.py", line 209, in main
ret = cmd.do_run()
^^^^^^^^^^^^
File "/home/egor/.conda/envs/dvc_test/lib/python3.12/site-packages/dvc/cli/command.py", line 26, in do_run
return self.run()
^^^^^^^^^^
File "/home/egor/.conda/envs/dvc_test/lib/python3.12/site-packages/dvc/commands/checkout.py", line 54, in run
raise exc
File "/home/egor/.conda/envs/dvc_test/lib/python3.12/site-packages/dvc/commands/checkout.py", line 34, in run
stats = self.repo.checkout(
^^^^^^^^^^^^^^^^^^^
File "/home/egor/.conda/envs/dvc_test/lib/python3.12/site-packages/dvc/repo/__init__.py", line 61, in wrapper
return f(repo, *args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/egor/.conda/envs/dvc_test/lib/python3.12/site-packages/dvc/repo/checkout.py", line 207, in checkout
raise CheckoutError([relpath(out_path) for out_path in failed], stats)
dvc.exceptions.CheckoutError: Checkout failed for following targets:
test.txt
Is your cache up to date?
<https://error.dvc.org/missing-files>
2023-10-28 13:05:02,872 DEBUG: Analytics is enabled.
2023-10-28 13:05:02,898 DEBUG: Trying to spawn '['daemon', '-q', 'analytics', '/tmp/tmpc4u_hu01']'
2023-10-28 13:05:02,900 DEBUG: Spawned '['daemon', '-q', 'analytics', '/tmp/tmpc4u_hu01']'
Expected
Successful operation as for the file owner, dvc checkout:
Building workspace index |0.00 [00:00, ?entry/s]
Comparing indexes |2.00 [00:00, 1.47kentry/s]
Applying changes |1.00 [00:00, 193file/s]
A test.txt
change group permissions on all the cache files to rwx -> currently that will contradict necessary 444 rights for cache files and lead to issues with other commands (like dvc exp run)
The text was updated successfully, but these errors were encountered:
Bug Report
Issue name
dvc checkout
: permission errors with hardlinks and shared cacheDescription
While working with hardlinks/symlinks and shared cache, DVC doesn't allow users different from owner to checkout (more precisely, create links between cache and their workspace) even if they belong to the same group as cache files.
Reproduce
0. Setup users, group and dvc cache path:
1. Create dvc cache dir with necessary permissions + setgid:
mkdir /home/storage/dvc_cache_test
sudo chown :scoring /home/storage/dvc_cache_test
sudo chmod -R g+s /home/storage/dvc_cache_test
2. Init repo, configure dvc cache
git init
dvc init
dvc config --local cache.dir /home/storage/dvc_cache_test
dvc config --local cache.type hardlink
dvc config --local cache.shared group
3. Add data to cache
echo "some data" > test.txt
dvc add test.txt
find /home/storage/dvc_cache_test -type d -exec ls -al {} +
:4. Change user and try to get data from cache
dvc checkout --verbose
Expected
Successful operation as for the file owner,
dvc checkout
:Environment information
Output of
dvc doctor
:Additional Information (if any):
dvc update *.dvc
on imported files from DVC data registry located on the same volume.P.S. Some bad ways to fix the problem:
rwx
-> currently that will contradict necessary444
rights for cache files and lead to issues with other commands (likedvc exp run
)The text was updated successfully, but these errors were encountered: