Support virtual-hosted–style only S3-compatible remote? #10256
Comments
|
DVC should work properly with the virtual-host style endpointurl addressing. The error indicates that you don't have the right permissions to access that bucket. Are you able to use the AWS CLI to access that byte-plus bucket? |
|
It looks like you have to pass |
|
In botocore it defaults to |
|
The error given with s3fs/DVC is a permission error when accessing a specific file. It does not look like s3fs is not failing to create the client session (InvalidPathAccess is not the raised exception when you use DVC). This would normally mean that the issue is specifically with the credentials you are setting for your DVC remote (and not that the issue is due to an incorrect addressing style) @0ut0fcontrol can you verify that you are able to access your bucket via the AWS CLI? |
|
@pmrowla Yes, I can access my bucket via aws cli: And, if I insert
|
|
This looks to me like it may be an aiobotocore or botocore bug since their behavior to default to virtual host addressing and fall back to path addressing may not be working correctly? But we can add support for setting this explicitly in dvc-s3 dvc-s3 (we set config timeouts in the same way). It will also need to get added to the DVC remote config schema |
pmrowla
added
fs: s3
Does dvc support virtual-hosted–style S3-compatible remote?
From doc, dvc seem dvc only support path-style S3 remote.
When I try to use dvc with Tinder Object Storage (TOS) which only support virtual-hosted–style S3, it report error.
Error info:
$ cat .dvc/config [core] remote = tos ['remote "tos"'] url = s3://xxx/xxx (edited) endpointurl = https://tos-s3-xxx.xxxx.com （edited） $ dvc push Collecting |0.00 [00:00, ?entry/s] Pushing ERROR: unexpected error - Forbidden: An error occurred (403) when calling the HeadObject operation: Forbidden Having any troubles? Hit us up at https://dvc.org/support, we are always happy to help! $ dvc push --verbose 2024-01-25 17:21:30,113 DEBUG: v3.42.0 (brew), CPython 3.12.1 on macOS-14.2.1-arm64-arm-64bit 2024-01-25 17:21:30,113 DEBUG: command: /opt/homebrew/bin/dvc push --verbose Collecting |0.00 [00:00, ?entry/s] 2024-01-25 17:21:30,346 DEBUG: Preparing to transfer data from 'x' 2024-01-25 17:21:30,346 DEBUG: Preparing to collect status from 'x' 2024-01-25 17:21:30,346 DEBUG: Collecting status from 'x' 2024-01-25 17:21:30,347 DEBUG: Querying 1 oids via object_exists Pushing 2024-01-25 17:21:30,689 ERROR: unexpected error - Forbidden: An error occurred (403) when calling the HeadObject operation: Forbidden Traceback (most recent call last): File "/opt/homebrew/Cellar/dvc/3.42.0/libexec/lib/python3.12/site-packages/s3fs/core.py", line 113, in _error_wrapper return await func(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/homebrew/Cellar/dvc/3.42.0/libexec/lib/python3.12/site-packages/aiobotocore/client.py", line 408, in _make_api_call raise error_class(parsed_response, operation_name) botocore.exceptions.ClientError: An error occurred (403) when calling the HeadObject operation: Forbidden The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/opt/homebrew/Cellar/dvc/3.42.0/libexec/lib/python3.12/site-packages/dvc/cli/__init__.py", line 211, in main ret = cmd.do_run() ^^^^^^^^^^^^ File "/opt/homebrew/Cellar/dvc/3.42.0/libexec/lib/python3.12/site-packages/dvc/cli/command.py", line 27, in do_run return self.run() ^^^^^^^^^^ File "/opt/homebrew/Cellar/dvc/3.42.0/libexec/lib/python3.12/site-packages/dvc/commands/data_sync.py", line 64, in run processed_files_count = self.repo.push( ^^^^^^^^^^^^^^^ File "/opt/homebrew/Cellar/dvc/3.42.0/libexec/lib/python3.12/site-packages/dvc/repo/__init__.py", line 65, in wrapper return f(repo, *args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/homebrew/Cellar/dvc/3.42.0/libexec/lib/python3.12/site-packages/dvc/repo/push.py", line 144, in push push_transferred, push_failed = ipush( ^^^^^^ File "/opt/homebrew/Cellar/dvc/3.42.0/libexec/lib/python3.12/site-packages/dvc_data/index/push.py", line 75, in push result = transfer( ^^^^^^^^^ File "/opt/homebrew/Cellar/dvc/3.42.0/libexec/lib/python3.12/site-packages/dvc_data/hashfile/transfer.py", line 203, in transfer status = compare_status( ^^^^^^^^^^^^^^^ File "/opt/homebrew/Cellar/dvc/3.42.0/libexec/lib/python3.12/site-packages/dvc_data/hashfile/status.py", line 178, in compare_status dest_exists, dest_missing = status( ^^^^^^^ File "/opt/homebrew/Cellar/dvc/3.42.0/libexec/lib/python3.12/site-packages/dvc_data/hashfile/status.py", line 150, in status exists.update(odb.oids_exist(hashes, jobs=jobs, progress=pbar.callback)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/homebrew/Cellar/dvc/3.42.0/libexec/lib/python3.12/site-packages/dvc_objects/db.py", line 422, in oids_exist return list(wrap_iter(remote_oids, callback)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/homebrew/Cellar/dvc/3.42.0/libexec/lib/python3.12/site-packages/dvc_objects/db.py", line 36, in wrap_iter for index, item in enumerate(iterable, start=1): File "/opt/homebrew/Cellar/dvc/3.42.0/libexec/lib/python3.12/site-packages/dvc_objects/db.py", line 370, in list_oids_exists in_remote = self.fs.exists(paths, batch_size=jobs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/homebrew/Cellar/dvc/3.42.0/libexec/lib/python3.12/site-packages/dvc_objects/fs/base.py", line 472, in exists return fut.result() ^^^^^^^^^^^^ File "/opt/homebrew/Cellar/python@3.12/3.12.1_1/Frameworks/Python.framework/Versions/3.12/lib/python3.12/concurrent/futures/_base.py", line 456, in result return self.__get_result() ^^^^^^^^^^^^^^^^^^^ File "/opt/homebrew/Cellar/python@3.12/3.12.1_1/Frameworks/Python.framework/Versions/3.12/lib/python3.12/concurrent/futures/_base.py", line 401, in __get_result raise self._exception File "/opt/homebrew/Cellar/dvc/3.42.0/libexec/lib/python3.12/site-packages/dvc_objects/executors.py", line 135, in batch_coros result = fut.result() ^^^^^^^^^^^^ File "/opt/homebrew/Cellar/dvc/3.42.0/libexec/lib/python3.12/site-packages/s3fs/core.py", line 1035, in _exists await self._info(path, bucket, key, version_id=version_id) File "/opt/homebrew/Cellar/dvc/3.42.0/libexec/lib/python3.12/site-packages/s3fs/core.py", line 1302, in _info out = await self._call_s3( ^^^^^^^^^^^^^^^^^^^^ File "/opt/homebrew/Cellar/dvc/3.42.0/libexec/lib/python3.12/site-packages/s3fs/core.py", line 348, in _call_s3 return await _error_wrapper( ^^^^^^^^^^^^^^^^^^^^^ File "/opt/homebrew/Cellar/dvc/3.42.0/libexec/lib/python3.12/site-packages/s3fs/core.py", line 140, in _error_wrapper raise err PermissionError: Forbidden 2024-01-25 17:21:30,721 DEBUG: Version info for developers: DVC version: 3.42.0 (brew) -------------------------- Platform: Python 3.12.1 on macOS-14.2.1-arm64-arm-64bit Subprojects: dvc_data = 3.8.0 dvc_objects = 3.0.6 dvc_render = 1.0.1 dvc_task = 0.3.0 scmrepo = 2.0.4 Supports: azure (adlfs = 2023.12.0, knack = 0.11.0, azure-identity = 1.15.0), gdrive (pydrive2 = 1.19.0), gs (gcsfs = 2023.12.2.post1), http (aiohttp = 3.9.1, aiohttp-retry = 2.8.3), https (aiohttp = 3.9.1, aiohttp-retry = 2.8.3), oss (ossfs = 2023.12.0), s3 (s3fs = 2023.12.2, boto3 = 1.34.22), ssh (sshfs = 2023.10.0), webdav (webdav4 = 0.9.8), webdavs (webdav4 = 0.9.8), webhdfs (fsspec = 2023.12.2) Config: Global: /Users/bytedance/Library/Application Support/dvc System: /opt/homebrew/share/dvc Cache types: reflink, hardlink, symlink Cache directory: apfs on /dev/disk3s3s1 Caches: local Remotes: s3 Workspace directory: apfs on /dev/disk3s3s1 Repo: dvc, git Repo.site_cache_dir: /opt/homebrew/var/cache/dvc/repo/b8147adaf473f039b47ac961430c05d4 Having any troubles? Hit us up at https://dvc.org/support, we are always happy to help! 2024-01-25 17:21:30,725 DEBUG: Analytics is enabled. 2024-01-25 17:21:30,755 DEBUG: Trying to spawn ['daemon', 'analytics', '/var/folders/x3/m9b0vhpn28d_557yb666lf4w0000gn/T/tmpay2t_qwd', '-v'] 2024-01-25 17:21:30,762 DEBUG: Spawned ['daemon', 'analytics', '/var/folders/x3/m9b0vhpn28d_557yb666lf4w0000gn/T/tmpay2t_qwd', '-v'] with pid 98616The text was updated successfully, but these errors were encountered: