-
Notifications
You must be signed in to change notification settings - Fork 1k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
s3fs-fuse to HCP (Hitachi Content Platform) issue #2423
Comments
Adding this extra debug logging from this morning: [root@centos8s hcpdata]# s3fs testns -o use_cache=/tmp -o use_path_request_style -o allow_other -o no_check_certificate -o multireq_max=5 -o passwd_file=/etc/s3fs-passwd -o dbglevel=info -f -o curldbg /hcpdata -o url=https://testtenant.hcp1.demo.com I see these successful connection attempts in the HCP logs like it mounted the bucket fine: 192.168.1.101 - security [27/Feb/2024:10:26:06 -0500] "GET /testns/ HTTP/1.1" 200 248 testtenant@hs3 10 101 0 |
I added my SSL certificate to the bundle in /etc/pki/tls/certs/ca-bundle.crt and tried again with these results: [root@centos8s hcpdata]# s3fs testns -o use_cache=/tmp -o use_path_request_style -o allow_other -o multireq_max=5 -o passwd_file=/etc/s3fs-passwd -o dbglevel=info -f -o curldbg /hcpdata -o url=https://testtenant.hcp1.demo.com |
FYI, I just tested and versions 1.85, 1.87, and 1.89 work FINE with NO issue. |
I tested the latest 1.93 against our HCP Cloudscale Product and it works with no problem, but not our HCP product. Both are SIGV4 compliant. |
@HDS-nlucas Below are the three s3fs execution commands you reported:
The only difference between (1) and (2) is the presence or absence of the debug output option, but the log for (2) appears to be the result of a successful startup. I don't know why it failed in (1) (I can't determine it from the logs in (2) and (3)), but what is the problem? |
@ggtakec Correct, (2) also gave the IO error. (3) also gave the IO error. So, with that said, version 1.93 and 1.91 works fine with our newer HCP Cloudscale product. |
@HDS-nlucas |
@ggtakec Thanks for the followup. I just tested 1.94 against our HCP platform and it still gives the error: -bash: cd: /hcpdata: Input/output error This error is AFTER mounting the bucket successfully to the /hcpdata folder. On the HCP, I see it mounts successfully: |
Could you get the s3fs log(started with |
S3FS Version: 1.94-1.el9 Log from S3FS: [root@localhost ~]# s3fs testns -o use_cache=/tmp -o use_path_request_style -o allow_other -o multireq_max=5 -o no_check_certificate -o passwd_file=/etc/passwd-s3fs -o dbglevel=info -f -o curldbg /hcpdata -o url=https://testtenant.hcp1.demo.com/ Log from HCP S3 storage: 192.168.1.102 - security [15/Apr/2024:14:32:14 -0400] "GET /testns/ HTTP/1.1" 200 248 testtenant@hs3 52 101 0 Log from trying to access mounted bucket: |
Additional Information
Version of s3fs being used (
s3fs --version
)[root@centos8s ~]# s3fs --version
Amazon Simple Storage Service File System V1.93 (commit:unknown) with OpenSSL
I installed via:
yum install epel-release
yum install s3fs-fuse
Version of fuse being used (
pkg-config --modversion fuse
,rpm -qi fuse
ordpkg -s fuse
)[root@centos8s ~]# rpm -qi fuse
Name : fuse
Version : 2.9.7
Release : 17.el8
Architecture: x86_64
Install Date: Thu 22 Feb 2024 05:20:13 PM EST
Group : Unspecified
Size : 208380
License : GPL+
Signature : RSA/SHA256, Fri 28 Apr 2023 07:35:03 AM EDT, Key ID 05b555b38483c65d
Source RPM : fuse-2.9.7-17.el8.src.rpm
Build Date : Tue 18 Apr 2023 02:26:34 PM EDT
Build Host : x86-04.stream.rdu2.redhat.com
Relocations : (not relocatable)
Packager : builder@centos.org
Vendor : CentOS
URL : http://fuse.sf.net
Summary : File System in Userspace (FUSE) v2 utilities
Kernel information (
uname -r
)[root@centos8s ~]# uname -r
4.18.0-540.el8.x86_64
GNU/Linux Distribution, if applicable (
cat /etc/os-release
)[root@centos8s ~]# cat /etc/os-release
NAME="CentOS Stream"
VERSION="8"
ID="centos"
ID_LIKE="rhel fedora"
VERSION_ID="8"
PLATFORM_ID="platform:el8"
PRETTY_NAME="CentOS Stream 8"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:centos:centos:8"
HOME_URL="https://centos.org/"
BUG_REPORT_URL="https://bugzilla.redhat.com/"
REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8"
REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream"
How to run s3fs, if applicable
s3fs testns -o use_cache=/tmp -o use_path_request_style -o allow_other -o no_check_certificate -o multireq_max=5 -o passwd_file=/etc/s3fs-passwd /hcpdata -o url=https://testtenant.hcp1.demo.com
s3fs syslog messages (
grep s3fs /var/log/syslog
,journalctl | grep s3fs
, ors3fs outputs
)Feb 26 11:47:02 centos8s s3fs[10154]: s3fs version 1.93(unknown) : s3fs -o use_cache=/tmp -o use_path_request_style -o allow_other -o no_check_certificate -o multireq_max=5 -o passwd_file= /etc/s3fs-passwd -o url=https://testtenant.hcp1.demo.com testns /hcpdata
Feb 26 11:47:02 centos8s s3fs[10154]: Loaded mime information from /etc/mime.types
Feb 26 11:47:02 centos8s s3fs[10155]: init v1.93(commit:unknown) with OpenSSL, credential-library(built-in)
Details about issue
All seems to work ok, except after mounting the bucket to /hcpdata, when I do a df -kh it shows up correctly:
[root@centos8s ~]# df -kh
Filesystem Size Used Avail Use% Mounted on
devtmpfs 1.8G 0 1.8G 0% /dev
tmpfs 1.8G 0 1.8G 0% /dev/shm
tmpfs 1.8G 11M 1.8G 1% /run
tmpfs 1.8G 0 1.8G 0% /sys/fs/cgroup
/dev/mapper/cs-root 70G 6.0G 64G 9% /
/dev/mapper/cs-home 421G 3.1G 418G 1% /home
/dev/nvme0n1p1 1014M 274M 741M 27% /boot
tmpfs 364M 32K 364M 1% /run/user/1000
s3fs 4.0G 0 4.0G 0% /hcpdata
Then I do a cd /hcpdata, I get the error:
[root@centos8s ~]# cd /hcpdata
-bash: cd: /hcpdata: Input/output error
This happens on 1.93 and 1.91 code levels. I am an expert on Hitachi Content Platform (HCP) which is S3 SIGv4 compliant storage. I can do a Teams meeting to share and display the issue if necessary.
When I mount the bucket, this is the log information from HCP indicating it mounted the namespace./bucket correctly:
192.168.1.101 - security [26/Feb/2024:11:02:26 -0500] "GET /testns/ HTTP/1.1" 200 248 testtenant@hs3 51 101 0
192.168.1.101 - security [26/Feb/2024:11:02:34 -0500] "HEAD /testns// HTTP/1.1" 200 0 testtenant@hs3 3 101 0
The text was updated successfully, but these errors were encountered: