BlobFuse Not working in RHEL 8.2 #803
Comments
|
Enabled rsyslog as well, below are the details: tail -100f blobfuse.log |
|
You are saying it was working with 1.4.1 and after the upgrade it has stopped working? |
|
Can you share logs after the line "Delaying authentication to post fork ". If you see some requests going and coming back with response status "0" then blobfuse is not able to connect to azure. |
|
Initially we have installed 1.4.1 version and it was working and later a week ago it stopped working, so I tried to configure again and then I upgraded but when I execute the blobfuse mount command it got hung and not showing any output, when I checked the logs it was showing only the above mentioned details. IT took time so I cancelled and if I list the files in the mounted directory it was throwing below error and the mount directory showing this permissions [root@USA****** blobfuse]# ll In logs directory I don't see any logs after this and there is no response as well |
|
Can you share your config file? |
|
Unable to use -d flag argument. [root@USA********** blobfuse]# blobfuse datafiles --tmp-path=/opt/data/softwares/blobfuse/temp --config-file=/opt/data/softwares/blobfuse/fuse_connection.cfg -o attr_timeout=240 -o entry_timeout=240 -o negative_timeout=120 -o allow_other --log-level=LOG_DEBUG --d flag blobfuse datafiles --tmp-path=/opt/data/softwares/blobfuse/temp --config-file=/opt/data/softwares/blobfuse/fuse_connection.cfg -o attr_timeout=240 -o entry_timeout=240 -o negative_timeout=120 -o allow_other --log-level=LOG_DEBUG -d flag I have tried both the options |
|
Run this command |
|
Please share the logs as well |
|
I have executed the mount command with -d option but it doesn't show any output and the command is still running. Below are the logs for the same. The logs are also not showing any details after ** Delaying authentication to post fork for older curl versions Jun 1 10:02:28 usa******** blobfuse[1027432]: Using syscall to detect directory is already mounted or not |
|
@GunaGopal, while the above command is running, can you try to run ls command in the mounted directory in a different shell. Let us know what it outputs. At the same time, it should also add some logs. |
|
When you run the above command, you should also see some logs in the console |
|
@souravgupta-msft - I have replicated the steps as you mentioned but while executing the mount command I'm unable to switch to the mounted directory, the terminal got hung and showing no output then I cancelled (ctrl+C) and then I got the this error Transport endpoint is not connected Below are the output: [root@USAZUAUDDE04776 blobfuse]# cd datafiles Logs are showing the same details, please find below logs: Jun 2 06:32:40 usa******** blobfuse[1524715]: Using syscall to detect directory is already mounted or not |
|
When you mount blobfuse with "-d" option, it will mount in foreground and hence your terminal will be blocked until unmount is done. Once you mount with this flag, use another terminal to check your file IO and while any read/write operation is in progress you will see some logs on the blobfuse console as well as in the log files. If this also does not show any signs of logs, can you try another cli option "--pre-mount-validate=true". As logs are not getting updated, we are not able to analyze the failure. |
|
Yes, we are unable to check any logs or list files in the mounted directory until it is unmounted. I have tried the cli option which you have mentioned and I got the below output [root@USA***** blobfuse]# blobfuse datafiles --tmp-path=/opt/data/softwares/blobfuse/temp --config-file=/opt/data/softwares/blobfuse/fuse_connection.cfg -o attr_timeout=240 -o entry_timeout=240 -o negative_timeout=120 -o allow_other --log-level=LOG_DEBUG --pre-mount-validate=true Below are the logs for the same: Jun 2 08:43:00 usa******** blobfuse[1570884]: Using syscall to detect directory is already mounted or not |
|
@souravgupta-msft in the logs it is showing successfully authenticated but the files are not synced with storage account. |
|
I have tried installing the blob fuse in the other VM and with different storage account but still facing the same issue. Here the blob fuse version is 1.4.1 and linux version is Redhat 8.2 Jun 2 12:46:31 usa******** blobfuse[2671100]: Using syscall to detect directory is already mounted or not Could you please let us know if anyother configuration is required in the server apart from the installation. I have installed blobfuse using "yum install blobfuse" |
|
@GunaGopal try out the following:
|
|
I have executed the above commands and still I don't see any update in logs. I have also tried the azcopy command, with this I'm able to download the files from the container using the same account key [root@USA*************** blobfuse]# az storage blob download -c repo -n asdasda.txt -f asdasda.txt --account-name np******** --account-key '******' { |
|
Are you mounting with sudo, if no can you give that a try. As logs are not coming at all, I suspect mount is not going through fine. Also in above thread you mentioned that it was working earlier, any idea what has changed at system level that led us to this issue. One more option is to give our next-gen blobfuse (Blobfuse2) a try (https://github.com/Azure/azure-storage-fuse/releases/tag/blobfuse2-2.0.0-preview.1). This requires configuration to be done in little different way and uses libfuse3 driver. You can refer readme for Blobfuse2 here (https://github.com/Azure/azure-storage-fuse/tree/main). However this is still in preview mode. |
|
Also just to confirm, your setup does not involve any sort of proxy? |
|
Hi @vibhansa-msft - I have tried mounting with sudo as well but still it is not working as expected, below is the output of the command and no additional logs were traced. [root@USA******* blobfuse]# sudo blobfuse datafiles --tmp-path=/opt/data/softwares/blobfuse/temp --config-file=/opt/data/softwares/blobfuse/fuse_connection.cfg -o attr_timeout=240 -o entry_timeout=240 -o negative_timeout=120 -o allow_other --log-level=LOG_DEBUG --pre-mount-validate=true |
|
"Mountpoint not empty" is a different error then the issue you are facing. It just tells that the mount directory you have given is not empty and hence mount cannot succeed. In such cases just clean-up the mount directory and retry. |
|
I have unmounted and removed the mounted directory (datafiles), created new directory and tried executing the below command but still facing the same issue. And I have exited and when I'm listing the files I'm getting "Transport endpoint not connected" Below are the logs for the same: [root@USA******** blobfuse]# sudo blobfuse datafiles --tmp-path=/opt/data/softwares/blobfuse/temp --config-file=/opt/data/softwares/blobfuse/fuse_connection.cfg -o attr_timeout=240 -o entry_timeout=240 -o negative_timeout=120 -o allow_other --log-level=LOG_DEBUG --pre-mount-validate=true |
|
"Transport endpoint not connected": generally, comes when blobfuse binary has terminated abruptly. Can you enable core dumping in your system so that if blobfuse crashes we can get a core file and analyze where its failing? |
|
Also post mount when you did "ll" I see there are two files listed, please confirm they are there in your container. Just want to check that we are trying to retrieve something from the container. As "ll" shows some output, it means blobfuse was connected to container and doing a listing there and hence there should have been some logs in the log file related to these operations. |
|
I have configured the logs as mentioned above and still I could see only the same set of logs |
|
Did you get a chance to try out core dumping? |
|
Were you able to give blobfuse2 a try ? |
|
Hi @vibhansa-msft, We have tried installing the blobfuse2 in one of our environment, the command gets completed and it is mounted but when I try to move or list the files in the mounted directory, the command wasn't not showing any output/ not responding. Attached the logs for your reference. Command Output: [root@USA******* blobfuse]# blobfuse2 mount list |
|
Thanks for sharing the logs. With Blobfuse2 atleast we are getting the logs. I can see below line in the shared log file Post this we expect libfuse library to mount the FS and call us back to begin our processing. However in logs I do not see that call coming. It shall look something like " Libfuse::libfuse3_init : init", which is missing. |
|
If the setup is something similar to containers or AKS then we need to give some sort of privileged access while creating the container. Please check you have sufficient permissions to use fuse on this system. |
|
I have installed the fuse drive using this "sudo yum install fuse3 fuse3-devel". I was trying to unmount the blobfuse directory but I couldn't so I clean up the directory and tried mounting again but it is still the same. [root@USA********** log]# fusermount -V We are trying to install this in Azure VM's only and not using containers/AKS |
|
For Azure VMs we have our nightly pipeline setup for RHEL 8.2 and we do not face any problem there. Just for the cross-check can you give this a try on Ubuntu-20 VM? Just want to make sure its some environment specific issue or not. We do support RHEL7.5/7.8 as well if you want to stick to RHEL. |
|
Were you able to try out on Ubn or older RHEL versions ? |
|
yes, we have tried installing the blobfuse in the RHEL Lower version (7.7). However facing the same issue and below are the logs for the same and unable to cancel the command execution/list the files in that directory. Also we are unable to unmount the directory as well. Jun 14 08:34:21 us************ yum[13947]: Installed: blobfuse-1.3.6-1.el7.x86_64 |
|
Are you creating all these VMs using some template or directly using Azure portal? |
|
Yes we are using ARM template to provision the Azure VM's |
|
If you can share the template with us, we can try to reproduce this issue locally and investigate further. We have earlier tried creating a RHEL 8.2 VM through Azure portal (without any template, basic config) and we do not see any issue there. |
|
If the issue still persists kindly share the arm template. We are not able to reproduce this locally. |
|
As per the ICM update, disabling "CrowdStrike" makes it work. You need to check on your end why this malware detection is blocking fuse functionality. Closing this from here. Feel free to reopen if issue persists or you have any further questions. |
# Logging Please ensure logging is turned on DEBUG mode when trying to reproduce an issue. This can help in many instances to understand what the underlying issue is. A useful setting in your configuration file to utilize when debugging is `sdk-trace: true` under the azstorage component. This will log all outgoing REST calls. # BlobFuse2 Health Monitor One of the biggest BlobFuse2 features is our brand new health monitor. It allows customers gain more insight into how their BlobFuse2 instance is behaving with the rest of their machine. Visit [here](https://github.com/Azure/azure-storage-fuse/blob/main/tools/health-monitor/README.md) to set it up. # Common Mount Problems **1. Error: fusermount: failed to open /etc/fuse.conf: Permission denied** Only the users that are part of the group fuse, and the root user can run fusermount command. In order to mitigate this add your user to the fuse group. ```sudo addgroup <user> fuse``` **2. Error: mount command successful but log shows 'Failed to init fuse'** If are you using 'allow-other: true' config then make sure `user_allow_other` is enabled in /etc/fuse.conf file. By default /etc/fuse.conf will have this option disabled we just need to enable it and save the file. **3. failed to mount : failed to authenticate credentials for azstorage** There might be something wrong about the storage config, please double check the storage account name, account key and container/filesystem name. errno = 1** Possible causes are: - Invalid account, or access key - Non-existing container (The container must be created prior to Blobfuse2 mount) - Windows line-endings (CRLF) - fix it by running dos2unix - Use of HTTP while 'Secure Transfer (HTTPS)' is enabled on a Storage account - Enabled VNET Security rule that blocks VM from connecting to the Storage account. Ensure you can connect to your Storage account using AzCopy or Azure CLI - DNS issues/timeouts - add the Storage account resolution to /etc/hosts to bypass the DNS lookup - If using a proxy endpoint - ensure that you use the correct transfer protocol HTTP vs HTTPS **4. For MSI or SPN auth, Http Status Code = 403 in the response. Authorization error** - Verify your storage account Access roles. Make sure you have both Contributor and Storage Blob Contributor roles for the MSI or SPN identity. - In the case of a private AAD endpoint (private MSI endpoitns) ensure that your env variables are configured correctly. **5. fusermount: mount failed: Operation not permitted (CentOS)** fusermount is a privileged operation on CentOS by default. You may work around this changing the permissions of the fusermount operation: chown root /usr/bin/fusermount chmod u+s /usr/bin/fusermount **6. Cannot access mounted directory** FUSE allows mounting filesystem in user space, and is only accessible by the user mounting it. For instance, if you have mounted using root, but you are trying to access it with another user, you will fail to do so. In order to workaround this, you can use the non-secure, fuse option '--allow-other'. sudo blobfuse2 mount /home/myuser/mount_dir/ --config-file=config.yaml --allow-other **7. fusermount: command not found** You try to unmount the blob storage, but the recommended command is not found. Whilst `umount` may work instead, fusermount is the recommended method, so install the fuse package, for example on Ubuntu 20+: sudo apt install fuse3 please note the fuse version (2 or 3) is dependent on the linux distribution you're using. Refer to fuse version for your distro. **8. Hangs while mounting to private link storage account** The Blobfuse2 config file should specify the accountName as the original Storage account name and not the privatelink storage account name. For Eg: myblobstorageaccount.blob.core.windows.net is correct while privatelink.myblobstorageaccount.blob.core.windows.net is wrong. If the config file is correct, please verify name resolution dig +short myblobstorageaccount.blob.core.windows.net should return a private Ip For eg : 10.0.0.5 or so. If for some reason the translation/name resolution fails please confirm the VNet settings to ensure that it is forwarding DNS translation requests to Azure Provided DNS 168.63.129.16. In case the Blobfuse2 hosting VM is set up to forward to a Custom DNS Server, the Custom DNS settings should be verified, it should forward DNS requests to the Azure Provided DNS 168.63.129.16. Here are few steps to resolve DNS issues when integrating private endpoint with Azure Private DNS: Validate Private Endpoint has proper DNS record on Private DNS Zone. In case Private Endpoint was deleted and recreated a new IP may exist or duplicated records which will cause clients to use round-robin and make connectivity instable. Validate if DNS settings of the Azure VM has Correct DNS Servers. a) DNS settings can be defined VNET level and NIC Level. b) DNS setting cannot be set inside Guest OS VM NIC. For Custom DNS server defined check the following: Custom DNS Server forwards all requests to 168.63.129.16 Yes – you should be able to consume Azure Private DNS zones correctly. No – In that case you may need to create a conditional forwarder either to: privatelink zone or original PaaS Service Zone (check validation 4). Custom DNS has: a) DNS has Root Hits only – In this case is the best to have a forwarder configured to 168.63.129.16 which will improve performance and doesn't require any extra conditional forwarding setting. b) DNS Forwarders to another DNS Server (not Azure Provided DNS) – In this case you need to create a conditional forwarder to original PaaS domain zone (i.e. Storage you should configure blob.core.windows.net conditional forwarder to 168.63.129.16). Keep in mind using that approach will make all DNS requests to storage account with or without private endpoint to be resolved by Azure Provided DNS. By having multiple Custom DNS Serves in Azure will help to get better high availability for requests coming from On-Prem. **9. Blobfuse2 killed by OOM** The "OOM Killer" or "Out of Memory Killer" is a process that the Linux kernel employs when the system is critically low on memory. Based on its algorithm it kills one or more process to free up some memory space. Blobfuse2 could be one such process. To investigate Blobfuse2 was killed by OOM or not run following command: ``` dmesg -T | egrep -i 'killed process'``` If Blobfuse2 pid is listed in the output then OOM has sent a SIGKILL to Blobfuse2. If Blobfuse2 was not running as a service it will not restart automatically and user has to manually mount again. If this keeps happening then user need to monitor the system and investigate why system is getting low on memory. VM might need an upgrade here if the such high usage is expected. **10. Unable to access HNS enabled storage account behind a private end point** For HNS account, always add `type: adls` under `azstorage` section in your config file. Avoid using `endpoint` unless your storage account is behind a private endpoint. Blobfuse2 uses both blob and dfs endpoints to connect to storage account. User has to expose both these endpoints over private-endpoint for blobfuse2 to function properly. To create a private-endpoint for DFS in Azure portal: Go to your storage account -> Networking -> Private Endpoint connections. Click `+ Private endpoint`, fill in Subscription, Resource Group, Name, Network Interface Name and Region. Click next and under Target sub-resource select `dfs`. Click Virtual network and select virtual network and Subnet. Click DNS. Select Yes for Integrate with private DNS. Select the Subscription and Resource Group for your private link DNS. Select Next, Next and select Create. # Common Problems after a Successful Mount **1. Errno 24: Failed to open file /mnt/tmp/root/filex in file cache. errno = 24 OR Too many files Open error** Errno 24 in Linux corresponds to 'Too many files open' error which can occur when an application opens more files than it is allowed on the system. Blobfuse2 typically allows 20 files less than the ulimit value set in Linux. Usually the Linux limit is 1024 per process (e.g. Blobfuse2 in this case will allow 1004 open file descriptors at a time). Recommended approach is to edit the /etc/security/limits.conf in Ubuntu and add these two lines, * soft nofile 16384 * hard nofile 16384 16384 here refers to the number of allowed open files you must reboot after editing this file for Blobfuse2 to pick up the new limits. You may increase the limit via the command `ulimit -n 16834` however this does not appear in work in Ubuntu. **2. Input/output error** If you mounted a Blob container successfully, but failed to create a directory, or upload a file, it may be that you mounted a Blob container from a Premium (Page) Blob account which does not support Block blob. Blobfuse2 uses Block Blobs as files hence requires accounts that support Block blobs. `mkdir: cannot create directory ‘directoryname' : Input/output error` **3. Unexplainably high Storage Account list usage. Costs $$** The mostly likely reason is scanning triggered automatically using updatedb by the built-in mlocation service that is deployed with Linux VMs. "mlocation" is a built-in service that acts as a search tool. It is added under /etc/cron.daily to run on daily basis and it triggers the "updatedb" service to scan every directory on the server to rebuild the index of files in database in order to get the search result up-to-date. Solution: Do an 'ls -l /etc/cron.daily/mlocate' at the shell prompt. If "mlocate" is added to the /etc/cron.daily then Blobfuse2 must be whitelisted, so that the Blobfuse2 mount directory is not scanned by updatedb. This is done by updating the updatedb.conf file . cat /etc/updatedb.conf It should look like this. PRUNE_BIND_MOUNTS="yes" PRUNENAMES=".git .bzr .hg .svn" PRUNEPATHS="/tmp /var/spool /media /var/lib/os-prober /var/lib/ceph /home/.ecryptfs /var/lib/schroot" PRUNEFS="NFS nfs nfs4 rpc_pipefs afs binfmt_misc proc smbfs autofs iso9660 ncpfs coda devpts ftpfs devfs devtmpfs fuse.mfs shfs sysfs cifs lustre tmpfs usbfs udf fuse.glusterfs fuse.sshfs curlftpfs ceph fuse.ceph fuse.rozofs ecryptfs fusesmb" 1) Add the Blobfuse2 mount path eg: /mnt to the PRUNEPATHS OR 1) Add "Blobfuse2" and "fuse" to the PRUNEFS It won't harm to do both. Below are the steps to automate this at pod creation: 1.Create a new configmap in the cluster which contains the new configuration about the script. 2.Create a DaemonSet with the new configmap which could apply the configuration changes to every node in the cluster. ``` Example: configmap fiie: (testcm.yaml) apiVersion: v1 kind: ConfigMap metadata: name: testcm data: updatedb.conf: | PRUNE_BIND_MOUNTS="yes" PRUNEPATHS="/tmp /var/spool /media /var/lib/os-prober /var/lib/ceph /home/.ecryptfs /var/lib/schroot /mnt /var/lib/kubelet" PRUNEFS="NFS nfs nfs4 rpc_pipefs afs binfmt_misc proc smbfs autofs iso9660 ncpfs coda devpts ftpfs devfs devtmpfs fuse.mfs shfs sysfs cifs lustre tmpfs usbfs udf fuse.glusterfs fuse.sshfs curlftpfs ceph fuse.ceph fuse.rozofs ecryptfs fusesmb fuse Blobfuse2" DaemonSet file: (testcmds.yaml) apiVersion: apps/v1 kind: DaemonSet metadata: name: testcmds labels: test: testcmds spec: selector: matchLabels: name: testcmds template: metadata: labels: name: testcmds spec: tolerations: - key: "kubernetes.azure.com/scalesetpriority" operator: "Equal" value: "spot" effect: "NoSchedule" containers: - name: mypod image: debian volumeMounts: - name: updatedbconf mountPath: "/tmp" - name: source mountPath: "/etc" command: ["/bin/bash","-c","cp /tmp/updatedb.conf /etc/updatedb.conf;while true; do sleep 30; done;"] restartPolicy: Always volumes: - name: updatedbconf configMap: name: testcm items: - key: "updatedb.conf" path: "updatedb.conf" - name: source hostPath: path: /etc type: Directory ``` **4. File contents are not in sync with storage** Please refer to the file cache component setting `timeout-sec`. **5. failed to unmount /path/<mount dir>** Unmount fails when a file is open or a user or process is cd'd into the mount directory or its sub directories. Please ensure no files are in use and try the unmount command again. Even umount -f will not work if the mounted files /directories are in use. umount -l does a lazy unmount meaning it will unmount automatically when the mounted files are no longer in use. **6. Blobfuse2 mounts but not functioning at all** Azure#803 There are cases where anti-malware / anti-virus software block the fuse functionality and in such case though mount command is successful and Blobfuse2 binary is running, the fuse functionality will not work. One way to identify that you are hitting this issue is turn on the debug logs and mount Blobfuse2. If you do not see any logs coming from Blobfuse2 and potentially you have run into this issue. Stop the anti-virus software and try again. In such cases we have seen mounting through /etc/fstab works, because that executes mount command before the anti-malware software kicks in. **7. file cache temp directory not empty** To ensure that you don't have leftover files in your file cache temp dir, unmount rather than killing Blobfuse2. If Blobfuse2 is killed without unmounting you can also set `cleanup-on-start` in your config file on the next mount to clear the temp dir. **8. Unable to modify existing file (error: invalid argument)** By default `writeback-cache` is enabled for libfuse3 and this may result in append/write operations to fail. Either you can disable writeback-cache, which might hurt the performance or you can configure blobfuse2 to ignore open flags given by user and make it work with writeback-cache. To disable writeback-cache : Add `disable-writeback-cache: true` under libfuse section in your config file. To make it work with writeback-cache : Add `ignore-open-flags: true` under libfuse section in your config file. **9. Unable to list files/directories for non-HNS (flat-namespace) accounts** For non-HNS accounts blobfuse expects special directory marker files to exist in container to identify a directory. If these files do not exist then `virtual-directory: true` in `azstorage` section is required. **10. File size and LMT are updated but file contents are not refreshed** Blobfuse2 supports both fuse2 and fuse3 compatible linux distros. In all linux distros kernel cached contents of file in its page-cache. As long as cache is valid read/write are served from cache and calls will not reach to file-system drivers (blobfuse in our case). This page-cache is invalidated when page is swapped-out, manually cleared by user through cli or file-system driver requests for it. In case of fuse2 compliant distros, libfuse does not support invalidating the page cache. Contents once cached will remain with kernel until user manually clears the page-cache or kernel decides to swap it out. This means even if the file size or LMT has changed and blobfuse decided to refresh the content by redownloading the file, on read user will still get the stale contents. In case of fuse3 compliant distros, blobfuse configures libfuse to invalidate the page cache on file size or LMT change so this issue will not be hit. If user is observing that list or stat call to file shows updated time or size but contents are not reflecting accordingly, first confirm with blobfuse logs that file was indeed downloaded afresh. If file-cache-timeout has not expired then blobfuse will keep using the current version of file persisted on temp cache and contents will not be refreshed. If blobfuse has downloaded the latest file and user still observes stale contents then clear the kernel page-cache manually using ```sysctl -w vm.drop_caches=3``` command. If your workflow involves updating the file directly on container (not using blobfuse) and you wish to get latest contents on blobfuse mount then do the following (for fuse3 compliant linux distro only): - set all timeouts in libfuse section to 0 (entry, attribute, negative) - remove attr_cache from your pipeline section in config - set file-cache-timeout to 0 - in libfuse section of you config file add "disable-writeback-cache: true" # Problems with build Make sure you have correctly setup your GO dev environment. Ensure you have installed fuse3/2 for example: sudo apt-get install fuse3 libfuse3-dev -y PROPOSED CHANGE Azure#11 **11. Access problems with Blob fuse through blob-csi driver from AKS applications** There are users using Blob fuse to access the data in their storage account from applications running on AKS, users access data using blob-csi driver. If users redeploy the blob-csi driver (upgrade) with the Pods running, they could be reporting access problems for blobs and folders. From storage service logs, there are no logs about access issues. In this scenario, the AKS support has to collaborate with troubleshooting using the following SAP: Azure/Kubernetes Service (AKS)/Storage for CSI storage driver issues. What is seen on the AKS client side and/or Storage service side when they see the "access" issue is? From the storage side: The storage account connection may look fine for the Pod, no mount issues or errors for the application Pod. No error logs found from the application Pod. From the AKS side/app Pod: Users will have issues browsing the folders under blob storage mounts. Even if users attempt reinstallation of the blob-csi driver the problem could persist. Users may not see issues with storageClass, persistent volume(s) (PVs) and PV's configuration which mounts the blob storage inside the application pod. Possible solution (AKS support handles it): Reinstall blob-csi driver using a different method. Delete and deploy again the Pod to fix the problem. Links of reference: https://learn.microsoft.com/en-us/troubleshoot/azure/azure-kubernetes/mounting-azure-blob-storage-container-fail https://github.com/kubernetes-sigs/blob-csi-driver/blob/master/docs/install-csi-driver-master.md#clean-up-blob-csi-driver https://github.com/kubernetes-sigs/blob-csi-driver/blob/master/docs/csi-debug.md
Which version of the blobfuse was used?
Blobfuse version : 1.4.3
Which OS (please include version) are you using?
RHEL 8.2
What problem was encountered?
Blobfuse is not working, when I'm executing the below command it got hung and it is not showing any command output.
blobfuse datafiles --tmp-path=/opt/data/softwares/blobfuse/temp --config-file=/opt/data/softwares/blobfuse/fuse_connection.cfg -o attr_timeout=240 -o entry_timeout=240 -o negative_timeout=120 -o allow_other --log-level=LOG_DEBUG
Have you found a mitigation/solution?
Earlier we were using blob fuse version 1.4.1 then upgraded to 1.4.3 even after update it is not working
By default, blobfuse logs errors to syslog. If this is relevant, is there anything in the syslog that might be helpful?
Below are the logs:
Jun 1 06:23:04 usa**** blobfuse[930995]: Using syscall to detect directory is already mounted or not
Jun 1 06:23:04 usa**** blobfuse[930995]: Setting logging level to : LOG_DEBUG
Jun 1 06:23:04 usa**** blobfuse[930995]: Disk Thresholds : 90 - 80, Cache Eviction : 1000-0, List Cancel time : 0 Retry Policy (0, 60.000000, 1.200000), Background Download 0, Evict on sync 0
Jun 1 06:23:04 usa**** blobfuse[930995]: Blobfuse version : 1.4.3
Jun 1 06:23:04 usa**** blobfuse[930995]: release is 4.180000, Kernel version is 4.18.0-193.14.3.el8_2.x86_64
Jun 1 06:23:04 usa**** blobfuse[930995]: CURL Version is 7.61.1
Jun 1 06:23:04 usa**** blobfuse[930995]: ** Delaying tls init to post fork for older libcurl version
Jun 1 06:23:04 usa**** blobfuse[930995]: ** Delaying authentication to post fork for older curl versions
If relevant, please share your mount command.
blobfuse datafiles --tmp-path=/opt/data/softwares/blobfuse/temp --config-file=/opt/data/softwares/blobfuse/fuse_connection.cfg -o attr_timeout=240 -o entry_timeout=240 -o negative_timeout=120 -o allow_other --log-level=LOG_DEBUG
The text was updated successfully, but these errors were encountered: