Bug: Can't mount with `-o allow_other` on WSL, and blobfuse2 print no error message and errno is 0 #1081

cvvz · 2023-03-13T07:36:40Z

Which version of blobfuse was used?

blobfuse2 2.0.2

Which OS distribution and version are you using?

WSL2.0

If relevant, please share your mount command.

blobfuse2 mount ~/myblob -o allow_other --file-cache-timeout-in-seconds=120 --use-attr-cache=true --cancel-list-on-mount-seconds=10 -o attr_timeout=120 -o entry_timeout=120 -o negative_timeout=120 --log-level=LOG_WARNING --cache-size-mb=1000 --container-name=pvc-0e85db3b-d632-49ab-b4ba-cc93d332a132 --pre-mount-validate=true --use-https=true --empty-dir-check=false --tmp-path=/mnt/tmp

What was the issue encountered?

mount failed, but blobfuse2 print no error message, and the errno is 0.

blobfuse mount failed as well, but it prints concrete error message which help me to solve this problem.

There is definitely a bug in blobfuse2, since it doesn't print error message and errno is 0 when the mount operation actually failed.
Another question is, why mount with -o allow_other on WSL would fail whereas Linux won't? I checked /etc/fuse.conf, the content of them is the same:

cat  /etc/fuse.conf   
                                                                                                                          15:26:36  羽3m 15.082s
# The file /etc/fuse.conf allows for the following parameters:
#
# user_allow_other - Using the allow_other mount option works fine as root, in
# order to have it work as user you need user_allow_other in /etc/fuse.conf as
# well. (This option allows users to use the allow_other option.) You need
# allow_other if you want users other than the owner to access a mounted fuse.
# This option must appear on a line by itself. There is no value, just the
# presence of the option.

#user_allow_other


# mount_max = n - this option sets the maximum number of mounts.
# Currently (2014) it must be typed exactly as shown
# (with a single space before and after the equals sign).

#mount_max = 1000

Have you found a mitigation/solution?

mount without -o allow_other on WSL

Please share logs if available.

	cat /var/log/blobfuse2.log
	Mar 13 14:30:28 WIN-QPHICKGMG9J blobfuse2[13282]: LOG_WARNING [mount.go (364)]: mount: unsupported v1 CLI parameter: pre-mount-validate is always true in blobfuse2.
	Mar 13 14:30:28 WIN-QPHICKGMG9J blobfuse2[13282]: LOG_CRIT [mount.go (384)]: Starting Blobfuse2 Mount : 2.0.2 on [Ubuntu 22.04.1 LTS]
	Mar 13 14:30:28 WIN-QPHICKGMG9J blobfuse2[13282]: LOG_CRIT [mount.go (385)]: Logging level set to : LOG_WARNING
	Mar 13 14:30:28 WIN-QPHICKGMG9J blobfuse2[13282]: LOG_WARNING [config.go (331)]: ParseAndValidateConfig : account endpoint not provided, assuming the default .core.windows.net style endpoint
	Mar 13 14:30:31 WIN-QPHICKGMG9J blobfuse2[13292]: LOG_WARNING [mount.go (364)]: mount: unsupported v1 CLI parameter: pre-mount-validate is always true in blobfuse2.
	Mar 13 14:30:31 WIN-QPHICKGMG9J blobfuse2[13292]: LOG_CRIT [mount.go (384)]: Starting Blobfuse2 Mount : 2.0.2 on [Ubuntu 22.04.1 LTS]
	Mar 13 14:30:31 WIN-QPHICKGMG9J blobfuse2[13292]: LOG_CRIT [mount.go (385)]: Logging level set to : LOG_WARNING
	Mar 13 14:30:31 WIN-QPHICKGMG9J blobfuse2[13292]: LOG_WARNING [config.go (331)]: ParseAndValidateConfig : account endpoint not provided, assuming the default .core.windows.net style endpoint
	Mar 13 14:30:31 WIN-QPHICKGMG9J blobfuse2[13292]: LOG_ERR [libfuse_handler.go (178)]: Libfuse::initFuse : failed to mount fuse
	Mar 13 14:30:31 WIN-QPHICKGMG9J blobfuse2[13292]: LOG_ERR [libfuse.go (150)]: Libfuse::Start : Failed to init fuse [failed to mount fuse]
	Mar 13 14:30:31 WIN-QPHICKGMG9J blobfuse2[13292]: LOG_ERR [mount.go (483)]: mount: error unable to start pipeline [failed to mount fuse]

The text was updated successfully, but these errors were encountered:

vibhansa-msft · 2023-03-13T08:52:58Z

"There is definitely a bug in blobfuse2, since it doesn't print error message and errno is 0 when the mount operation actually failed." : When fuse based system is mounted it forks into another process. This failure is hit in child process and by that time parent process is already complete and hence it can not print anything on that console. This is the reason to print the logs in a log file. Blobfuse will do config validation at its end before the fork where if any of your config is invalid, error will be printed and exit code will be non-zero but allow_other is a fuse config and hence it can not be validated by blobfuse binary itself.

cvvz · 2023-03-13T09:15:50Z

Why Blobfuse v1 can print error logs as expected but blobfuse2 can't?

cvvz · 2023-03-13T09:20:50Z

When fuse based system is mounted it forks into another process. This failure is hit in child process and by that time parent process is already complete and hence it can not print anything on that console.

I think parent process should wait for child process until it has completed the mount operation successfully, otherwise we can never know whether the mount operation is successful or not.

andyzhangx · 2023-03-13T09:26:36Z

can we reopen this issue?

cvvz · 2023-03-13T09:28:39Z

@vibhansa-msft blob csi driver rely on the result of blobfuse2 mount operation. If the actual result of mount is failed but the operation return success anyway it will cause confusing problem. I think this is a problem needs to solve.

vibhansa-msft · 2023-03-13T09:59:04Z

In case of v1 we cache the stderr handle and it uses that to print out the error message, but allow_other failure is not returned back in form of an error code. Once mount is complete (fork the child) parent always returns back with success there as well.
What we have done in v1 is also a hacky way only and not a clean design.

vibhansa-msft · 2023-03-13T09:59:50Z

One potential way to get the error on console and get a non-0 error code is to mount in foreground which means your shell will be blocked till the time blobfuse is running.

vibhansa-msft · 2023-03-13T10:00:58Z

User who has executed the mount command may not have access to /etc/fuse.conf file and hence we can not pre-validate that mount will work or not. Hence relying on fuse to provide the error is the only option we have here.

vibhansa-msft · 2023-03-13T10:03:32Z

Also, do you really need allow_other option to be there? It's more of a security threat using this option as this means your mount point is available to all the users on the system. So, unless really required you shall not use this feature.

cvvz · 2023-03-13T10:53:22Z

In case of v1 we cache the stderr handle and it uses that to print out the error message, but allow_other failure is not returned back in form of an error code.

I've tried blobfuse v1 and the errno is non-zero.

cvvz · 2023-03-13T10:56:09Z

One potential way to get the error on console and get a non-0 error code is to mount in foreground which means your shell will be blocked till the time blobfuse is running.

I think this behavior is acceptable, IMO, mount operation should be synchronized, otherwise we cannot know whether the mount point is ready for usage.

cvvz · 2023-03-13T10:56:56Z

do you really need allow_other option to be there?

No, it's just for testing.

vibhansa-msft · 2023-03-13T12:32:29Z

In case of v1 we cache the stderr handle and it uses that to print out the error message, but allow_other failure is not returned back in form of an error code.

I've tried blobfuse v1 and the errno is non-zero.

Yes I was investigating that from my end too. Reason for this is in v1 we fork using libfuse's mount api directly but in case of Golang, that does not work directly so we need to create a child process before we call libfuse's mount. Child process starts again from beginging and not from the point its forked and hence it's not possible here to catch that error code. I do not see any easy way out of this as of now. If possible either you can avoid using allow_other or try mounting in foreground which just means your shell will be hung up till blobfuse runs and may be you will need another bash to access the mount point.

cvvz · 2023-03-13T12:35:41Z

Is it possible to choose mounting in foreground using blobfuse2?

vibhansa-msft · 2023-03-14T02:44:05Z

Yes it is, you can provide a cli option "--foreground=true"to do the same.

cvvz · 2023-03-14T03:40:12Z

Is this a new option in blobfuse2 or blobfuse v1 also support?
Which mode (foreground or background) is the same behavior as v1?

vibhansa-msft · 2023-03-14T05:28:58Z

V1 also supports foreground mount with "-d" cli option, in V2 both -d and --foreground shall work fine. Also, just to reiterate if you mount in foreground your shell will be blocked until you unmount blobfuse.

cvvz · 2023-03-15T07:43:22Z

V1 also supports foreground mount with "-d" cli option

We never used this option in V1 before, but we did not encounter such problem ever. I wonder is there any behavior change between V1 and V2?

cvvz · 2023-03-15T07:49:21Z

Could you take a look to confirm whether there are any other similar issues like this could happen as well?

…ot enable in fuse.conf

… in fuse config (#1087) * Fix for #1081 to return with non zero status code if allow_other is not enable in fuse.conf

#1079 Pod deletion will cause global mount point to disappear. The root cause is csi driver will bind mount the global mount point into Pod volume immediately after blobfuse2 mount return successfully. However, fuse is still mounting at background, so actually bind mount and fuse mount happened at the same time, this will lead to a problem that unmount the bind mount (when deleting a Pod) will cause the original mount point unmounted as well. #1081 Mount is actually failed, but the errno returned by blobfuse2 is 0 and no error log (both terminal and log file).

cvvz changed the title ~~Bug: Can't mount with -o allow_other on WSL, but blobfuse2 print no error message and errno is 0~~ Bug: Can't mount with -o allow_other on WSL, and blobfuse2 print no error message and errno is 0 Mar 13, 2023

vibhansa-msft self-assigned this Mar 13, 2023

vibhansa-msft added this to the V2-2.0.3 milestone Mar 13, 2023

vibhansa-msft added the user-error label Mar 13, 2023

vibhansa-msft closed this as completed Mar 13, 2023

vibhansa-msft reopened this Mar 13, 2023

vibhansa-msft added the triaged label Mar 13, 2023

vibhansa-msft added a commit that referenced this issue Mar 17, 2023

Fix for #1081 to return with non zero status code if allow_other is n…

3e923dc

…ot enable in fuse.conf

vibhansa-msft mentioned this issue Mar 17, 2023

Exit with non-zero status code if allow_other is used but not enabled in fuse config #1087

Merged

vibhansa-msft linked a pull request Mar 17, 2023 that will close this issue

Exit with non-zero status code if allow_other is used but not enabled in fuse config #1087

Merged

cvvz mentioned this issue Mar 19, 2023

Fix: async (background) mount may cause problems #1088

Merged

vibhansa-msft closed this as completed in #1087 Mar 21, 2023

vibhansa-msft added a commit that referenced this issue Mar 21, 2023

Exit with non-zero status code if allow_other is used but not enabled…

5e06d43

… in fuse config (#1087) * Fix for #1081 to return with non zero status code if allow_other is not enable in fuse.conf

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Bug: Can't mount with `-o allow_other` on WSL, and blobfuse2 print no error message and errno is 0 #1081

Bug: Can't mount with `-o allow_other` on WSL, and blobfuse2 print no error message and errno is 0 #1081

cvvz commented Mar 13, 2023

vibhansa-msft commented Mar 13, 2023

cvvz commented Mar 13, 2023

cvvz commented Mar 13, 2023 •

edited

Loading

andyzhangx commented Mar 13, 2023

cvvz commented Mar 13, 2023 •

edited

Loading

vibhansa-msft commented Mar 13, 2023

vibhansa-msft commented Mar 13, 2023

vibhansa-msft commented Mar 13, 2023

vibhansa-msft commented Mar 13, 2023

cvvz commented Mar 13, 2023

cvvz commented Mar 13, 2023

cvvz commented Mar 13, 2023

vibhansa-msft commented Mar 13, 2023

cvvz commented Mar 13, 2023

vibhansa-msft commented Mar 14, 2023

cvvz commented Mar 14, 2023

vibhansa-msft commented Mar 14, 2023

cvvz commented Mar 15, 2023

cvvz commented Mar 15, 2023

Bug: Can't mount with -o allow_other on WSL, and blobfuse2 print no error message and errno is 0 #1081

Bug: Can't mount with -o allow_other on WSL, and blobfuse2 print no error message and errno is 0 #1081

Comments

cvvz commented Mar 13, 2023

Which version of blobfuse was used?

Which OS distribution and version are you using?

If relevant, please share your mount command.

What was the issue encountered?

Have you found a mitigation/solution?

Please share logs if available.

vibhansa-msft commented Mar 13, 2023

cvvz commented Mar 13, 2023

cvvz commented Mar 13, 2023 • edited Loading

andyzhangx commented Mar 13, 2023

cvvz commented Mar 13, 2023 • edited Loading

vibhansa-msft commented Mar 13, 2023

vibhansa-msft commented Mar 13, 2023

vibhansa-msft commented Mar 13, 2023

vibhansa-msft commented Mar 13, 2023

cvvz commented Mar 13, 2023

cvvz commented Mar 13, 2023

cvvz commented Mar 13, 2023

vibhansa-msft commented Mar 13, 2023

cvvz commented Mar 13, 2023

vibhansa-msft commented Mar 14, 2023

cvvz commented Mar 14, 2023

vibhansa-msft commented Mar 14, 2023

cvvz commented Mar 15, 2023

cvvz commented Mar 15, 2023

Bug: Can't mount with `-o allow_other` on WSL, and blobfuse2 print no error message and errno is 0 #1081

Bug: Can't mount with `-o allow_other` on WSL, and blobfuse2 print no error message and errno is 0 #1081

cvvz commented Mar 13, 2023 •

edited

Loading

cvvz commented Mar 13, 2023 •

edited

Loading