mkdir: cannot stat ‘/mnt/hgfs’: Input/output error during dnf update to 12.0.5 on Fedora 36 guest

[hermidalc]

Describe the bug

During dnf update to 12.0.5-1.fc36.x86_64 I get this error even though my /mnt/hgfs is there and I can stat it:

Running scriptlet: open-vm-tools-12.0.5-1.fc36.x86_64
mkdir: cannot stat ‘/mnt/hgfs’: Input/output error

$ stat /mnt/hgfs
  File: /mnt/hgfs
  Size: 4192      	Blocks: 9          IO Block: 1024   directory
Device: 0,37	Inode: 1           Links: 1
Access: (0555/dr-xr-xr-x)  Uid: ( 1000/hermidalc)   Gid: ( 1000/hermidalc)
Context: system_u:object_r:fusefs_t:s0
Access: 2022-06-03 09:46:52.000000000 -0400
Modify: 2022-06-03 09:46:52.000000000 -0400
Change: 2022-06-03 09:46:52.000000000 -0400
 Birth: -

Not sure if this is a big deal or not.

Reproduction steps

1. sudo dnf update
2.
3.
...

Expected behavior

No errors during dnf update

Additional context

No response

[dsouders]

EIO (Input/Output Error) usually points to a hardware failure, such as a bad disk or disk controller.
If you run "mkdir /mnt/hgfs", do you get the same error? Does it change if hgfs is unmounted?

[hermidalc]

EIO (Input/Output Error) usually points to a hardware failure, such as a bad disk or disk controller.
If you run "mkdir /mnt/hgfs", do you get the same error? Does it change if hgfs is unmounted?

Sorry if I’m not understanding what you are pointing to, but the directory path already existed and was mounted during the dnf update. I had a Windows host shared directory mounted under /mnt/hgfs via standard systemd VMware instructions.

[dsouders]

The dnf installer isn't part of open-vm-tools, it's provided by Fedora, so we don't know exactly what it's doing, but from the error message it appears that it was running "mkdir /mnt/hgfs" to create the hgfs mount point. There are a few possible scenarios, depending on when during the upgrade the error occurred:

1. hgfs is mounted and hgfs is running, so accessing /mnt/hgfs is accessing the shared file system.
2. hgfs is unmounted, so /mnt/hgfs is simply a directory in the root file system.
3. hgfs is mounted, but hgfs is either not running and/or in the process of being updated. If this is the case, then accessing /mnt/hgfs will try to acccess hgfs, but that will fail because hgfs is not loaded. This should not happen, and if it does, then it's a bug in the dnf installer.

The error message says that the stat system call failed, so the mkdir command is probably checking whether the mount point already exists prior to creating it. The error code corresponds to EIO, which is an input/out error. That error is usually from a hardware issue, eg. something like a bad disk block or a faulty disk controller, though it's also possible that if things were in state (3) above, something in the hgfs file system stack would be broken.

For both (1) and (2), you can check whether you see the same EIO error by running "sudo mkdir /mnt/hgfs". In both cases the expected behavior is that mkdir reports the directory already exists with no message about an input/output error.

I'm not sure how state (1) could generate the stat EIO error, since if hgfs is up and running, then the stat syscall should succeed and report the directory already exists. I think you've already confirmed that by running the stat command, but the command that is failing is "mkdir", so you should consider trying that. Running mkdir should be non-destructive if the directory is already present.

If there is an underlying disk error in the root file system, then state (2) might report stat failing with EIO. To check that, you'd need to first unmount hgfs with "sudo umount /mnt/hgfs". I think that is a potential scenario, eg. during the update hgfs was unmounted and stopped, and the installer was (re)creating the mount point. If that's the case, then then problem could be hidden while hgfs is mounted, because when it's running, as soon as the file system code hits the mount point, it transfers control to hgfs, so you won't see the underlying error in the root file system. If you check this, be sure to remount hgfs after you're done

If checking state (2) generates an EIO error, that might indicate a hardware issue with the root drive. Since the error is observed inside a VM, that would suggest an issue with the vmdk file on the host. Running a disk verification utility inside the VM
might flush that out, or you could try running a file system check on the Windows host.

State (3) should not happen, but there could be an installer bug in the upgrade path. There's no simple way to recreate that scenario, but if (1) and (2) both return that the directory exists and don't generate the stat EIO error, then it points to an installer bug as being the problem. If you want to check for that, I think the best you can do is to completely remove open-vm-tools, make sure the mount point doesn't exist, and then reinstall open-vm-tools. If there's no stat error, then it's probably a bug in the upgrade path where it's not unmounting /mnt/hgfs before it stops the hgfs service. That's not a serious issue, since hgfs is not expected to work while the upgrade is being done, and you've already confirmed that stat and hgfs are working after the upgrade completes.

Hope this helps explain it.

[johnwvmw]

The Fedora packaging of open-vm-tools has not changed much over the past year. I believe that there was a change in the 11.3.0 packaging to improve the conditional around the "mkdir -p /mnt/hgfs" in the post install scriptlet.

Did you experience any problems using the Shared Folders following the update?

I will try to reproduce the problem internally.

[hermidalc]

The Fedora packaging of open-vm-tools has not changed much over the past year. I believe that there was a change in the 11.3.0 packaging to improve the conditional around the "mkdir -p /mnt/hgfs" in the post install scriptlet.

Did you experience any problems using the Shared Folders following the update?

I will try to reproduce the problem internally.

It actually didn't do anything negative. My shared folder mount already existed was mounted before the dnf update of open-vm-tools and nothing bad happened during the upgrade. Not sure why cannot stat error because after the post-install scriptlet I immediately stat-ed it and it was fine. Just wanted to notify you in this issue because never had seen error before.

[johnwvmw]

@hermidalc Thanks for the update.

[hermidalc]

On the next open-vm-tools update if I see this go away I'll close this issue

[hermidalc]

This has gone away and is fixed now in version 12.1.0 and I believe versions even before that, sorry for forgetting about the open issue.