Concurrent Map iteration and map write

[rpemsel]

Describe the issue
A user of our platform that uses GCS FUSE via GCS FUSE CSI Driver reported an issue with the GCS Fuse part of the implementation. The sidecar container running GCS Fuse broke down with an error message of "Concurrent Map iteration and map write". Also see the following logs
downloaded-logs-20240219-062431.json.

According to the logs it looks a bit like an issue with finding a census for concurrent read/write processes. Unfortunately this issue cannot easily be reproduced.

System (please complete the following information):

• OS: COS
• Platform Kubernetes via GCS FUSE CSI Driver
• Version: 0.1.12 (GCS Fuse CSI Driver)

Additional context
The issue occurred during high load read/write operations

SLO:
We strive to respond to all bug reports within 24 business hours.

[sethiay]

Thanks @rpemsel for reaching out.

We got similar issue recently and this happens when listing and creation of new files are done concurrently on the same mount.

While we fix this as soon as possible, we suggest you to avoid doing listing and creation of new files at the time time if possible.

Thanks for your patience !

[ashmeenkaur]

Hi @rpemsel,

Thanks for reporting this issue! The fix has been merged into master and will be included in the March 2024 release.

Thanks,
Ashmeen

[Tulsishah]

This issue is resolved in GCSFuse v2.0.0. Please upgrade and reopen the issue if necessary.

Thanks,
Tulsi Shah

[tred77]

Thanks for the fix,
Is there any page mapping with gcsfuse/driver is used in which GKE release channel version?
Could not find any documentation not in GKE side nor gcsfuse/Driver side

[sethiay]

@tred77 Please take a look at https://github.com/GoogleCloudPlatform/gcs-fuse-csi-driver/blob/main/docs/releases.md#gke-compatibility

[tred77]

@sethiay I have one more question
Since in my case the new version is not available in the gke stable channel, I am not able to upgrade it. Meanwhile I was thinking to implement a health check using kubernetees livenessProbe and readinessProbe. I was wondering if there is better way to make sure that the mount is live than using ls command. Because, behind the scene ls command would be translated to list which is already part of the problem and I don't want to make my system more flaky by adding extra list operation.

[sethiay]

@tred77 Could you please bring this on https://github.com/GoogleCloudPlatform/gcs-fuse-csi-driver/ repository ? Thanks !