CloudStorageFileSystem: ConcurrentModificationException belies thread safety

[cjllanwarne]

We recently updated from an older version of the storage NIO library and are now seeing ConcurrentModificationExceptions from CloudStorageFileSystem. At a guess, it seems like this function is perhaps not perfectly resilient if CONFIG_TO_PROVIDERS_MAP is updated while the existingProviders value is being iterated over?

Caused by: java.util.ConcurrentModificationException: null
	at java.base/java.util.HashMap$HashIterator.nextNode(Unknown Source)
	at java.base/java.util.HashMap$KeyIterator.next(Unknown Source)
	at com.google.cloud.storage.contrib.nio.CloudStorageFileSystem.getCloudStorageFileSystemProvider(CloudStorageFileSystem.java:164)
	at com.google.cloud.storage.contrib.nio.CloudStorageFileSystem.forBucket(CloudStorageFileSystem.java:196)
	at [...our client code]

[breilly2]

java-storage-nio/google-cloud-nio/src/main/java/com/google/cloud/storage/contrib/nio/CloudStorageFileSystem.java

Line 154 in 7f6f65d

private static CloudStorageFileSystemProvider getCloudStorageFileSystemProvider(

Looks very problematic:

  private static CloudStorageFileSystemProvider getCloudStorageFileSystemProvider(
      CloudStorageConfiguration config, StorageOptions storageOptions) {
    CloudStorageFileSystemProvider newProvider =
        (storageOptions == null)
            ? new CloudStorageFileSystemProvider(config.userProject())
            : new CloudStorageFileSystemProvider(config.userProject(), storageOptions);
    Set<CloudStorageFileSystemProvider> existingProviders = CONFIG_TO_PROVIDERS_MAP.get(config);
    if (existingProviders == null) {
      existingProviders = new HashSet<>();
    } else {
      for (CloudStorageFileSystemProvider existiningProvider : existingProviders) {
        if (existiningProvider.equals(newProvider)) {
          return existiningProvider;
        }
      }
    }
    existingProviders.add(newProvider);
    CONFIG_TO_PROVIDERS_MAP.put(config, existingProviders);
    return newProvider;
  }

1. CloudStorageFileSystem is annotated as @ThreadSafe. Perhaps its instances are, but this static function is not. existingProviders is a HashSet which gets modified near the end of the function without any synchronization. Concurrent callers with the same config and storageOptions can run into this ConcurrentModificationException.
2. The values of existingProviders are CloudStorageFileSystemProvider instances (which are oddly annotated as @Singleton). CloudStorageFileSystemProvider's equals and hashCode are based on the storage property which is modified after object instantiation (from initStorage called from various places). Therefore, once a "cached" instance has been initialized, existingProvider.equals(newProvider) will yield false (since newProvider hasn't yet been initialized).

[JesseLovelace]

Thanks for the feedback, we're taking a look at this. How often does this issue come up for you?

[aednichols]

The issue reproduces more or less continuously when our multitenant server app is in production with the affected release.

We also created a unit test that triggers it in a minimal scenario.

[JesseLovelace]

Thanks! We'll have an update on this tomorrow

[aednichols]

Sounds great, thank you!

[BenWhitehead]

Thank you very much for providing the minimised repo test that helped a lot.

I've proposed a fix in #719 which fixes the thread safety issue, and fixes the actual caching mapping outlined in #698 as well.