Add gvfs as a storage scheme in Hudi

[architlatkar27]

I am working on a use case where we would like to enable Hudi tables to work with Apache Gravitino's "gvfs" filesystem.
Currently when we try to run spark jobs with gravitino over Hudi we face the following error

Caused by: java.lang.IllegalArgumentException: Unsupported scheme :gvfs at org.apache.hudi.storage.StorageSchemes.isWriteTransactional(StorageSchemes.java:144)

The solution would be to add gvfs as a storage scheme under StorageSchemes

[danny0405]

Supporting GVFS sounds reasonable, but I think we should validate a few storage semantics before treating an entry in StorageSchemes as sufficient.

GVFS can resolve to different underlying filesystems, including HDFS and S3-compatible storage. StorageSchemes also determines transactional-write and atomic-create capabilities, so could you clarify which backing storage you are using and what values you propose for those flags? A single set of flags needs to be justified across the supported backends.

There is also a separate scheme-dependent path in HoodieStorage.needCreateTempFile(): it currently enables temporary-file-then-rename publication only for hdfs, viewfs, and file. Adding gvfs to the enum alone would leave this returning false even for an HDFS-backed fileset. We should verify how partially written immutable/timeline files are prevented from becoming visible in that case.

It would help to include integration coverage for GVFS over both HDFS and S3-compatible storage (MinIO would be useful for the latter), covering commits, failed-write recovery/rollback, and Merge-on-Read updates if included in the supported scope. Please also document any locking or concurrency limitations.

For context, MinIO does not need its own scheme entry: Hudi accesses it through the existing s3a scheme with a configured endpoint. GVFS exposes a distinct gvfs scheme, which is why it encounters this check.

[architlatkar27]

Hi @danny0405 thanks for your inputs here

I went through some of the code myself and appreciate the complexity with gvfs. Let me try to answer some of the questions you have posed.

could you clarify which backing storage you are using and what values you propose for those flags?

We were using gvfs on top of hdfs filesystem. We are also planning to register and consume S3, Azure, Gcs catalogs as filesets in the future and we may need some solution there as well.
In a patch that i tried out i had set GVFS("gvfs", null, null, null) and it had worked for us. But admittedly we are trying out some simple stuff as of now so it might not cause much issues. I understand that this might not exactly be the right configuration for S3, ABFS etc.
The complexity also comes due to S3, Abfs, Gcs and Hdfs using different configurations.
I kept everything null above since that felt like the lowest common denominator.

We should verify how partially written immutable/timeline files are prevented from becoming visible in that case.

I will try to reproduce something like this. Do we have integration tests that can cover such a scenario?