[Bug] Kerberos FileSystem wrapper silently disables atomic rename for metadata writes

[LuciferYang]

Search before asking

• I searched in the issues and found nothing similar.

Paimon version

master, 475be566f (2.1-SNAPSHOT).

Compute Engine

Any engine on a Kerberos-secured HDFS catalog, that is with security.kerberos.login.keytab and security.kerberos.login.principal set.

Minimal reproduce step

Write any file that Paimon commits by atomic rename on such a cluster: a snapshot EARLIEST/LATEST hint, a consumer reset, TagManager.createOrReplaceTag, _SUCCESS, or a service address file. All of them go through FileIO.overwriteHintFile style code that first tries:

public boolean tryAtomicOverwriteViaRename(Path dst, String content) throws IOException {
    org.apache.hadoop.fs.Path hadoopDst = path(dst);
    FileSystem fs = getFileSystem(hadoopDst);
    ...
    method = ReflectionUtils.getMethod(fs.getClass(), "rename", 3);

With Kerberos configured, fs is a HadoopSecuredFileSystem. ReflectionUtils.getMethod uses clz.getMethods(), which only returns public methods, and the wrapper overrides just the two-argument rename; FileSystem's three-argument rename(Path, Path, Options.Rename...) is protected. The lookup throws NoSuchMethodException, the method reference is cached as null, and this method always returns false.

The caller then falls back to newOutputStream(path, true), an in-place overwrite. Nothing reports it. So on a secured cluster the hint and tag files are written non-atomically: a reader can observe an empty or half-written file, and a failed write leaves it truncated.

A unit-level version: wrap a local FileSystem that exposes a public three-argument rename in HadoopSecuredFileSystem, install it with HadoopFileIO.setFileSystem, and call tryAtomicOverwriteViaRename. It returns false, and the delegate's atomic rename is never called.

What doesn't meet your expectations?

Configuring Kerberos should not silently turn atomic metadata commits into non-atomic overwrites. The wrapper is transparent for everything else, and this one call reaches for a method the wrapper cannot expose.

Anything else?

Whatever reaches past the wrapper has to keep the identity: every delegating method in HadoopSecuredFileSystem runs inside ugi.doAs, so a rename invoked directly on the unwrapped file system would run as whatever the calling thread is, while the temporary file it renames was created as the login user.

Are you willing to submit a PR?

• I'm willing to submit a PR!