Apache Oozie job failing with the hotpatch

[aajisaka]

Apache Oozie job intermittently fails after applying the hotpatch. This is because Oozie will list and the print the file and directory name in the current directory. On the other hand, the hotpatch creates attach_pid file in the current directory to attach the JVM and then delete it. When the attach_pid file is deleted between the listing and the printing, the Oozie job fails by NoSuchFileException. The below is the stacktrace:

java.nio.file.NoSuchFileException: ./.attach_pid1517
    at sun.nio.fs.UnixException.translateToIOException(UnixException.java:86)
    at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
    at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)
    at sun.nio.fs.UnixFileAttributeViews$Basic.readAttributes(UnixFileAttributeViews.java:55)
    at sun.nio.fs.UnixFileSystemProvider.readAttributes(UnixFileSystemProvider.java:144)
    at sun.nio.fs.LinuxFileSystemProvider.readAttributes(LinuxFileSystemProvider.java:99)
    at java.nio.file.Files.readAttributes(Files.java:1737)
    at java.nio.file.FileTreeWalker.getAttributes(FileTreeWalker.java:225)
    at java.nio.file.FileTreeWalker.visit(FileTreeWalker.java:276)
    at java.nio.file.FileTreeWalker.next(FileTreeWalker.java:372)
    at java.nio.file.Files.walkFileTree(Files.java:2706)
    at org.apache.oozie.action.hadoop.LocalFsOperations.printContentsOfDir(LocalFsOperations.java:59)
    at org.apache.oozie.action.hadoop.LauncherAM.printDebugInfo(LauncherAM.java:293)
    at org.apache.oozie.action.hadoop.LauncherAM.access$100(LauncherAM.java:55)
    at org.apache.oozie.action.hadoop.LauncherAM$2.run(LauncherAM.java:221)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:422)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1844)
    at org.apache.oozie.action.hadoop.LauncherAM.run(LauncherAM.java:217)
    at org.apache.oozie.action.hadoop.LauncherAM$1.run(LauncherAM.java:153)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:422)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1844)
    at org.apache.oozie.action.hadoop.LauncherAM.main(LauncherAM.java:141)

The mitigation is to disable the hotpatch on the Oozie server.

I plan to provide a fix to Oozie to retry the directory listing, however, it is worth creating the issue here for searchability. Thanks.

[lutkerd]

Thanks for bringing this to our attention and sorry for the delayed response.

The JVM creates the .attach_pidXXXX file in the current working directory for the target JVM process by default, see LinuxVirtualMachine.java. The file should only exist while the hotpatching process is connected and be cleaned up automatically. Unfortunately there isn't much we can do about the creation of that file.

If your service does not contain the log4j vulnerability, you can disable the hotpatcher by creating a /dev/shm/log4j-cve-2021-44228-hotpatch.kill file.

[aajisaka]

The JVM creates the .attach_pidXXXX file in the current working directory for the target JVM process by default, see LinuxVirtualMachine.java. The file should only exist while the hotpatching process is connected and be cleaned up automatically. Unfortunately there isn't much we can do about the creation of that file.

Agreed. That's why I plan to provide a fix to Oozie.

[aajisaka]

Filed https://issues.apache.org/jira/browse/OOZIE-3652 and attached a patch there.