caching resource has race condition in case of multiple JVMs

[mythly]

extractResource didn't consider the race condition, so the target resource file like ~/.javacpp/cache/opencv-3.2.0-1.3-linux-x86_64.jar/org/bytedeco/javacpp/linux-x86_64/libjniopencv_core.so could be corrupted when multiple JVMs try to write the same file.

One can easily reproduce this by

rm -rf ~/.javacpp/cache/*
java <some jar using javacpp> & java <some jar using javacpp>

and see error likes

Caused by: java.lang.UnsatisfiedLinkError: /home/somebody/.javacpp/cache/something.jar/org/bytedeco/javacpp/linux-x86_64/libjniopencv_core.so: /home/somebody/.javacpp/cache/something.jar/org/bytedeco/javacpp/linux-x86_64/libjniopencv_core.so: ELF load command past end of file

or

Caused by: java.lang.UnsatisfiedLinkError: /home/somebody/.javacpp/cache/something.jar/org/bytedeco/javacpp/linux-x86_64/libjniopencv_core.so: /home/somebody/.javacpp/cache/something.jar/org/bytedeco/javacpp/linux-x86_64/libopencv_core.so.3.2: undefined symbol: _ZN2cv9AlgorithmD1Ev

Possible (but not good) solutions:

1. changing "org.bytedeco.javacpp.cachedir" of different JVMs to different folders
2. preparing all the cache before the first running of the application

Any better ideas?

[saudet]

1. It used to work that way by default, and we can still make it work like that by setting the "org.bytedeco.javacpp.cachedir" system property to whatever we want:
   http://bytedeco.org/javacpp/apidocs/org/bytedeco/javacpp/Loader.html#getCacheDir--
2. I'm already doing something like that by waiting that files are at least 1 second old:
   https://github.com/bytedeco/javacpp/blob/master/src/main/java/org/bytedeco/javacpp/Loader.java#L426
   If that isn't working anymore, it would be the first thing to investigate!

[mythly]

"waiting that files are at least 1 second old" seems not working as both JVMs fall in the earlier if branch.

The test code

import org.bytedeco.javacpp.opencv_core;

public class CacheTest {

    public static void main(String args[]) throws Exception {
        System.setProperty("org.bytedeco.javacpp.logger.debug", "true");
        opencv_core.Mat mat = new opencv_core.Mat(200, 100, opencv_core.CV_8UC3);
        System.out.printf("%d %d\n", mat.cols(), mat.rows());
    }
}

The output of JVM1

Loading class org.bytedeco.javacpp.opencv_core
Loading class org.bytedeco.javacpp.opencv_core
Loading /home/motao/.javacpp/cache/play-ground-assembly-1.0.0.jar/org/bytedeco/javacpp/linux-x86_64/libgomp.so.1
Extracting jar:file:/home/motao/play-ground/target/scala-2.11/play-ground-assembly-1.0.0.jar!/org/bytedeco/javacpp/linux-x86_64/libopencv_core.so.3.2
Loading /home/motao/.javacpp/cache/play-ground-assembly-1.0.0.jar/org/bytedeco/javacpp/linux-x86_64/libopencv_core.so.3.2
Extracting jar:file:/home/motao/play-ground/target/scala-2.11/play-ground-assembly-1.0.0.jar!/org/bytedeco/javacpp/linux-x86_64/libopencv_imgproc.so.3.2
Loading /home/motao/.javacpp/cache/play-ground-assembly-1.0.0.jar/org/bytedeco/javacpp/linux-x86_64/libopencv_imgproc.so.3.2
Extracting jar:file:/home/motao/play-ground/target/scala-2.11/play-ground-assembly-1.0.0.jar!/org/bytedeco/javacpp/linux-x86_64/libjniopencv_core.so
Loading /home/motao/.javacpp/cache/play-ground-assembly-1.0.0.jar/org/bytedeco/javacpp/linux-x86_64/libjniopencv_core.so
Loading class org.bytedeco.javacpp.opencv_core
Creating symbolic link to /home/motao/.javacpp/cache/play-ground-assembly-1.0.0.jar/org/bytedeco/javacpp/linux-x86_64/libgomp.so.1
Creating symbolic link to /home/motao/.javacpp/cache/play-ground-assembly-1.0.0.jar/org/bytedeco/javacpp/linux-x86_64/libopencv_core.so.3.2
Creating symbolic link to /home/motao/.javacpp/cache/play-ground-assembly-1.0.0.jar/org/bytedeco/javacpp/linux-x86_64/libopencv_imgproc.so.3.2
Creating symbolic link to /home/motao/.javacpp/cache/play-ground-assembly-1.0.0.jar/org/bytedeco/javacpp/linux-x86_64/libjniopencv_core.so
Registering org.bytedeco.javacpp.opencv_core$CvSlice[address=0x7fcf84185ea0,position=0,limit=1,capacity=1,deallocator=org.bytedeco.javacpp.Pointer$NativeDeallocator[ownerAddress=0x7fcf84185ea0,deallocatorAddress=0x7fcf4c670b60]]
Registering org.bytedeco.javacpp.opencv_core$CvSlice[address=0x7fcf841da6b0,position=0,limit=1,capacity=1,deallocator=org.bytedeco.javacpp.Pointer$NativeDeallocator[ownerAddress=0x7fcf841da6b0,deallocatorAddress=0x7fcf4c670b60]]
Loading class org.bytedeco.javacpp.opencv_core
Creating symbolic link to /home/motao/.javacpp/cache/play-ground-assembly-1.0.0.jar/org/bytedeco/javacpp/linux-x86_64/libgomp.so.1
Creating symbolic link to /home/motao/.javacpp/cache/play-ground-assembly-1.0.0.jar/org/bytedeco/javacpp/linux-x86_64/libopencv_core.so.3.2
Creating symbolic link to /home/motao/.javacpp/cache/play-ground-assembly-1.0.0.jar/org/bytedeco/javacpp/linux-x86_64/libopencv_imgproc.so.3.2
Creating symbolic link to /home/motao/.javacpp/cache/play-ground-assembly-1.0.0.jar/org/bytedeco/javacpp/linux-x86_64/libjniopencv_core.so
Registering org.bytedeco.javacpp.opencv_core$Mat[width=100,height=200,depth=8,channels=3]
100 200

The output of JVM2

Loading class org.bytedeco.javacpp.opencv_core
Loading class org.bytedeco.javacpp.opencv_core
Extracting jar:file:/home/motao/play-ground/target/scala-2.11/play-ground-assembly-1.0.0.jar!/org/bytedeco/javacpp/linux-x86_64/libgomp.so.1
Loading /home/motao/.javacpp/cache/play-ground-assembly-1.0.0.jar/org/bytedeco/javacpp/linux-x86_64/libgomp.so.1
Extracting jar:file:/home/motao/play-ground/target/scala-2.11/play-ground-assembly-1.0.0.jar!/org/bytedeco/javacpp/linux-x86_64/libopencv_core.so.3.2
Loading /home/motao/.javacpp/cache/play-ground-assembly-1.0.0.jar/org/bytedeco/javacpp/linux-x86_64/libopencv_core.so.3.2
#
# A fatal error has been detected by the Java Runtime Environment:
#
#  SIGBUS (0x7) at pc=0x00007f7f8130e9ba, pid=15576, tid=140185607010048
#
# JRE version: Java(TM) SE Runtime Environment (8.0_91-b14) (build 1.8.0_91-b14)
# Java VM: Java HotSpot(TM) 64-Bit Server VM (25.91-b14 mixed mode linux-amd64 compressed oops)
# Problematic frame:
# C  [ld-linux-x86-64.so.2+0x1a9ba]
#
# Failed to write core dump. Core dumps have been disabled. To enable core dumping, try "ulimit -c unlimited" before starting Java again
#
# An error report file with more information is saved as:
# /home/motao/play-ground/hs_err_pid15576.log
#
# If you would like to submit a bug report, please visit:
#   http://bugreport.java.com/bugreport/crash.jsp
# The crash happened outside the Java Virtual Machine in native code.
# See problematic frame for where to report the bug.
#

You can found that both JVMs try to write libopencv_core.so.3.2

Extracting jar:file:/home/motao/play-ground/target/scala-2.11/play-ground-assembly-1.0.0.jar!/org/bytedeco/javacpp/linux-x86_64/libopencv_core.so.3.2

[saudet]

I see, so we'd have to move that check before calling extractResources()...

[saudet]

Actually, that won't help much. I think this is the answer:
https://docs.oracle.com/javase/7/docs/api/java/io/File.html#createNewFile()

[thumlx12]

So would you mind detailing how File.createNewFile() can be of help to solve this problem?

[saudet]

Not sure yet, but we need something like this.

[mythly]

It helps if we lock L397 to L435 of Loader.java using something like FileLock, and the required lock file could be put under getCacheDir().

[saudet]

Good idea! I was under the impression that we wouldn't have the required API in Android for something like that, but it appears we do: https://developer.android.com/reference/java/nio/channels/FileLock.html

Could you send a PR for that?

[mythly]

close as PR was merged