crash on linux caused by loading wrong version of native libs

[ja2142]

I've tried to play back some audio based on AudioPlayback example, and on linux (manjaro, though I doubt that matters) I got crash in libavutil:

[main] INFO velvet-video - Loading velvet-video-natives version 0.2.8.full
[main] DEBUG velvet-video - Velvet-video native extraction location is /home/<user>/.velvet-video/natives/0.2.8.full
[main] DEBUG velvet-video - Requesting loading native lib avutil.56
[main] DEBUG velvet-video - Checking native lib libavutil.so.56 at /home/<user>/.velvet-video/natives/0.2.8.full
[main] DEBUG velvet-video - Loaded libavutil.so.56
[main] DEBUG velvet-video - Requesting loading native lib swresample.3
[main] DEBUG velvet-video - Checking native lib libswresample.so.3 at /home/<user>/.velvet-video/natives/0.2.8.full
[main] DEBUG velvet-video - Loaded libswresample.so.3
[main] DEBUG velvet-video - Preloading native lib dependendcy: /home/<user>/.velvet-video/natives/0.2.8.full/libopenh264.so.5
[main] DEBUG velvet-video - Requesting loading native lib avcodec.58
[main] DEBUG velvet-video - Checking native lib libavcodec.so.58 at /home/<user>/.velvet-video/natives/0.2.8.full
[main] DEBUG velvet-video - Loaded libavcodec.so.58
[main] DEBUG velvet-video - Requesting loading native lib avformat.58
[main] DEBUG velvet-video - Checking native lib libavformat.so.58 at /home/<user>/.velvet-video/natives/0.2.8.full
[main] DEBUG velvet-video - Loaded libavformat.so.58
#
# A fatal error has been detected by the Java Runtime Environment:
#
#  SIGSEGV (0xb) at pc=0x00007f8aa2e7dc77, pid=55483, tid=55484
#
# JRE version: OpenJDK Runtime Environment (17.0.3+3) (build 17.0.3+3)
# Java VM: OpenJDK 64-Bit Server VM (17.0.3+3, mixed mode, tiered, compressed oops, compressed class ptrs, g1 gc, linux-amd64)
# Problematic frame:
# C  [libavutil.so.57.17.100+0x39c77]  av_opt_child_next+0x7
#
# Core dump will be written. Default location: Core dumps may be processed with "/usr/lib/systemd/systemd-coredump %P %u %g %s %t %c %h" (or dumping to /home/<user>/Documents/src/opus_decode_test/core.55483)
#
# An error report file with more information is saved as:
# /home/<user>/Documents/src/opus_decode_test/hs_err_pid55483.log
#
# If you would like to submit a bug report, please visit:
#   https://bugreport.java.com/bugreport/crash.jsp
# The crash happened outside the Java Virtual Machine in native code.
# See problematic frame for where to report the bug.
#
[1]    55483 IOT instruction (core dumped)

It seems like even though logs at the beginning say that libavutil.so.56 was loaded (from ~/.velvet-video), crash occured in libavutil.so.57.17.100. Looking at error report (hs_err_pid55483.log), it turns out that libavutil from my system was loaded: velvet is using /usr/lib/libavutil.so.57.17.100, /usr/lib/libswresample.so.4.3.100, /usr/lib/libavcodec.so.59.18.100 and /usr/lib/libavformat.so.59.16.100. Only lib that was actually loaded from ~/.velvet-video is libopenh264.so.5, which I don't have installed.

A weird thing from the error report is the fact that it seems like java tried to load the libs from ~/.velvet-video, but failed? Here are relevant lines:

Event: 0,606 Thread 0x00007f845c012630 Exception <a 'java/lang/UnsatisfiedLinkError'{0x0000000716640630}: avutil: cannot open shared object file: No such file or directory> (0x0000000716640630) 
thrown [src/hotspot/share/prims/jni.cpp, line 535]
Event: 0,672 Thread 0x00007f845c012630 Exception <a 'java/lang/NoSuchMethodError'{0x0000000716bd6d58}: 'java.lang.Object java.lang.invoke.DirectMethodHandle$Holder.newInvokeSpecial(java.lang.Object, java.lang.Object, java.lang.Object, java.lang.Object, java.lang.Object)'> (0x0000000716bd6d58) 
thrown [src/hotspot/share/interpreter/linkResolver.cpp, line 765]
Event: 0,741 Thread 0x00007f845c012630 Exception <a 'java/lang/UnsatisfiedLinkError'{0x00000007169ecc28}: /usr/lib64/libavutil.so.57.17.100: undefined symbol: av_frame_get_pkt_duration> (0x00000007169ecc28) 
thrown [src/hotspot/share/prims/jni.cpp, line 535]
Event: 0,754 Thread 0x00007f845c012630 Exception <a 'java/lang/UnsatisfiedLinkError'{0x0000000716674630}: swresample: cannot open shared object file: No such file or directory> (0x0000000716674630) 
thrown [src/hotspot/share/prims/jni.cpp, line 535]
Event: 0,795 Thread 0x00007f845c012630 Exception <a 'java/lang/UnsatisfiedLinkError'{0x0000000716b9a240}: avcodec: cannot open shared object file: No such file or directory> (0x0000000716b9a240) 
thrown [src/hotspot/share/prims/jni.cpp, line 535]
Event: 0,840 Thread 0x00007f845c012630 Exception <a 'java/lang/UnsatisfiedLinkError'{0x0000000715f04280}: /usr/lib64/libavcodec.so.59.18.100: undefined symbol: avcodec_open2x> (0x0000000715f04280) 
thrown [src/hotspot/share/prims/jni.cpp, line 535]
Event: 0,842 Thread 0x00007f845c012630 Exception <a 'java/lang/UnsatisfiedLinkError'{0x0000000715f21058}: /usr/lib64/libavcodec.so.59.18.100: undefined symbol: avcodec_register_all> (0x0000000715f21058) 
thrown [src/hotspot/share/prims/jni.cpp, line 535]
Event: 0,843 Thread 0x00007f845c012630 Exception <a 'java/lang/UnsatisfiedLinkError'{0x0000000715f2ad50}: /usr/lib64/libavcodec.so.59.18.100: undefined symbol: avcodec_encode_video2> (0x0000000715f2ad50) 
thrown [src/hotspot/share/prims/jni.cpp, line 535]
Event: 0,857 Thread 0x00007f845c012630 Exception <a 'java/lang/UnsatisfiedLinkError'{0x0000000715fb8e10}: avformat: cannot open shared object file: No such file or directory> (0x0000000715fb8e10) 
thrown [src/hotspot/share/prims/jni.cpp, line 535]

I'd look into it some more, but at the moment I have pretty much no idea how java is handling native libraries.

[ja2142]

I took a closer look, and discovered three things:

• On linux even though velvet-video is saying it's loading specific version of library, the version isn't really specified (for example Requesting loading native lib avutil.56 -> any version avutil is being loaded). Because major version might differ, there can be binary incompatibility - and that's exactly what's happening for me, resulting in a crash. That's something that can be easily fixed in velvet-video.
• With a fix mentioned above (loading libraries with their versions - e.g. loading libavutil.so.56 isntead of just avutil) libs with correct major version are loaded. However because I also have libavutil.so.56 installed on my system, that results in a crash just like above. Why that happens I don't understand, I'd expect for any two versions with the same major version to be binary compatible, so maybe there's ffmpeg's fault?
• With a versioning fix, and with LD_LIBRARY_PATH set to ~/.velvet-video/natives/0.2.8.full the example is finally working. Why is modyfing environment variables necessary is unclear to me, I've raised an issue in jnr-ffi (LibraryLoader.search() path not taken into consideration unless also in LD_LIBRARY_PATH jnr/jnr-ffi#303).