[PATCH] Hide non-JNI symbols from native_ref-linux libs

[kkofler]

Add an ld version script to the GNU/Linux native_ref JNI libraries,
hiding all the symbols except for the Java_* JNI symbols. This matches
the other platforms, which only export explicitly exported symbols.

GNU/Linux defaults to exporting all symbols. For C and C++ code, this
can be overridden by compiling it with the -fvisibility=hidden flag, but
gfortran does not support that flag, so it has to be done at link time
through the --version-script linker flag. Without the symbol.map version
script, all the symbols of the bundled reference BLAS, LAPACK and ARPACK
(and also some internal functions of the JNI wrapper itself) were being
exported, causing symbol conflicts.

One example of such a symbol conflict is MATLAB: Loading JAVA code using
netlib-java into MATLAB and using the native_ref implementation would
crash MATLAB, due to symbol conflicts between MATLAB's versions of BLAS
and LAPACK and the native_ref ones. (This was found during testing at
the University of Vienna.)

I uploaded the patch exported from git to:
http://repo.calcforge.org/temp/0001-Hide-non-JNI-symbols-from-native_ref-linux-libs.patch
(because I cannot attach non-image files directly).

[fommil]

@kkofler thanks! I'll make sure this makes the 1.1 release. If you could send pull requests, instead of patches, in the future that would really help clean up my workflow. Github is great!

[fommil]

@kkofler I can restrict the symbols on Linux and OS X, but if you ever want this for Windows too then you're going to have to create a .def file containing an explicit list of all the Java_ symbols.

I've uploaded the corrected binaries for all systems. Please try them and let me know if they work for you. I'm planning on making the 1.1 release this weekend.

[kkofler]

but if you ever want this for Windows too then you're going to have to create a .def file containing an explicit
list of all the Java_ symbols.

The symbols are already restricted there. (Just check your DLLs in any export viewer. Do not trust mingw32-nm: That one shows debugging symbols, not exported symbols!) The DLL system defaults to exporting only the symbols explicitly marked dllexport in the code, which is exactly what you want here.

I also do not understand the changes you made to stripping. They may make sense on OS X (not sure), but on GNU/Linux, stripping is entirely unrelated to symbol exporting, it does not touch the dynamic symbol table (the table of exported symbols).You should just use -s on GNU/Linux. GNU ld will not strip the dynamic symbol table with -s. Your -Wl,-xS produces a larger shared object, because it keeps a second copy of the exported symbols in the debugging symbol table. That makes no sense, because GDB can process the dynamic symbol table just fine. Use nm -D to view the dynamic symbol table, the one that matters for symbol exports. As I said in the previous paragraph, Windows/MinGW also has 2 symbol tables, and there too, you should use -s when linking to strip the debugging symbols. However, unfortunately, the MinGW nm does not support -D. (It does not show anything, but the DLL is actually exporting symbols.) So you cannot trust the MinGW nm for that either.

[kkofler]

PS: MinGW does provide a working tool to view the export table of a DLL (the equivalent of the dynamic symbol table of the GNU/Linux ELF format), it's called pexports:

http://www.mirrorservice.org/sites/downloads.sourceforge.net/m/mi/mingw/MinGW/Extension/pexports/

Unfortunately, that tool is not part of the standard MinGW or cross-MinGW distributions.

[fommil]

@kkofler if I do -s when stripping, then nm -a claims there are no symbols at all, so I do not wish to strip everything, I only want to strip non-global -x and debugging -S. However, ld seems to ignore this and I have no way in this workflow to run strip afterwards: http://stackoverflow.com/questions/19289901

I'm running another compile with flags to remove unused code. I'll upload tonight.

Re: Windows, see here http://sourceforge.net/p/mingw/bugs/1134/ specifically

--version-script=version-scriptfile
           Specify the name of a version script to the linker.  This is typically used when creating shared libraries to specify additional information about the version
           hierarchy for the library being created.  This option is only fully supported on ELF platforms which support shared libraries

so a .DEF file is required. I have no plans to build or ship such files.

[fommil]

I'm getting some assertion errors during the Raspberry Pi compile. I need to check that all the tests pass at the weekend before feeling comfortable enough to release this.

[kkofler]

I've uploaded the corrected binaries for all systems. Please try them and let me know if they work for you.

We tried them at the university today (the snapshot that was current when you posted that comment), and we can confirm that they work fine. In particular, the native_ref .so no longer crashes MATLAB.

(For what it's worth, the native_system library still triggers the same fatal symbol conflict, but we cannot fix that case in the same way, because the system BLAS/LAPACK libraries obviously have to export the symbols or they wouldn't work at all. It is really MATLAB's fault, I cannot think of any sane way to work around that right now. But at least native_ref works now.)

if I do -s when stripping, then nm -a claims there are no symbols at all

nm -a does not show the dynamic symbol table / export table, you really have to use nm -D for GNU/Linux or pexports for MinGW. OS X is actually the odd one out in this context, because (as I understand it) it uses only one symbol table where the other operating systems use 2 separate ones.

[kkofler]

Re: Windows, see here http://sourceforge.net/p/mingw/bugs/1134/ specifically
[snip]
so a .DEF file is required. I have no plans to build or ship such files.

A .DEF file is required if you want anything other than the default behavior. But on Windows, exporting only the Java_* functions is already the default behavior, so you don't actually have to do anything. (Try running pexports or something equivalent on the DLL if you don't believe me. I checked the export table in a hex viewer.)

More precisely, MinGW (like Visual C++, whose behavior it emulates in this context) only exports functions marked as __declspec(dllexport) (or the GCC syntax __attribute__((dllexport))). This is exactly what the JNIEXPORT macro in include/win32/jni_md.h expands to. Any function not marked dllexport is not exported from the DLL. This is unlike the behavior on other platforms, where symbols are exported by default.

See:

• http://msdn.microsoft.com/en-us/library/vstudio/3y1sfaz2.aspx
• http://gcc.gnu.org/onlinedocs/gcc-4.8.1/gcc/Function-Attributes.html#index-g_t_0040code_007b_005f_005fdeclspec_0028dllexport_0029_007d-2597

So you can safely remove that TODO comment. Your code is already doing the right thing on Windows.

[fommil]

well, I'm doing the symbol table lookup on windows just to be on the safe side.

The native system libs you tried today weren't actually the new ones.

I need to check the ARM bins before I can release: the dead code removal was causing assertion errors and I want to make sure the assertion errors were just overzealous warnings and not that my binaries are borked.