Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Please upgrade libffi to ≥3.4.2 #1381

Closed
DaveCTurner opened this issue Sep 21, 2021 · 10 comments
Closed

Please upgrade libffi to ≥3.4.2 #1381

DaveCTurner opened this issue Sep 21, 2021 · 10 comments

Comments

@DaveCTurner
Copy link
Contributor

@DaveCTurner DaveCTurner commented Sep 21, 2021

The latest libffi (at time of writing) adds support for specifying its temporary directory via the LIBFFI_TMPDIR environment variable. Earlier versions only use more generally-applicable locations such as TMPDIR and HOME. In Elasticsearch we'd like to be able to control where libffi writes its executables by setting LIBFFI_TMPDIR rather than TMPDIR, because administrators of locked-down systems may prefer to keep TMPDIR pointing somewhere that's mounted noexec.

Relates #1378 which fixes a segfault if TMPDIR is mounted noexec, but doesn't solve the underlying problem that libffi needs to write executables somewhere.

Relates elastic/elasticsearch#77014 which tracks the changes we intend to make in Elasticsearch once JNA is using a sufficiently-recent libffi.

@matthiasblaesing
Copy link
Member

@matthiasblaesing matthiasblaesing commented Oct 2, 2021

A first rough update can be found in PR #1383. This is not complete, but a first working version.

@matthiasblaesing
Copy link
Member

@matthiasblaesing matthiasblaesing commented Oct 7, 2021

Ok - ran unittests on:

  • Linux x86-64
  • Windows x86-64
  • Windows x86
  • Solaris x86-64
  • Solaris Sparc9 (64Bit)
  • macOS x86-64

All existing native libraries, with the exception of AIX, were rebuild with the new libffi. As this does not touch the binary interface, AIX should still work.

I would appreciate a test, before I merge this into master.

@DaveCTurner
Copy link
Contributor Author

@DaveCTurner DaveCTurner commented Oct 11, 2021

I would appreciate a test, before I merge this into master.

I can confirm that Elasticsearch successfully works with the new version built from your branch. However I'm struggling to test that it supports LIBFFI_TMPDIR as expected, because this only has an effect when SELinux is enabled and I don't have access to such a machine at the moment.

If you would like me to check that too, could you point me to instructions to rebuild the native code? That way I can skip the SELinux checks and force it always to use file-backed pages for its executables. I tried cd native && ant but it failed:

~/jna/native$ ant
Buildfile: /home/davidturner/jna/native/build.xml
     [echo] java.home: /usr/lib/jvm/java-8-oracle/jre
     [echo] jdk.home: /usr/lib/jvm/java-8-oracle
     [echo] JAVA_HOME: /usr/lib/jvm/java-8-oracle

workspace-base:

compile-base:

compile-core:
     [echo] Compile /home/davidturner/jna/src - compatibility=1.6 - target=/home/davidturner/jna/native/target/classes

javah:
     [echo] Create native class headers in /home/davidturner/jna/native/target/native-linux-x86-64
    [javah] [Forcefully writing file RegularFileObject[/home/davidturner/jna/native/target/native-linux-x86-64/com_sun_jna_Function.h]]
    [javah] [Forcefully writing file RegularFileObject[/home/davidturner/jna/native/target/native-linux-x86-64/com_sun_jna_Function_PointerArray.h]]
    [javah] [Forcefully writing file RegularFileObject[/home/davidturner/jna/native/target/native-linux-x86-64/com_sun_jna_Function_NativeMappedArray.h]]
    [javah] [Forcefully writing file RegularFileObject[/home/davidturner/jna/native/target/native-linux-x86-64/com_sun_jna_Function_PostCallRead.h]]
    [javah] [Forcefully writing file RegularFileObject[/home/davidturner/jna/native/target/native-linux-x86-64/com_sun_jna_Native.h]]
    [javah] [Forcefully writing file RegularFileObject[/home/davidturner/jna/native/target/native-linux-x86-64/com_sun_jna_Native_AWT.h]]
    [javah] [Forcefully writing file RegularFileObject[/home/davidturner/jna/native/target/native-linux-x86-64/com_sun_jna_Native_Buffers.h]]
    [javah] [Forcefully writing file RegularFileObject[/home/davidturner/jna/native/target/native-linux-x86-64/com_sun_jna_Native_ffi_callback.h]]
    [javah] [Forcefully writing file RegularFileObject[/home/davidturner/jna/native/target/native-linux-x86-64/com_sun_jna_win32_DLLCallback.h]]

create-version-resource:

compile-native:
     [echo] ARCH: x86-64
     [echo] make.USE_MSVC: IGNORE=
     [echo] make.CC: IGNORE=
     [echo] make.ARCH: ARCH=x86-64
     [echo] make.BUILD: BUILD=/home/davidturner/jna/native/target/native-linux-x86-64
     [echo] make.PATH: IGNORE=
     [echo] make.OPTS: IGNORE=
     [echo] make /home/davidturner/jna/native/Makefile - target=/home/davidturner/jna/native/target/native-linux-x86-64
     [exec] Configuring libffi (x86-64)
     [exec] configure: WARNING: unrecognized options: --enable-static, --disable-shared, --with-pic
     [exec] Makefile:509: recipe for target '/home/davidturner/jna/native/target/native-linux-x86-64/libffi/.libs/libffi.a' failed
     [exec] configure: error: cannot find install-sh, install.sh, or shtool in "/home/davidturner/jna/native/libffi" "/home/davidturner/jna/native/libffi/.." "/home/davidturner/jna/native/libffi/../.."
     [exec] make: *** [/home/davidturner/jna/native/target/native-linux-x86-64/libffi/.libs/libffi.a] Error 1

BUILD FAILED
/home/davidturner/jna/native/build.xml:364: exec returned: 2

Total time: 0 seconds

@matthiasblaesing
Copy link
Member

@matthiasblaesing matthiasblaesing commented Oct 11, 2021

Ah - I just noticed, that there are recipes to build on mac and windows, but not Linux - at some point that needs to be corrected. For debian/ubuntu this should work:

# fetch the build dependencies for the distribution libjna-java package
apt-get build-dep -yq --force-yes libjna-java
# install the remaining required dependencies
apt-get install -yq --force-yes zip unzip libtool automake libltdl-dev texinfo

@matthiasblaesing
Copy link
Member

@matthiasblaesing matthiasblaesing commented Oct 19, 2021

I now also ran unittests on solaris and found no regressions. I intent to merge this the next few days.

@matthiasblaesing
Copy link
Member

@matthiasblaesing matthiasblaesing commented Oct 20, 2021

Change was merged to master.

@DaveCTurner
Copy link
Contributor Author

@DaveCTurner DaveCTurner commented Oct 22, 2021

Sorry it's taken so long to get back to this - testing this turned into quite a yak-shaving exercise for various reasons. Anyway I can now confirm that with this version of JNA Elasticsearch does indeed respect the $LIBFFI_TMPDIR environment variable if it's prevented from using anonymous mappings, which was what we wanted 🎉

Thanks so much for doing this. Can you give us a sense of how long it'll be until the next release?

@matthiasblaesing
Copy link
Member

@matthiasblaesing matthiasblaesing commented Oct 22, 2021

Thank you for the check and verification!

For a release: It depends if anyone asks for it and you just did. I'll look into it.

@dkocher
Copy link
Contributor

@dkocher dkocher commented Nov 8, 2021

Thank you for the check and verification!

For a release: It depends if anyone asks for it and you just did. I'll look into it.

May I second that.

@matthiasblaesing
Copy link
Member

@matthiasblaesing matthiasblaesing commented Nov 8, 2021

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
3 participants