-
Notifications
You must be signed in to change notification settings - Fork 5.7k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
8268129: LibraryLookup::ofDefault leaks symbols from loaded libraries #4316
Conversation
/csr |
👋 Welcome back mcimadamore! A progress list of the required criteria for merging this PR into |
@mcimadamore this pull request will not be integrated until the CSR request JDK-8268130 for issue JDK-8268129 has been approved. |
@mcimadamore The following labels will be automatically applied to this pull request:
When this pull request is ready to be reviewed, an "RFR" email will be sent to the corresponding mailing lists. If you would like to change these labels, use the /label pull request command. |
Some notes on how the system lookup is implemented (see class SystemLookup):
In both cases, the required libraries are loaded in a classloader-independent fashion, by taking advantage of the support available in NativeLibraries. Class loader lookups are built on top of ClassLoader::findNative (which is also the method used by JNI to find JNI methods). This patch removes all the native code which was required to support the default lookup abstraction. The implementation changes are relatively straightforward - but several test and microbenchmark updates were required. |
Webrevs
|
Mailing list message from Chapman Flack on security-dev: On 06/02/21 13:30, Maurizio Cimadamore wrote:
Hi, While I am thinking about this, what will be the behavior when the JVM Will there be a *Lookup flavor that is able to find any exported symbol Regards, |
1 similar comment
Mailing list message from Chapman Flack on security-dev: On 06/02/21 13:30, Maurizio Cimadamore wrote:
Hi, While I am thinking about this, what will be the behavior when the JVM Will there be a *Lookup flavor that is able to find any exported symbol Regards, |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM, minor nit inline
test/micro/org/openjdk/bench/jdk/incubator/foreign/points/support/PanamaPoint.java
Outdated
Show resolved
Hide resolved
…ort/PanamaPoint.java Co-authored-by: Jorn Vernee <JornVernee@users.noreply.github.com>
Hi, https://gist.github.com/mcimadamore/0883ea6f4836ae0c1d2a31c48197da1a Perhaps something like that could be useful in the use case you mention? |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Looks good. Just minor comments to accept/reject.
The shim library is an interesting approach. I did wonder if the libvm library could be used, but it might have some weird side-effects or bring in more symbols than necessary.
src/jdk.incubator.foreign/share/classes/jdk/incubator/foreign/CLinker.java
Outdated
Show resolved
Hide resolved
src/jdk.incubator.foreign/share/classes/jdk/internal/foreign/abi/SharedUtils.java
Outdated
Show resolved
Hide resolved
src/jdk.incubator.foreign/share/classes/jdk/incubator/foreign/CLinker.java
Outdated
Show resolved
Hide resolved
…bi/SharedUtils.java Co-authored-by: Paul Sandoz <paul.d.sandoz@googlemail.com>
Co-authored-by: Paul Sandoz <paul.d.sandoz@googlemail.com>
Co-authored-by: Paul Sandoz <paul.d.sandoz@googlemail.com>
Co-authored-by: Paul Sandoz <paul.d.sandoz@googlemail.com>
Co-authored-by: Paul Sandoz <paul.d.sandoz@googlemail.com>
Co-authored-by: Paul Sandoz <paul.d.sandoz@googlemail.com>
Co-authored-by: Paul Sandoz <paul.d.sandoz@googlemail.com>
Co-authored-by: Paul Sandoz <paul.d.sandoz@googlemail.com>
Co-authored-by: Paul Sandoz <paul.d.sandoz@googlemail.com>
Co-authored-by: Paul Sandoz <paul.d.sandoz@googlemail.com>
@mcimadamore This change now passes all automated pre-integration checks. ℹ️ This project also has non-automated pre-integration requirements. Please see the file CONTRIBUTING.md for details. After integration, the commit message for the final commit will be:
You can use pull request commands such as /summary, /contributor and /issue to adjust it as needed. At the time when this comment was updated there had been 5 new commits pushed to the
Please see this link for an up-to-date comparison between the source branch of this pull request and the ➡️ To integrate this PR with the above commit message to the |
@mcimadamore this pull request can not be integrated into git checkout symbolLookup
git fetch https://git.openjdk.java.net/jdk master
git merge FETCH_HEAD
# resolve conflicts and follow the instructions given by git merge
git commit -m "Merge master"
git push |
/cc build |
@mcimadamore |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Looks pretty good, just a few comments.
/integrate |
@mcimadamore Since your change was applied there have been 15 commits pushed to the
Your commit was automatically rebased without conflicts. Pushed as commit 59a539f. 💡 You may see a message that your pull request was closed with unmerged commits. This can be safely ignored. |
Hi @mcimadamore, As you mentioned at #4316 (comment), the system lookup is supported by the underlying But my finding is that it
with a simple test & output:
So I am wondering what happened to the system lookup in such case given there should be no fundamental difference in leveraging |
In 17, SystemLookup loads a specific library that is generated at build time - which contains all the c stdlib symbols. That's what the Lib.gmk changes are for. What I suspect is that the library is not generated at all, or not correctly on AIX, which then causes the system lookup to misbehave. I would start by double checking that you have a file like this: <JDK_ROOT>/lib/libsyslookup.so And then, if the library exists, check that it has the right dependency; on my linux it's an empty library, but which depends on libc, libm and libdl:
Given that you don't seem to get an error when loading syslookup.so - the most likely thing is that you have an empty library with no dynamic dependencies (given you mention libc.a - which seems a static library?). So, perhaps on AIX something similar to what we did for Windows [1] would be better. |
Hi @mcimadamore, Here's output of
So it is high likely that there were no dependencies in this generated library.
That's what I thought to be the only way around but might need to figure out the specifics on AIX. |
Is |
I tried to load
and
|
Sorry - what I meant is -
|
Just tried with
|
Yes and no. System::loadLibrary wants a library name (no extension). It will add a library prefix (e.g.
You will end up with
|
If so, I am wondering whether there is anything else left (inherited from JDK16) in JDK17 we can leverage to support |
I think the reason it worked in JDK 16 is because all the symbols in the JVM were visible through the system lookup, and the JVM statically links the standard library (AFAIU). So, just because the VM code depended on something, it could be loaded by the system lookup. But, that would mean that not all symbols in the standard library were visible... and also, being able to find any symbol in the JVM was a bug. I think the only solution here - assuming that libc is not available as a dynamic library on typical AIX systems - is to figure out how to repackage these static libraries as a dynamic library, retaining all the symbols, and then bundle that dynamic library with the JDK. |
This patch overhauls the library loading mechanism used by the Foreign Linker API. We realized that, while handy, the default lookup abstraction (
LibraryLookup::ofDefault
) was behaving inconsistentlt across platforms.This patch replaces
LibraryLookup
with a simplerSymbolLookup
abstraction, a functional interface. Crucially,SymbolLookup
does not concern with library loading, only symbol lookup. For this reason, two factories are added:SymbolLookup::loaderLookup
- which obtains a lookup that can be used to lookup symbols in libraries loaded by current loaderCLinker::systemLookup
- a more stable replacement for the default lookup, which looks for symbols in libc.so (or its equivalent in other platforms). The contents of this lookup are unspecified.Both factories are restricted, so they can only be called when
--enable-native-access
is set.Progress
Issue
Reviewers
Reviewing
Using
git
Checkout this PR locally:
$ git fetch https://git.openjdk.java.net/jdk pull/4316/head:pull/4316
$ git checkout pull/4316
Update a local copy of the PR:
$ git checkout pull/4316
$ git pull https://git.openjdk.java.net/jdk pull/4316/head
Using Skara CLI tools
Checkout this PR locally:
$ git pr checkout 4316
View PR using the GUI difftool:
$ git pr show -t 4316
Using diff file
Download this PR as a diff file:
https://git.openjdk.java.net/jdk/pull/4316.diff