8200559: Java agents doing instrumentation need a means to define auxiliary classes #3546
Conversation
|
👋 Welcome back winterhalter! A progress list of the required criteria for merging this PR into |
openjdk
bot
added
serviceability
Webrevs
|
|
Mailing list message from Remi Forax on core-libs-dev: ----- Mail original -----
You can already use Lookup.defineClass() + privateLookupIn() + Instrumentation.redefineModule() for that ? R?mi |
1 similar comment
|
Mailing list message from Remi Forax on core-libs-dev: ----- Mail original -----
You can already use Lookup.defineClass() + privateLookupIn() + Instrumentation.redefineModule() for that ? R?mi |
|
JDK-8200559 is about defining auxiliary classes in the same runtime package at load-time whereas I think the proposal in this PR is adding an unrestricted defineClass to the Instrumentation class. I think this will require a lot of discussion as there are significant issues and concerns here. An unrestricted defineClass might be okay for tool/java agents when started from the command line with -javaagent but only if the Instrumentation object is never ever leaked to library or application code. It could potentially be part of a large effort to reduce the capabilities of agents loaded via the attach mechanism. More generally I think we need clearer separation of the requirements of tool agents from the requirement of framework/libraries that want to inject proxy and other classes at runtime. Separately, the proposal in JEP 410 is to terminally deprecate ProtectionDomain. |
|
Mailing list message from Rafael Winterhalter on core-libs-dev: Not by my understanding. A suitable lookup requires a loaded class for the Am Fr., 16. Apr. 2021 um 16:21 Uhr schrieb Remi Forax <forax at univ-mlv.fr>:
|
|
The ticket was created as a reaction to a write-up I made in January 2018. I certainly did not intend to limit the scope to same-package class definitions for instrumented classes, and even for those I do not think a package-restricted API would be sufficient as I outlined in the comments to JDK-8200559. I will try to make my case on the mailing list. I hoped this could get resolved within the release of Java 17 as this would make it possible to write agents without use of Unsafe API to support Java 17 and later. Since agents often are supplementary to a broad range of Java applications, the LTS release will likely be an important support boundary for years and years to come. You surely mean JEP 411 when mentioning
Also, since this is still a proposal, I would believe that APIs that are implemented before JEP 411 is realized should support |
|
Mailing list message from forax at univ-mlv.fr on core-libs-dev:
yes, you need a witness class in the package you want to define a new class. R?mi
|
1 similar comment
|
Mailing list message from forax at univ-mlv.fr on core-libs-dev:
yes, you need a witness class in the package you want to define a new class. R?mi
|
|
Mailing list message from Rafael Winterhalter on core-libs-dev: This would be a problem, however. Since there is currently no official way Am Fr., 16. Apr. 2021 um 18:35 Uhr schrieb <forax at univ-mlv.fr>:
|
|
Mailing list message from Alan Bateman on core-libs-dev: On 16/04/2021 17:40, Rafael Winterhalter wrote:
"are supplementary to a board range of Java applications" is part of the Just to add to R?mi's comment: For frameworks/libraries, the On Java agents, then I think the discussion will eventually lead into -Alan |
1 similar comment
|
Mailing list message from Alan Bateman on core-libs-dev: On 16/04/2021 17:40, Rafael Winterhalter wrote:
"are supplementary to a board range of Java applications" is part of the Just to add to R?mi's comment: For frameworks/libraries, the On Java agents, then I think the discussion will eventually lead into -Alan |
|
Mailing list message from forax at univ-mlv.fr on core-libs-dev:
I was thinking about adding a dummy class in the package you want to load classes. Anyway, you can also call ClassLoader.defineClass directly. Put the code that calls defineClass in a module and use Instrumentation.redefineModule() to open java.base to this module. R?mi
|
1 similar comment
|
Mailing list message from forax at univ-mlv.fr on core-libs-dev:
I was thinking about adding a dummy class in the package you want to load classes. Anyway, you can also call ClassLoader.defineClass directly. Put the code that calls defineClass in a module and use Instrumentation.redefineModule() to open java.base to this module. R?mi
|
|
Mailing list message from Rafael Winterhalter on core-libs-dev: I have never seen a need for a non-agent to define a class in a I have never heard about a 'discussion [that] will eventually lead into That said, even if it was restricted in the future, this would mean that Am Fr., 16. Apr. 2021 um 19:31 Uhr schrieb Alan Bateman <
|
raphw
changed the title
|
Mailing list message from Alan Bateman on core-libs-dev: On 16/04/2021 21:09, Rafael Winterhalter wrote:
Yes, the injection of proxy classes and the like is mainly the same
There was fuss on this topic during JDK 9. I'll try to find the mails in To re-cap, the main concern is the Instrumentation object is Time has moved on and maybe a better approach is to not change the XX Hopefully this helps sets some context as to why we have to be cautious -Alan |
1 similar comment
|
Mailing list message from Alan Bateman on core-libs-dev: On 16/04/2021 21:09, Rafael Winterhalter wrote:
Yes, the injection of proxy classes and the like is mainly the same
There was fuss on this topic during JDK 9. I'll try to find the mails in To re-cap, the main concern is the Instrumentation object is Time has moved on and maybe a better approach is to not change the XX Hopefully this helps sets some context as to why we have to be cautious -Alan |
|
Mailing list message from Rafael Winterhalter on core-libs-dev: As for the need to inject classes into 'new' packages, in Mockito we solve I remember the discussions about the dynamic agent attachment rather well, As for the proposed API, I understand that it needs thought, my hope is I think that the problem you want to solve is rather that JVMs should not Am So., 18. Apr. 2021 um 18:24 Uhr schrieb Alan Bateman <
|
1 similar comment
|
Mailing list message from Rafael Winterhalter on core-libs-dev: As for the need to inject classes into 'new' packages, in Mockito we solve I remember the discussions about the dynamic agent attachment rather well, As for the proposed API, I understand that it needs thought, my hope is I think that the problem you want to solve is rather that JVMs should not Am So., 18. Apr. 2021 um 18:24 Uhr schrieb Alan Bateman <
|
|
From: Alan Bateman
I think it would be easy to limit the use of this Instrumentation method to the agent code as agent classes are loaded by the bootstrap classloader. Simply make the method implementation caller-sensitive and check the caller is from bootstrap class loader. This way Instrumentation instance escaped to application code would not give that code any ability to define arbitrary classes. Good enough? Peter
|
|
This won't work as agents are loaded by the system class loader, not the
boot loader.
Peter Levart ***@***.***> schrieb am Mo., 19. Apr. 2021,
11:02:
… From: Alan Bateman
JDK-8200559 is about defining auxiliary classes in the same runtime
package at load-time whereas I think the proposal in this PR is adding an
unrestricted defineClass to the Instrumentation class. I think this will
require a lot of discussion as there are significant issues and concerns
here. An unrestricted defineClass might be okay for tool/java agents when
started from the command line with -javaagent but only if the
Instrumentation object is never ever leaked to library or application code.
It could potentially be part of a large effort to reduce the capabilities
of agents loaded via the attach mechanism. More generally I think we need
clearer separation of the requirements of tool agents from the requirement
of framework/libraries that want to inject proxy and other classes at
runtime.
I think it would be easy to limit the use of this Instrumentation method
to the agent code as agent classes are loaded by the bootstrap classloader.
Simply make the method implementation caller-sensitive and check the caller
is from bootstrap class loader. This way Instrumentation instance escaped
to application code would not give that code any ability to define
arbitrary classes.
Good enough?
Peter
Separately, the proposal in JEP 410 is to terminally deprecate
ProtectionDomain.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#3546 (comment)>, or
unsubscribe
<https://github.com/notifications/unsubscribe-auth/ABCIA4EYHK5OHTDAN3FKYULTJPWS5ANCNFSM43BSDEGQ>
.
|
Ah sorry, I didn't know that. So what about the following: agent is able to define (or load) a class from bootstrap loader. So it would be able to instantiate an intermediary class, loaded by bootstrap loader which would serve as an intermediary to call into this limited Instrumentation API point... |
The agent JAR file is added to application class path and is loaded using the system class loader. So almost always the defining loader will be the application class loader. In general it's a hard problem to try to balance the integrity and security of the platform with the needs of agents that do arbitrary injection and instrumentation. Specifying an agent on the command line with -javaagent is the opt-in to trust that agent and a defineClass that allows arbitrary injection is plausible for that deployment. As Rafael's mentioned in one of the messages, there is enough power in the existing Instrumentation API to do that in a round about way already. We don't have anything equivalent for agents that are loaded by tools into a target VM. I added the attach mechanism and the dynamic agent loading back in JDK 6 and regret not putting more restrictions around this. As it stands, it is open to mis-use in that an application or library can use the attach mechanism (directly or via the attach API in a child VM) to load an agent and leak the Instrumentation object. This is the genie that somehow needs to be put back in its bottle. One approach that I mentioned here to create is a less powerful Instrumentation object for untrusted agents. Trusted agents would continue to the full-power Instrumentation object. A less powerful Instrumentation object would not be able to redefine java.base or other core modules and would not be able to retransform classes in those modules. The option on the table during JDK 9 was to disable dynamic loading of agents by default but that was kicked down the road. I don't particularly like that option and I think we can do better. |
I hear Rafael that dynamic attach is important to support monitoring and instrumenting large numbers of JVMs with no preparations (i.e. without issueing special command-line options to enable it). As I understand, current attach mechanism is designed to allow a process running under the same UID as the JVM or under root to attach to the JVM. What if this dynamic attach mechanism was modified so that only a process running under root could dynamically attach to the JVM? Old behavior would be enabled by special command line option, so by default, dynamic attach would be limited to tools running under root. Rafael mentions discovery, monitoring and instrumenting large number of JVMs running on hosts, so if one such tool has to attach to different JVMs running under different UIDs, it has to run as root now anyway. With such default "secure" dynamic attach and when the JVM is not running as root (which is a recommended security practice anyway), a library in such JVM could not attach back to the same JVM even through spawning sub-processes. How to achieve such "secure" dynamic attach? One way that comes to mind is a modified handshake. Currently, I think at least on Linux, the tool that wishes to attach to the JVM searches for a special UNIX socket ( In modified handshake, JVM not running as root could not create a UNIX socket file with permissions to allow only root user to connect to it, but a tool running under root could create a listening UNIX socket with permission to allow JVM to connect to it in a way that the JVM connecting to such socket would know that the listening process is running as root. Simply by checking the owner of the listening UNIX socket file. Such socket file would have to have permission 0666 in order to allow JVMs running under any UID to connect to it, but otherwise be "hidden". This can be achieved by the tool creating a special directory and a UNIX socket in this directory, say: WDYT? |
|
Mailing list message from Alan Bateman on core-libs-dev: On 19/04/2021 15:10, Peter Levart wrote:
is a modified handshake. Currently, I think at least on Linux, the tool that wishes to attach to the JVM searches for a special UNIX socket (`$PWD/.java_pid<vmid>`, `/tmp/.java_pid<vmid>`) and if not found, creates a special attach file (`$PWD/.attach_pid<vmid>`, `/tmp/.attach_pid<vmid>`) to
such socket would know that the listening process is running as root. Simply by checking the owner of the listening UNIX socket file. Such socket file would have to have permission 0666 in order to allow JVMs running under any UID to connect to it, but otherwise be "hidden". This can be achieved by the tool creating a special directory and a UNIX socket in this directory, say: `/tmp/.attach_dir<tool pid>/<random string>`, The directory UID:GID would be 0:0 and have permission 0711. This means, any user could connect to the socket as long as it knows the `<random string>`, but no user but root can list the content of the directory to discover the name of the socket file. The last piece of the puzzle
creating a file with the content holding the name of the socket file would be OK, as long as only target JVM could read it. File permissions could One initial comment is that the attach mechanism is also used for the A second comment is that the attach mechanism on Linux and macOS is -Alan |
1 similar comment
|
Mailing list message from Alan Bateman on core-libs-dev: On 19/04/2021 15:10, Peter Levart wrote:
is a modified handshake. Currently, I think at least on Linux, the tool that wishes to attach to the JVM searches for a special UNIX socket (`$PWD/.java_pid<vmid>`, `/tmp/.java_pid<vmid>`) and if not found, creates a special attach file (`$PWD/.attach_pid<vmid>`, `/tmp/.attach_pid<vmid>`) to
such socket would know that the listening process is running as root. Simply by checking the owner of the listening UNIX socket file. Such socket file would have to have permission 0666 in order to allow JVMs running under any UID to connect to it, but otherwise be "hidden". This can be achieved by the tool creating a special directory and a UNIX socket in this directory, say: `/tmp/.attach_dir<tool pid>/<random string>`, The directory UID:GID would be 0:0 and have permission 0711. This means, any user could connect to the socket as long as it knows the `<random string>`, but no user but root can list the content of the directory to discover the name of the socket file. The last piece of the puzzle
creating a file with the content holding the name of the socket file would be OK, as long as only target JVM could read it. File permissions could One initial comment is that the attach mechanism is also used for the A second comment is that the attach mechanism on Linux and macOS is -Alan |
|
At the moment, it is required for root to switch to the user that owns the
JVM process as the domain socket is only accessible to that user to avoid
that users without access to the JVM can inject themselves into a JVM. I am
not sure if operations teams would be thrilled to have a monitoring agent
required to run as root, even in these times of Kubernetes.
I mainly have two comments:
1. The problem is the possibility of self-attach. I think this is the
problem to solve, a library getting agent privileges without being an
agent. I think this should be prevented while dynamic attach should
continue to be possible in today's format. It has proven to be so useful,
it would be a shame if the current tooling convenience would disappear from
the JVM. As it's my understanding, JNI is supposed to be restricted in the
future, in line with Panama. Without this restriction, JNI already allows
for random class definition, for example, which similarly to an agent
offers surpassing the majority of JVM restrictions. The second restriction
would be a control to restrict how a JVM process starts new processes. I
think both are reasonable restrictions for a library to face which require
explicit enabling. Especially with the security manager on it's way out,
certain capabilities should be rethought to begin with. If both are no
longer freely available, self-attachment is no longer possible anyways and
dynamic agents could retain their capabilities.
2. The question of introducing an Instrumentation::defineClass method is
fully independent of that first question. If a dynamic agent was to be
restricted, the method could reject classloader/package combinations for
dynamically loaded agents the same way that
Instrumentation::retransformClasses would need to. At the same time,
introducing the method would allow agents to move to an official API with a
Java 17 baseline which will be the next long-standing base line. I fully
understand it needs a thorough discussion but it is a less complicated
problem then (1) and could therefore be decided prior to having found a
satisfactory solution for it.
Am Mo., 19. Apr. 2021 um 16:07 Uhr schrieb Peter Levart <
***@***.***>:
… an application or library can use the attach mechanism (directly or via
the attach API in a child VM) to load an agent and leak the Instrumentation
object. This is the genie that somehow needs to be put back in its bottle.
One approach that I mentioned here to create is a less powerful
Instrumentation object for untrusted agents. Trusted agents would continue
to the full-power
I hear Rafael that dynamic attach is important to support monitoring and
instrumenting large numbers of JVMs with no preparations (i.e. without
issueing special command-line options to enable it). As I understand,
current attach mechanism is designed to allow a process running under the
same UID as the JVM or under root to attach to the JVM.
What if this dynamic attach mechanism was modified so that only a process
running under root could dynamically attach to the JVM? Old behavior would
be enabled by special command line option, so by default, dynamic attach
would be limited to tools running under root. Rafael mentions discovery,
monitoring and instrumenting large number of JVMs running on hosts, so if
one such tool has to attach to different JVMs running under different UIDs,
it has to run as root now anyway.
With such default "secure" dynamic attach and when the JVM is not running
as root (which is a recommended security practice anyway), a library in
such JVM could not attach back to the same JVM even through spawning
sub-processes.
How to achieve such "secure" dynamic attach? One way that comes to mind is
a modified handshake. Currently, I think at least on Linux, the tool that
wishes to attach to the JVM searches for a special UNIX socket (
$PWD/.java_pid<vmid>, /tmp/.java_pid<vmid>) and if not found, creates a
special attach file ($PWD/.attach_pid<vmid>, /tmp/.attach_pid<vmid>) to
signal the JVM to create a listening UNIX socket under mentioned special
path, then it connects to the socket. The UNIX socket file has UID:GID set
to effective UID:GID of the JVM process and permissions to 0600, so only a
tool running under same UID or root can connect to such socket.
In modified handshake, JVM not running as root could not create a UNIX
socket file with permissions to allow only root user to connect to it, but
a tool running under root could create a listening UNIX socket with
permission to allow JVM to connect to it in a way that the JVM connecting
to such socket would know that the listening process is running as root.
Simply by checking the owner of the listening UNIX socket file. Such socket
file would have to have permission 0666 in order to allow JVMs running
under any UID to connect to it, but otherwise be "hidden". This can be
achieved by the tool creating a special directory and a UNIX socket in this
directory, say: /tmp/.attach_dir<tool pid>/<random string>, The directory
UID:GID would be 0:0 and have permission 0711. This means, any user could
connect to the socket as long as it knows the <random string>, but no
user but root can list the content of the directory to discover the name of
the socket file. The last piece of the puzzle is how to signal to the JVM
about the name of the socket file. Well, creating a file with the content
holding the name of the socket file would be OK, as long as only target JVM
could read it. File permissions could be set such that any process running
under the same UID as the JVM could read the file. This would give a rouge
library a chance to connect to the tool and pretend to be the monitoring
JVM, but it could not connect to back to the JVM though...
WDYT?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#3546 (comment)>, or
unsubscribe
<https://github.com/notifications/unsubscribe-auth/ABCIA4GPK73MWBRBICZS6Y3TJQ2LLANCNFSM43BSDEGQ>
.
|
|
Mailing list message from Alan Bateman on core-libs-dev: On 19/04/2021 22:20, Rafael Winterhalter wrote:
I should have been clearer, it's the combination of the two that creates ByteBuddy looks great for code generation and transforming classes but You mentioned restricting JNI in the future. I'm not aware of a definite As regards this PR then I would be happy to work with you on a revised -Alan. |
|
I fully understand your concerns about ByteBuddyAgent.install(). It is
simply a convenience for something that can be meaningful in some contexts
where I prefer offering a simple API. I use it mainly for two purposes:
a) For testing Java agents and integrations against Instrumentation within
the current VM when tests are triggered by tools that do not support
javaagents, also because builds do not bundle jars until after tests are
executed.
b) For purposefully "hacky" test libraries like Mockito that need agent
capabilities without this being meant to be used in production
environments. I have earlier proposed to offer a "jdk.test" module that
offers the Instrumentation instance via a simple API similar to Byte
Buddy's. The JVM would not load this module unless requested on the command
line. Build tools like Maven's surefire or Gradle's testrunner could then
standardize on loading this module as a convention to give access to this
test module by default such that libraries like Mockito could continue to
function out of the box without the libraries functioning on a standard VM
without extra configuration. As far as I know, mainly test libraries need
this API. This would also emphasise that Mockito and others are meant for
testing and fewer people would abuse it for production applications. People
would also have an explicit means of running a JVM for a production
application or for executing a test.
As for adding the API, my thought is that if the Instrumentation API were
to throw exceptions on some methods/arguments for dynamic agents in the
future, for example for retransformClasses(Object.class), this breaking
change would then simply extend to the proposed "defineClass" method. In
this sense, the Instrumentation API already assumes full power, I find it
not problematic to add the missing bit to this API even if it was
restricted in the future in the same spirit as other methods of the API
would be.
I mentioned JNI as it is a well-known approach to defining a class today,
using a minimal native binding to an interface that directly calls down to
JNI's:
jclass DefineClass(JNIEnv *env, const char *name, jobject loader, const
jbyte *buf, jsize bufLen);
This interface can then simply be used to define any class just as I
propse, even when not writing an agent or attaching. This method makes
class definitions also already trivial for JVMTI agents compared to Java
agents. Unless restricting JNI, the defineClass method is already a low
hanging fruit, but at the cost of having to maintain a tiny bit of native
code. I'd rather see this avoided and a standard API being offered to
agents up to the time that Panama is in place and a JNI restriction is
possibly also included. As a bonus: Once JNI is restricted, Byte Buddy's
"install" would no longer work unless self-attachment (or JNI) is
explicitly allowed. The emulation already requires to run native code while
the Virtual Machine API explicitly checks for the process id of the current
VM against the one that is targeted. With both disabled, self-attachment
would no longer be practically be possible without needing to prune the
capabilities of dynamic agents which is what I understand would be the
desired effect.
From this viewpoint, I think that adding Instrumentation::defineClass
method does no harm compared to the status quo. And on the upside, it gives
agents an API to migrate to, avoiding the last need of using unsafe. To
make the JVM a safe platform, binding native code would anyways need
restriction and this would then also solve the problem of dynamic agents
attaching from the same VM being used in libraries. This would in my eyes
be the cleanest solution to the self-attachment problem without disturbing
the existing landscape of dynamic agents. To run Mockito, one would then
instead configure Maven surefire or Gradle to run the JVM with
-Djdk.attach.allowAttachSelf=true. Ideally, some "jdk.test" module would be
added at some point, to avoid the overhead of self-attachment, but I think
this better fits into separate debate.
Am Di., 20. Apr. 2021 um 15:38 Uhr schrieb mlbridge[bot] <
***@***.***>:
… *Mailing list message from Alan Bateman ***@***.***> on
core-libs-dev ***@***.***>:*
On 19/04/2021 22:20, Rafael Winterhalter wrote:
:
At the moment, it is required for root to switch to the user that owns the
JVM process as the domain socket is only accessible to that user to avoid
that users without access to the JVM can inject themselves into a JVM. I am
not sure if operations teams would be thrilled to have a monitoring agent
required to run as root, even in these times of Kubernetes.
I mainly have two comments:
1. The problem is the possibility of self-attach. I think this is the
problem to solve, a library getting agent privileges without being an
agent. I think this should be prevented while dynamic attach should
continue to be possible in today's format. It has proven to be so useful,
it would be a shame if the current tooling convenience would disappear from
the JVM. As it's my understanding, JNI is supposed to be restricted in the
future, in line with Panama. Without this restriction, JNI already allows
for random class definition, for example, which similarly to an agent
offers surpassing the majority of JVM restrictions. The second restriction
would be a control to restrict how a JVM process starts new processes. I
think both are reasonable restrictions for a library to face which require
explicit enabling. Especially with the security manager on it's way out,
certain capabilities should be rethought to begin with. If both are no
longer freely available, self-attachment is no longer possible anyways and
dynamic agents could retain their capabilities.
2. The question of introducing an Instrumentation::defineClass method is
fully independent of that first question. If a dynamic agent was to be
restricted, the method could reject classloader/package combinations for
dynamically loaded agents the same way that
Instrumentation::retransformClasses would need to. At the same time,
introducing the method would allow agents to move to an official API with a
Java 17 baseline which will be the next long-standing base line. I fully
understand it needs a thorough discussion but it is a less complicated
problem then (1) and could therefore be decided prior to having found a
satisfactory solution for it.
I should have been clearer, it's the combination of the two that creates
the attractive nuisance. I don't think there are any objections to a
defineClass for agents specified on the command line with -javaagent.
However we have to be cautious about extending that capability to agents
that are loaded into a running VM with the attach mechanism.
ByteBuddy looks great for code generation and transforming classes but
ByteBuddyAgent makes me nervous. It looks like I can deploy
byte-buddy-agent-<version>.jar on my class path and invoke the public
static ByteBuddyAgent.install() method to get the Instrumentation object
for the current VM. That may be convenient for some but this is the
all-powerful Instrumentation object that shouldn't be leaked to library
or application code. Now combine this with the proposed defineClass and
it means that any code on the class path could inject a class into
java.lang or any run-time package without any agent voodoo or opt-in via
the command line. That would be difficult genie to re-bottle if it were
to get traction.
You mentioned restricting JNI in the future. I'm not aware of a definite
plan or time-frame. Project Panama is pioneering restricting access to
native operations as a bug or mis-use with the linker API can easily
crash the VM or breakage in other ways. Extending this to JNI would be a
logical next step but I could imagine it taking a long time and many
releases to get there.
As regards this PR then I would be happy to work with you on a revised
proposed that would limit it to agents specified with -javaagent. That
would not preclude extending the capability, maybe in a more restricted
form, to agents loaded into a running VM in the future.
-Alan.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#3546 (comment)>, or
unsubscribe
<https://github.com/notifications/unsubscribe-auth/ABCIA4FE2B4DGBZS4QO6SM3TJV7T5ANCNFSM43BSDEGQ>
.
|
|
Mailing list message from Ron Pressler on core-libs-dev: I think you?re coming to this discussion with a very different perspective from us, which Our goal is not to maintain some status quo until such time we decide to restrict it. We?ve But this is a process. As long as there are gaps in the garden fence ? Unsafe, self-attach, JNI ? Suggestions should, therefore, focus on ideas compatible with this vision. To be more constructive For example, self-attach is not the only issue. Leaking powerful Instrumentation objects to libraries ? Ron |
1 similar comment
|
Mailing list message from Ron Pressler on core-libs-dev: I think you?re coming to this discussion with a very different perspective from us, which Our goal is not to maintain some status quo until such time we decide to restrict it. We?ve But this is a process. As long as there are gaps in the garden fence ? Unsafe, self-attach, JNI ? Suggestions should, therefore, focus on ideas compatible with this vision. To be more constructive For example, self-attach is not the only issue. Leaking powerful Instrumentation objects to libraries ? Ron |
|
Mailing list message from Rafael Winterhalter on core-libs-dev: Hi Ron, we certainly do come from different backgrounds and therefore perspectives, I do believe that I have an understanding of your angle, I tried to give my I am very much aware that this metaphoric fence between Java agents and My second argument is that restricting dynamic agents would require to stub Therefore, I hope this API can be considered. Agent developers would Best regards, Rafael Am Mi., 21. Apr. 2021 um 19:00 Uhr schrieb Ron Pressler <
|
1 similar comment
|
Mailing list message from Rafael Winterhalter on core-libs-dev: Hi Ron, we certainly do come from different backgrounds and therefore perspectives, I do believe that I have an understanding of your angle, I tried to give my I am very much aware that this metaphoric fence between Java agents and My second argument is that restricting dynamic agents would require to stub Therefore, I hope this API can be considered. Agent developers would Best regards, Rafael Am Mi., 21. Apr. 2021 um 19:00 Uhr schrieb Ron Pressler <
|
|
Mailing list message from Alan Bateman on core-libs-dev: On 20/04/2021 21:26, Rafael Winterhalter wrote:
Helping testing is good but a jdk.test module that hands out the
I think it would really hard to put this genie back into a bottle. It's
Sure, if you are using native code then you have the full power of JVM -Alan |
|
Sorry to be late to this party. I've been wanting to read this thread for a while but have bene too busy up to now. I have just a few comments I too was party to the discussions about agent capabilities and recall well the decision to gradually impose restrictions, the first one being to control dynamic agent loading. I was happy to accept that general decision and the specific one to limit the opportunity for an agent self-hoisting into the current JVM. However, a key part of the plan to move forward on restrictions was to provide an override switch. I'd very much like to see that retained as an option. I know that in some uses self-hoisting is much preferable to having to install the agent on the command line and I'd expect he same to be true for any other capabilities for which restrictions were adopted. Although it is true -- as Ron said - -that configuring -javaagent on the command line is always /possible/ there are actually many scenarios for agent use where deployment of an agent after startup is pragmatically much more desirable. An obvious use is trouble shooting, where you only want an agent in place when something goes wrong. That turns out to be critical to solving some seriously difficult support cases. The interesting use cases also fall under testing where self-hoisting of a test agent by a test framework can result in an enormous simplification of test configuration. Usage of Byteman for testing went through the roof with this capability in place. Never underestimate the degree to which even the most minimal configuration complexity puts off Enterprise java developers when it is multiplied by the test suite size of a large project. So, likewise with other restrictions on behaviour, I'm very happy to see them put in place for dynamically hoisted agents so long as there is still a command line override along the lines of the agent attach property that allows a dynamic agent to do all that a command line agent can. One other thing I'd like to correct is a point made in the discussion about agent code residing in the system loader. It is true that the main agent class gets loaded by the system loader. However, it is perfectly possible to ensure that all the rest of the agent code is loaded by the bootstrap loader. A Main class can add the agent jar to the bootstrap path and then load and use reflection to invoke an effective main routine on a bootstrap loaded SubMain class. Byteman uses this trick on request in order to allow it to instrument bootstrap classes. Because all the Byteman classes except for the original Main shell class are loaded by the bootstrap loader Byteman can safely inject references to the Byteman rule engine and Byteman exception classes into bootstrap code. |
|
Setting '-javaagent' is mainly an operations problem. Many build tools do
not allow to declare a test dependency this way as the life cycles are not
laid out for it, the internal repository location might be machine
dependent, for example, and it's difficult to point to a specific folder
and file reliably. In this case, I'd say it would be easier to specify a
parameter in the sense of '-Djdk.attach.allowAttachSelf=true' as it is a
static value. This would however only work for build tools that initiate a
new VM for running tests which is not overly attractive for simple builds
due to the overhead. Of course, Maven, Gradle and similar tools could set
this parameter by default for their own JVM, that part is likely
overcomeable but it will certainly create confusion about how to run
libraries that are omnipresent today and make the JVM ecosystem less
approachable.
What bothers me more is the tooling perspective for non-self-attached
agents. For example, Aaeron offers a Java agent that adds plenty of debug
logging to relevant lines of code. This affects method size and so forth,
with Aaeron as a high-performance tool for banking and finance which is
written very consciously with regards to the JIT, adding it directly was
not feasible. Normally this logging is therefore thrown into a VM in
retrospect, once an application starts failing and is already taken off the
load balancer. For such a post-mortem, it would be rather annoying to
realize that a JVM cannot be attached to with full capabilities if you
forgot to set some flag. And often you did of course not consider the VM
instance to fail, sometimes it takes months to get a JVM into this buggy
state. This would be fairly inconvenient to face.
Therefore, I really hope that the dynamic attach from 'outside' the VM will
survive without imposing limits and that rather the self-attachment problem
will be targeted as such. When I mention a 'jdk.test' module in the Mockito
context, I am also rather hoping to improve performance compared to
convenience. The problem with '-Djdk.attach.allowAttachSelf=true' is that
you still need to start a new VM etc. Since Java 9, running single tests
with Mockito has for example become much slower compared to Java 8. Despite
the startup performance improvements in the JVM. If one could avoid the
location-bound '-javaagent:...', but get the Instrumentation instance
directly, I think this would render a real performance improvement in
actual execution scenarios.
Am Mi., 21. Apr. 2021 um 20:42 Uhr schrieb mlbridge[bot] <
***@***.***>:
… *Mailing list message from Alan Bateman ***@***.***> on
core-libs-dev ***@***.***>:*
On 20/04/2021 21:26, Rafael Winterhalter wrote:
I have earlier proposed to offer a "jdk.test" module that
offers the Instrumentation instance via a simple API similar to Byte
Buddy's. The JVM would not load this module unless requested on the command
line. Build tools like Maven's surefire or Gradle's testrunner could then
standardize on loading this module as a convention to give access to this
test module by default such that libraries like Mockito could continue to
function out of the box without the libraries functioning on a standard VM
without extra configuration. As far as I know, mainly test libraries need
this API. This would also emphasise that Mockito and others are meant for
testing and fewer people would abuse it for production applications. People
would also have an explicit means of running a JVM for a production
application or for executing a test.
Helping testing is good but a jdk.test module that hands out the
Instrumentation object could be problematic. Is there a reason why the
runner for Mockito and other mocking frameworks can't specify -javaagent
when launching? I would expect at least some mocking scenarios to
require load time transformations (to drop the final modifier from some
API elements for example) so important to have the transformer set
before classes are loaded.
As for adding the API, my thought is that if the Instrumentation API were
to throw exceptions on some methods/arguments for dynamic agents in the
future, for example for retransformClasses(Object.class), this breaking
change would then simply extend to the proposed "defineClass" method. In
this sense, the Instrumentation API already assumes full power, I find it
not problematic to add the missing bit to this API even if it was
restricted in the future in the same spirit as other methods of the API
would be.
I think it would really hard to put this genie back into a bottle. It's
way more attractive to use that than the very agent oriented
redefineModule and retransformClasses.
I mentioned JNI as it is a well-known approach to defining a class today,
using a minimal native binding to an interface that directly calls down to
JNI's:
jclass DefineClass(JNIEnv *env, const char *name, jobject loader, const
jbyte *buf, jsize bufLen);
This interface can then simply be used to define any class just as I
propse, even when not writing an agent or attaching. This method makes
class definitions also already trivial for JVMTI agents compared to Java
agents. Unless restricting JNI, the defineClass method is already a low
hanging fruit, but at the cost of having to maintain a tiny bit of native
code.
Sure, if you are using native code then you have the full power of JVM
TI and JNI available. Project Panama is exploring how to restrict access
to native code, I think too early to say how this might extend to JNI.
-Alan
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#3546 (comment)>, or
unsubscribe
<https://github.com/notifications/unsubscribe-auth/ABCIA4HHMA67R24GWUPFVATTJ4MBRANCNFSM43BSDEGQ>
.
|
|
Mailing list message from Alan Snyder on core-libs-dev: On Apr 21, 2021, at 11:40 AM, Alan Bateman <Alan.Bateman at oracle.com> wrote:
I looked at some of the Panama documents and saw no hint that it might be extended to JNI. It seems to be positioned as an (partial) alternative to JNI. What I do see is that native code has no direct access to the JDK via Panama, only the ability to invoke provided upcalls, which can only access objects and methods that the caller has access to. That is indeed much more restricted than JNI. Even though it is ?too early?, can you explain why you think Panama?s restrictions might apply to JNI? As a library developer, I have often made use of JNI to work around limitations in current and older JDKs. I would hate to lose that ability. Alan |
1 similar comment
|
Mailing list message from Alan Snyder on core-libs-dev: On Apr 21, 2021, at 11:40 AM, Alan Bateman <Alan.Bateman at oracle.com> wrote:
I looked at some of the Panama documents and saw no hint that it might be extended to JNI. It seems to be positioned as an (partial) alternative to JNI. What I do see is that native code has no direct access to the JDK via Panama, only the ability to invoke provided upcalls, which can only access objects and methods that the caller has access to. That is indeed much more restricted than JNI. Even though it is ?too early?, can you explain why you think Panama?s restrictions might apply to JNI? As a library developer, I have often made use of JNI to work around limitations in current and older JDKs. I would hate to lose that ability. Alan |
|
@raphw This pull request has been inactive for more than 4 weeks and will be automatically closed if another 4 weeks passes without any activity. To avoid this, simply add a new comment to the pull request. Feel free to ask for assistance if you need help with progressing this pull request towards integration! |
|
@raphw This pull request has been inactive for more than 8 weeks and will now be automatically closed. If you would like to continue working on this pull request in the future, feel free to reopen it! This can be done using the |
To allow agents the definition of auxiliary classes, an API is needed to allow this. Currently, this is often achieved by using
sun.misc.Unsafeorjdk.internal.misc.Unsafeever since thedefineClassmethod was removed fromsun.misc.Unsafe.Progress
Issue
Reviewing
Using
gitCheckout this PR locally:
$ git fetch https://git.openjdk.java.net/jdk pull/3546/head:pull/3546$ git checkout pull/3546Update a local copy of the PR:
$ git checkout pull/3546$ git pull https://git.openjdk.java.net/jdk pull/3546/headUsing Skara CLI tools
Checkout this PR locally:
$ git pr checkout 3546View PR using the GUI difftool:
$ git pr show -t 3546Using diff file
Download this PR as a diff file:
https://git.openjdk.java.net/jdk/pull/3546.diff