8200559: Java agents doing instrumentation need a means to define auxiliary classes #3546
Conversation
|
👋 Welcome back winterhalter! A progress list of the required criteria for merging this PR into |
openjdk
bot
added
serviceability
Webrevs
|
|
Mailing list message from Remi Forax on core-libs-dev: ----- Mail original -----
You can already use Lookup.defineClass() + privateLookupIn() + Instrumentation.redefineModule() for that ? R?mi |
1 similar comment
|
Mailing list message from Remi Forax on core-libs-dev: ----- Mail original -----
You can already use Lookup.defineClass() + privateLookupIn() + Instrumentation.redefineModule() for that ? R?mi |
|
JDK-8200559 is about defining auxiliary classes in the same runtime package at load-time whereas I think the proposal in this PR is adding an unrestricted defineClass to the Instrumentation class. I think this will require a lot of discussion as there are significant issues and concerns here. An unrestricted defineClass might be okay for tool/java agents when started from the command line with -javaagent but only if the Instrumentation object is never ever leaked to library or application code. It could potentially be part of a large effort to reduce the capabilities of agents loaded via the attach mechanism. More generally I think we need clearer separation of the requirements of tool agents from the requirement of framework/libraries that want to inject proxy and other classes at runtime. Separately, the proposal in JEP 410 is to terminally deprecate ProtectionDomain. |
|
Mailing list message from Rafael Winterhalter on core-libs-dev: Not by my understanding. A suitable lookup requires a loaded class for the Am Fr., 16. Apr. 2021 um 16:21 Uhr schrieb Remi Forax <forax at univ-mlv.fr>:
|
|
The ticket was created as a reaction to a write-up I made in January 2018. I certainly did not intend to limit the scope to same-package class definitions for instrumented classes, and even for those I do not think a package-restricted API would be sufficient as I outlined in the comments to JDK-8200559. I will try to make my case on the mailing list. I hoped this could get resolved within the release of Java 17 as this would make it possible to write agents without use of Unsafe API to support Java 17 and later. Since agents often are supplementary to a broad range of Java applications, the LTS release will likely be an important support boundary for years and years to come. You surely mean JEP 411 when mentioning
Also, since this is still a proposal, I would believe that APIs that are implemented before JEP 411 is realized should support |
|
Mailing list message from forax at univ-mlv.fr on core-libs-dev:
yes, you need a witness class in the package you want to define a new class. R?mi
|
1 similar comment
|
Mailing list message from forax at univ-mlv.fr on core-libs-dev:
yes, you need a witness class in the package you want to define a new class. R?mi
|
|
Mailing list message from Rafael Winterhalter on core-libs-dev: This would be a problem, however. Since there is currently no official way Am Fr., 16. Apr. 2021 um 18:35 Uhr schrieb <forax at univ-mlv.fr>:
|
|
Mailing list message from Alan Bateman on core-libs-dev: On 16/04/2021 17:40, Rafael Winterhalter wrote:
"are supplementary to a board range of Java applications" is part of the Just to add to R?mi's comment: For frameworks/libraries, the On Java agents, then I think the discussion will eventually lead into -Alan |
1 similar comment
|
Mailing list message from Alan Bateman on core-libs-dev: On 16/04/2021 17:40, Rafael Winterhalter wrote:
"are supplementary to a board range of Java applications" is part of the Just to add to R?mi's comment: For frameworks/libraries, the On Java agents, then I think the discussion will eventually lead into -Alan |
|
Mailing list message from forax at univ-mlv.fr on core-libs-dev:
I was thinking about adding a dummy class in the package you want to load classes. Anyway, you can also call ClassLoader.defineClass directly. Put the code that calls defineClass in a module and use Instrumentation.redefineModule() to open java.base to this module. R?mi
|
1 similar comment
|
Mailing list message from forax at univ-mlv.fr on core-libs-dev:
I was thinking about adding a dummy class in the package you want to load classes. Anyway, you can also call ClassLoader.defineClass directly. Put the code that calls defineClass in a module and use Instrumentation.redefineModule() to open java.base to this module. R?mi
|
|
Mailing list message from Rafael Winterhalter on core-libs-dev: I have never seen a need for a non-agent to define a class in a I have never heard about a 'discussion [that] will eventually lead into That said, even if it was restricted in the future, this would mean that Am Fr., 16. Apr. 2021 um 19:31 Uhr schrieb Alan Bateman <
|
raphw
changed the title
|
Mailing list message from Alan Bateman on core-libs-dev: On 16/04/2021 21:09, Rafael Winterhalter wrote:
Yes, the injection of proxy classes and the like is mainly the same
There was fuss on this topic during JDK 9. I'll try to find the mails in To re-cap, the main concern is the Instrumentation object is Time has moved on and maybe a better approach is to not change the XX Hopefully this helps sets some context as to why we have to be cautious -Alan |
1 similar comment
|
Mailing list message from Alan Bateman on core-libs-dev: On 16/04/2021 21:09, Rafael Winterhalter wrote:
Yes, the injection of proxy classes and the like is mainly the same
There was fuss on this topic during JDK 9. I'll try to find the mails in To re-cap, the main concern is the Instrumentation object is Time has moved on and maybe a better approach is to not change the XX Hopefully this helps sets some context as to why we have to be cautious -Alan |
|
Mailing list message from Rafael Winterhalter on core-libs-dev: As for the need to inject classes into 'new' packages, in Mockito we solve I remember the discussions about the dynamic agent attachment rather well, As for the proposed API, I understand that it needs thought, my hope is I think that the problem you want to solve is rather that JVMs should not Am So., 18. Apr. 2021 um 18:24 Uhr schrieb Alan Bateman <
|
1 similar comment
|
Mailing list message from Rafael Winterhalter on core-libs-dev: As for the need to inject classes into 'new' packages, in Mockito we solve I remember the discussions about the dynamic agent attachment rather well, As for the proposed API, I understand that it needs thought, my hope is I think that the problem you want to solve is rather that JVMs should not Am So., 18. Apr. 2021 um 18:24 Uhr schrieb Alan Bateman <
|
|
From: Alan Bateman
I think it would be easy to limit the use of this Instrumentation method to the agent code as agent classes are loaded by the bootstrap classloader. Simply make the method implementation caller-sensitive and check the caller is from bootstrap class loader. This way Instrumentation instance escaped to application code would not give that code any ability to define arbitrary classes. Good enough? Peter
|
|
This won't work as agents are loaded by the system class loader, not the
boot loader.
Peter Levart ***@***.***> schrieb am Mo., 19. Apr. 2021,
11:02:
… From: Alan Bateman
JDK-8200559 is about defining auxiliary classes in the same runtime
package at load-time whereas I think the proposal in this PR is adding an
unrestricted defineClass to the Instrumentation class. I think this will
require a lot of discussion as there are significant issues and concerns
here. An unrestricted defineClass might be okay for tool/java agents when
started from the command line with -javaagent but only if the
Instrumentation object is never ever leaked to library or application code.
It could potentially be part of a large effort to reduce the capabilities
of agents loaded via the attach mechanism. More generally I think we need
clearer separation of the requirements of tool agents from the requirement
of framework/libraries that want to inject proxy and other classes at
runtime.
I think it would be easy to limit the use of this Instrumentation method
to the agent code as agent classes are loaded by the bootstrap classloader.
Simply make the method implementation caller-sensitive and check the caller
is from bootstrap class loader. This way Instrumentation instance escaped
to application code would not give that code any ability to define
arbitrary classes.
Good enough?
Peter
Separately, the proposal in JEP 410 is to terminally deprecate
ProtectionDomain.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#3546 (comment)>, or
unsubscribe
<https://github.com/notifications/unsubscribe-auth/ABCIA4EYHK5OHTDAN3FKYULTJPWS5ANCNFSM43BSDEGQ>
.
|
Ah sorry, I didn't know that. So what about the following: agent is able to define (or load) a class from bootstrap loader. So it would be able to instantiate an intermediary class, loaded by bootstrap loader which would serve as an intermediary to call into this limited Instrumentation API point... |
The agent JAR file is added to application class path and is loaded using the system class loader. So almost always the defining loader will be the application class loader. In general it's a hard problem to try to balance the integrity and security of the platform with the needs of agents that do arbitrary injection and instrumentation. Specifying an agent on the command line with -javaagent is the opt-in to trust that agent and a defineClass that allows arbitrary injection is plausible for that deployment. As Rafael's mentioned in one of the messages, there is enough power in the existing Instrumentation API to do that in a round about way already. We don't have anything equivalent for agents that are loaded by tools into a target VM. I added the attach mechanism and the dynamic agent loading back in JDK 6 and regret not putting more restrictions around this. As it stands, it is open to mis-use in that an application or library can use the attach mechanism (directly or via the attach API in a child VM) to load an agent and leak the Instrumentation object. This is the genie that somehow needs to be put back in its bottle. One approach that I mentioned here to create is a less powerful Instrumentation object for untrusted agents. Trusted agents would continue to the full-power Instrumentation object. A less powerful Instrumentation object would not be able to redefine java.base or other core modules and would not be able to retransform classes in those modules. The option on the table during JDK 9 was to disable dynamic loading of agents by default but that was kicked down the road. I don't particularly like that option and I think we can do better. |
|
Sorry to be late to this party. I've been wanting to read this thread for a while but have bene too busy up to now. I have just a few comments I too was party to the discussions about agent capabilities and recall well the decision to gradually impose restrictions, the first one being to control dynamic agent loading. I was happy to accept that general decision and the specific one to limit the opportunity for an agent self-hoisting into the current JVM. However, a key part of the plan to move forward on restrictions was to provide an override switch. I'd very much like to see that retained as an option. I know that in some uses self-hoisting is much preferable to having to install the agent on the command line and I'd expect he same to be true for any other capabilities for which restrictions were adopted. Although it is true -- as Ron said - -that configuring -javaagent on the command line is always /possible/ there are actually many scenarios for agent use where deployment of an agent after startup is pragmatically much more desirable. An obvious use is trouble shooting, where you only want an agent in place when something goes wrong. That turns out to be critical to solving some seriously difficult support cases. The interesting use cases also fall under testing where self-hoisting of a test agent by a test framework can result in an enormous simplification of test configuration. Usage of Byteman for testing went through the roof with this capability in place. Never underestimate the degree to which even the most minimal configuration complexity puts off Enterprise java developers when it is multiplied by the test suite size of a large project. So, likewise with other restrictions on behaviour, I'm very happy to see them put in place for dynamically hoisted agents so long as there is still a command line override along the lines of the agent attach property that allows a dynamic agent to do all that a command line agent can. One other thing I'd like to correct is a point made in the discussion about agent code residing in the system loader. It is true that the main agent class gets loaded by the system loader. However, it is perfectly possible to ensure that all the rest of the agent code is loaded by the bootstrap loader. A Main class can add the agent jar to the bootstrap path and then load and use reflection to invoke an effective main routine on a bootstrap loaded SubMain class. Byteman uses this trick on request in order to allow it to instrument bootstrap classes. Because all the Byteman classes except for the original Main shell class are loaded by the bootstrap loader Byteman can safely inject references to the Byteman rule engine and Byteman exception classes into bootstrap code. |
|
Setting '-javaagent' is mainly an operations problem. Many build tools do
not allow to declare a test dependency this way as the life cycles are not
laid out for it, the internal repository location might be machine
dependent, for example, and it's difficult to point to a specific folder
and file reliably. In this case, I'd say it would be easier to specify a
parameter in the sense of '-Djdk.attach.allowAttachSelf=true' as it is a
static value. This would however only work for build tools that initiate a
new VM for running tests which is not overly attractive for simple builds
due to the overhead. Of course, Maven, Gradle and similar tools could set
this parameter by default for their own JVM, that part is likely
overcomeable but it will certainly create confusion about how to run
libraries that are omnipresent today and make the JVM ecosystem less
approachable.
What bothers me more is the tooling perspective for non-self-attached
agents. For example, Aaeron offers a Java agent that adds plenty of debug
logging to relevant lines of code. This affects method size and so forth,
with Aaeron as a high-performance tool for banking and finance which is
written very consciously with regards to the JIT, adding it directly was
not feasible. Normally this logging is therefore thrown into a VM in
retrospect, once an application starts failing and is already taken off the
load balancer. For such a post-mortem, it would be rather annoying to
realize that a JVM cannot be attached to with full capabilities if you
forgot to set some flag. And often you did of course not consider the VM
instance to fail, sometimes it takes months to get a JVM into this buggy
state. This would be fairly inconvenient to face.
Therefore, I really hope that the dynamic attach from 'outside' the VM will
survive without imposing limits and that rather the self-attachment problem
will be targeted as such. When I mention a 'jdk.test' module in the Mockito
context, I am also rather hoping to improve performance compared to
convenience. The problem with '-Djdk.attach.allowAttachSelf=true' is that
you still need to start a new VM etc. Since Java 9, running single tests
with Mockito has for example become much slower compared to Java 8. Despite
the startup performance improvements in the JVM. If one could avoid the
location-bound '-javaagent:...', but get the Instrumentation instance
directly, I think this would render a real performance improvement in
actual execution scenarios.
Am Mi., 21. Apr. 2021 um 20:42 Uhr schrieb mlbridge[bot] <
***@***.***>:
… *Mailing list message from Alan Bateman ***@***.***> on
core-libs-dev ***@***.***>:*
On 20/04/2021 21:26, Rafael Winterhalter wrote:
I have earlier proposed to offer a "jdk.test" module that
offers the Instrumentation instance via a simple API similar to Byte
Buddy's. The JVM would not load this module unless requested on the command
line. Build tools like Maven's surefire or Gradle's testrunner could then
standardize on loading this module as a convention to give access to this
test module by default such that libraries like Mockito could continue to
function out of the box without the libraries functioning on a standard VM
without extra configuration. As far as I know, mainly test libraries need
this API. This would also emphasise that Mockito and others are meant for
testing and fewer people would abuse it for production applications. People
would also have an explicit means of running a JVM for a production
application or for executing a test.
Helping testing is good but a jdk.test module that hands out the
Instrumentation object could be problematic. Is there a reason why the
runner for Mockito and other mocking frameworks can't specify -javaagent
when launching? I would expect at least some mocking scenarios to
require load time transformations (to drop the final modifier from some
API elements for example) so important to have the transformer set
before classes are loaded.
As for adding the API, my thought is that if the Instrumentation API were
to throw exceptions on some methods/arguments for dynamic agents in the
future, for example for retransformClasses(Object.class), this breaking
change would then simply extend to the proposed "defineClass" method. In
this sense, the Instrumentation API already assumes full power, I find it
not problematic to add the missing bit to this API even if it was
restricted in the future in the same spirit as other methods of the API
would be.
I think it would really hard to put this genie back into a bottle. It's
way more attractive to use that than the very agent oriented
redefineModule and retransformClasses.
I mentioned JNI as it is a well-known approach to defining a class today,
using a minimal native binding to an interface that directly calls down to
JNI's:
jclass DefineClass(JNIEnv *env, const char *name, jobject loader, const
jbyte *buf, jsize bufLen);
This interface can then simply be used to define any class just as I
propse, even when not writing an agent or attaching. This method makes
class definitions also already trivial for JVMTI agents compared to Java
agents. Unless restricting JNI, the defineClass method is already a low
hanging fruit, but at the cost of having to maintain a tiny bit of native
code.
Sure, if you are using native code then you have the full power of JVM
TI and JNI available. Project Panama is exploring how to restrict access
to native code, I think too early to say how this might extend to JNI.
-Alan
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#3546 (comment)>, or
unsubscribe
<https://github.com/notifications/unsubscribe-auth/ABCIA4HHMA67R24GWUPFVATTJ4MBRANCNFSM43BSDEGQ>
.
|
|
Mailing list message from Alan Snyder on core-libs-dev: On Apr 21, 2021, at 11:40 AM, Alan Bateman <Alan.Bateman at oracle.com> wrote:
I looked at some of the Panama documents and saw no hint that it might be extended to JNI. It seems to be positioned as an (partial) alternative to JNI. What I do see is that native code has no direct access to the JDK via Panama, only the ability to invoke provided upcalls, which can only access objects and methods that the caller has access to. That is indeed much more restricted than JNI. Even though it is ?too early?, can you explain why you think Panama?s restrictions might apply to JNI? As a library developer, I have often made use of JNI to work around limitations in current and older JDKs. I would hate to lose that ability. Alan |
1 similar comment
|
Mailing list message from Alan Snyder on core-libs-dev: On Apr 21, 2021, at 11:40 AM, Alan Bateman <Alan.Bateman at oracle.com> wrote:
I looked at some of the Panama documents and saw no hint that it might be extended to JNI. It seems to be positioned as an (partial) alternative to JNI. What I do see is that native code has no direct access to the JDK via Panama, only the ability to invoke provided upcalls, which can only access objects and methods that the caller has access to. That is indeed much more restricted than JNI. Even though it is ?too early?, can you explain why you think Panama?s restrictions might apply to JNI? As a library developer, I have often made use of JNI to work around limitations in current and older JDKs. I would hate to lose that ability. Alan |
|
@raphw This pull request has been inactive for more than 4 weeks and will be automatically closed if another 4 weeks passes without any activity. To avoid this, simply add a new comment to the pull request. Feel free to ask for assistance if you need help with progressing this pull request towards integration! |
|
@raphw This pull request has been inactive for more than 8 weeks and will now be automatically closed. If you would like to continue working on this pull request in the future, feel free to reopen it! This can be done using the |
|
@raphw and everyone else, I apologise for commenting on an auto-closed issue, but I have no other way of contacting anyone, because neither do I have an account to comment on JDK-8200559 nor am I a member in any other of the JDK development or "JVM rock star" circles for good reason: I simply have a very limited understanding of the topic at hand. I am, however, the current maintainer and project lead of Eclipse AspectJ, and since JDK 16 the weaving agent requires Here, there has been a lot of discussion about Byte Buddy and Mockito. For AspectJ, which is widely used in the industry, not just in Spring but also in countless other productive contexts, there still is no solution to the problem that we need to instrument a target class to call an auxiliary class (generated by the agent) that does not have a Lookup object on the target class to begin with. The target class is still being transformed, i.e. not loaded yet, and hence we cannot look it up. Is this case being considered? Can any of you advise me how to solve that problem? I am open for advice and hands-on chat, desktop sharing or audio/video call sessions. |
|
@kriegaex I resolved this problem in Byteman by opening up the module of the target class to a module (dynamically) created by Byteman. The Instrumentation instance provided to the agent allows you to perform such an opens operation at runtime rather than on the command line. Byteman relies on an API provided by this module to create the lookups and hand them back for use in woven code. n.b. although this relies on the module exporting a public API, that API can be secured by requiring callers to pass a non-null Instrumentation instance i.e. to have agent capabilities. |
|
Requiring such an API opens the module to anybody, though, punching a hole into the module boundary. BB currently opens the jdk.internal.misc.Unsafe class to a module on a seperate class loader that is not reachable outside an agent, using Instrumentation. This also caters the need to inject utility classes from an agent before any class file transformer is triggered, to maintain a well-defined life-cycle. Of course I'd prefer if there was a way to resolve a lookup from Instrumentation for a given class loader and for example a package-info class, however rendering the issue that packages might not exist (yet). |
How so? Any module created to print Lookups can easily rely on a shared secret to secure the API. Byteman employs a non-null Instrumentation object (a value which any agent ought to keep secret). However, it could just as easily have employed an arbitrary bit length hash key. The key can be used to initialize a module-private static long[] field of an API implementation class generated into the module i.e. the hole can actually be a keyhole in the shape of a key known only to the API client and implementation. |
|
Thanks @adinn, @raphw for your feedback. I am not pretending to fully understand what you just explained or to have the slightest clue how to do what you suggested, but reading it for the second time since yesterday seems to make it clearer already. I guess, I cannot hope for a how-to with sample code. But I can immerse myself into the topic some more next time I have a vacant time box and look at Byte Buddy and Byteman source code. |
@raphw, may I ask how? Is there any sample code that is not connected to the BB code base with its nested classes, interfaces etc.? I know, that caters nicely to the fluent DSL BB provides, but to me it is just a maze of code that is hard to comprehend. |
Agents should not be using the JDK's internal Unsafe. This needs to said in the strongest possible terms. For the discussion, it would be useful to provide a brief summary on what AspectJ is trying to do with this weaving. The JBS issue was originally about load time instrumentation defining auxiliary classes, as you might get at compile time when compiling a source file containing more than one class.. I can't tell from your comments here or in Eclipse 546305 if this is relevant to what you are trying to do or not. It may even be better to start a new discussion on serviceability-dev. |
|
@AlanBateman, the AspectJ weaving agent creates an auxiliary class to implement an "around" advice for a method, i.e. a method execution is intercepted and the user has options to do something before optionally calling the target method and then do something afterwards too. In doing so, she can modify method arguments before calling the target method, then also modify the result. Instead of calling the target method, she may also return a result of the expected type instead. Before, after or instead of calling the target method, she can also throw an exception. The target class is transformed in such a way to call the auxiliary class, which necessitates the the aux-class to be in the same classloader as the target class. But because the aux-class is defined while the target class is still being transformed during class-loading, we cannot have any look-up instance pointinmg to it yet. IMO, this is absolutely on topic, which is why the Bugzilla bug is linked to the JDK issue and was discussed with Mandy in this context. |
|
I migrate the day https://bugs.openjdk.org/browse/JDK-8200559 finds a solution (and fallback for older JDKs). I can start a new discussion, but briefly summarized from the last time this was on the mailing list:
|
|
@AlanBateman: As for using means to achieve user benefit that the JDK team deems unfit, nobody would do that, if there was an adequate way to provide such user value. Neither AspectJ nor Byte Buddy have a reputation of being some kind of shady hacker tools. Edit, because nobody posted in between yet: The JDK issue has been open for a long time, and the chance was missed to provide a canonical way to do what tool providers and users need for LTS 17, now again for LTS 21. I.e., even if in the future such JDK functionality will finally be provided, legacy support will carry on suboptimal, hacky solutions for many years. Please do not blame tool providers. Our customers simply want to be able to continue using tools without setting lots of extra JVM options. |
Right, this is what JDK-8200559 was originally about. Mandy and I discussed it several times and load-time instrumentation that defines auxiliary classes in the same run-time package is a reasonable addition. The more general request for an "unrestricted defineClass" conflicts with several ongoing efforts so we've had to kick that to touch. |
Thanks for finding some common ground. I appreciate it.
I better do not start to share my general opinion about JMS (edit: sorry, couldn't resist in the end) - I mean the ongoing effort that started with Jigsaw - in detail, because that would end up in a flame war. Let me just say, that "well-meant" is not the necessarily same as "constructive", "helpful" or "necessary". Back when Jigsaw was designed, it seemed to be a good idea. But it has been introduced a long time ago, and all my enterprise customers since then are trying to ignore its existence as much as they can, mostly seeing it as an impediment or at least a major annoyance. I think, one of the reasons why Python gained so much traction for enterprise-level big data and machine learning stuff is that you have a dynamic language environment in which you can pretty much do whatever you want and just get things done. Those same enterprise customers want to use the tools of their choosing with as many and as easy to use features as possible. They do not trade those features for security, but implement security in a different way, such as devising a micro service architecture and isolating containers from each other in OpenShift or what have you, defining service mesh rules on top of that, if necessary. Deprecating security managers was a laudable idea, but adding those unnecessary restrictions in the JVM in exchange was kind of one step forward, two steps back. They just get in the way of getting useful things done. This is what a language and a VM are for: getting things done in a productive way with as few barriers as possible. Instead, with every new Java release, users and tool providers alike are forced to jump through more hoops. There is so much great, innovative stuff in the JVM and Java SE API. E.g., virtual threads are the greatest thing since bread came sliced. We want more of that and fewer barriers. Streams, NIO, the whole lambda and functional bunch of stuff, structured concurrency, better GC, records and neat ways to deconstruct them etc. - wow, great stuff. But new "security features" (a.k.a. restrictions) like modules and sealed classes - not so great from a productivity perspective. Since JDK 9, I have seen generations of developers trying to be more productive not because but despite those new JVM "features". I know that my perspective is quite subjective and might be biased. But as an agile coach dealing mostly with developers in the JVM ecosystem, I have yet to find a team in which more than a tiny minority thinks that JMS is useful. That is not because they are lazy to learn or do not want to leave their confort zones. It is, because the JVM makes it progressively harder to get things done instead of treating users like adults and letting them take responsibility for security as they see fit. Enforcing things like a helicopter parent is not the smartest move. I apologise for polluting the thread with what ended up to be too much text that is related to the topic, but not exactly focused on the problem at hand. |
|
@kriegaex Luckily, you and your customers are not obliged to use the JPMS, nor find it useful for whatever libraries or apps you write or deploy. However, the fact that you or many other programmers do not use it does not mean it has not been a success. Anyone deeply involved with JDK and/or JVM development in recent years knows that it has been and continues to be critical to maintaining and extending the Java platform. Regarding my previous comment about Byteman using its own dedicated, dynamic module to provide secure access to MethodLookup instance you might want to look at the relevant code. It relies on a sanctioned API of Instrumentation that was introduced as part of the negotiation of JPMS integration precisely to allow agents to interact with and reconfigurethe module system at runtime. The resulting Byteman code provides a simple API that allows methods to be executed indirectly, either via reflection in jdk8- or via method handles in jdk9+. You can see the details of how I achieved this in the Byteman layer and jigsaw subdirectories. |
It seems like you could lazily define the aux class when the target class first needs it, instead of eagerly while the target class is being defined. e.g. generate the class bytes for the aux class up front, and embed them in the target class (For instance as a Base64 encoded string, which fits well into the constant pool). Then, have the transformed code in the target class define the the aux class when it is first needed, at which point you do have access to a lookup. |
They are obliged to deal with it, and so am I as a tool maintainer. Just look the the approaches mentioned here. They all are in the category which in German we would call "von hinten durch die Brust ins Auge". That literally translates as "(a shot) from behind through the chest into the eye". Think Kennedy and "magic bullet". It means that something is unnecessarily complicated. But it is not of any developer's own choosing, but because the Java API and runtime environment requires them to jump through hoops. That is exactly what I meant before. Bytecode transformation should not be rocket science, but it progressively is developing in that direction. I have seen what Byte Buddy does to get access to an Unsafe instance, thought about what @JornVernee just suggested and have yet to look into the ByteMan source code. This all sounds pretty much like black magic and a maintenance nightmare to me. A language ought to provide means to be productive and maybe offer some (opt-in) guard railing, but not be a corset so tight that I cannot breathe anymore. I need something that supports me without strangling me. |
|
I tried to give the discussion a fresh start but moved it to the mailing list: https://mail.openjdk.org/pipermail/core-libs-dev/2024-January/118622.html I have given this some thought the recent years, so I hope that there is a solution in reach that satisfies everybody's concerns and needs. |
What stops people from supplying a fake instance? Wouldn't you need to "test run" the instance first? |
Not necessarily. When the generated API implementation relies on the capabilities of class However, if you want the implementation to validate an incoming call you can easily arrange for that. For example, provide a method on the agent class that says yes to its own instance and no for any other instances e.g. . . . Method validate can be used to ensure API calls only proceed when invoked by the agent or code that the agent trusts. |
In passing, Instrumentation was a candidate to be sealed at one point as the only implementations should be in the java.instrument module. I haven't seen any delegating or other implementations but they might exist so we would need to be careful with compatibility. |
|
There's plenty of them in Byte Buddy and I have seen a bunch in other agents. |
Hmm? Bytecode transformation of the JDK runtime implementation is a lot more complicated than your comments seem to acknowledge and, here's the important thing, it always has been. You need to remember that instrumenting JDK runtime code involves rebuilding the engine that you are driving while you are in mid-flight. If you think there are few-to-none hidden gotchas involved in doing that then I suggest you are significantly underestimating the opportunity for things to go wrong -- not just when it comes to instrumenting some specific release of OpenJDK but also when it comes to keeping instrumentation working across legacy and future releases, with all the variety of moving parts that the (necessary) development of the platform requires. The same observation explains why project Jigsaw was needed. The danger of clients using internal JDK runtime APIs -- especially the core runtime APIs -- is much more subtle than many of the programmers who have routinely relied on it recognize. The biggest threat is that public runtime APIs are often implemented via calls to multiple internal APIs -- which may themselves involve multiple entries and re-entries to the JVM. It has always been (and always will be) the case that an isolated call from a client to an internal API can leave the JDK runtime and/or the JVM in an incoherent state because correct use of that internal API requires a correct sequence of invocations with matched inputs and outputs. It is easy even for OpenJDK developers to fails to get this right, especially when calls involve entry to the JVM. The possibility for a programmer who is not very familiar with the JDK runtime code and the JVM code to get it wrong are significant. Worse, the problems may not manifest immediately or in all cases so the danger can be unapparent until disaster strikes. |
That is, because I was not talking about JDK runtime transformation but about what the AspectJ weaving agent does: transformation of application classes during class-loading. I am aware of the fact, that it is also possible to retransform already loaded classes, as a special case also bootstrap ones from the JRE. Of course, this is more complicated than the simple case. But my point was, that even the simple case is not simple, if I need to define classes in an arbitrary class loader - not because technically it is difficult, but simply because the JRE API to do so is more and more sealed off with each new Java release. This is also what I mean, when I said, that developers are not treated as adults but "protected" by well-meaning, but ill-doing helicopter parents.
No, byte code transformation is not complicated per se. Getting the transformed classes where they need to be is complicated, but artificially so.
No, I do not need to remember, because I am aware of that fact. It is just off-topic here with regard to what I asked about. But that other use case, which I have experimented with in another context (test mocking and stubbing) in the past, is an intriguing one, too. I am not underestimating anything there, but for AspectJ it is simply out of scope. Should I ever decide to add the capability to weave aspects into JRE classes, of course that will up the complexity by a notch or two. BTW, if developers want to experiment with instrumenting JRE classes during runtime and in doing so crash the engine against the wall, they are still adults. They might learn something along the way, e.g. how not to do certain things. But they also have a chance to discover how to do it right. |
To allow agents the definition of auxiliary classes, an API is needed to allow this. Currently, this is often achieved by using
sun.misc.Unsafeorjdk.internal.misc.Unsafeever since thedefineClassmethod was removed fromsun.misc.Unsafe.Progress
Issue
Reviewing
Using
gitCheckout this PR locally:
$ git fetch https://git.openjdk.java.net/jdk pull/3546/head:pull/3546$ git checkout pull/3546Update a local copy of the PR:
$ git checkout pull/3546$ git pull https://git.openjdk.java.net/jdk pull/3546/headUsing Skara CLI tools
Checkout this PR locally:
$ git pr checkout 3546View PR using the GUI difftool:
$ git pr show -t 3546Using diff file
Download this PR as a diff file:
https://git.openjdk.java.net/jdk/pull/3546.diff