-
|
Hello, At the moment, Having to register all the potential resources seems too tedious. I wonder if an option to exclude specific code-paths, classes, or packages from the cc @vjovanov |
Beta Was this translation helpful? Give feedback.
Replies: 7 comments 13 replies
-
|
I'm a bit unclear what the request is. From a technical standpoint, native-image must know about a resource at buildtime to be able to package it into the binary for runtime. Otherwise, it's up to the program to read the data from an external source (e.g. file, database). In previous versions of native-image, it was possible to manually specify sets of resources using a regex. I think that went away with the https://www.graalvm.org/latest/reference-manual/native-image/metadata/ also mentions |
Beta Was this translation helpful? Give feedback.
-
Correct, the issue manifests when the program tries to access some resource at runtime that we don't necessarily need and thus didn't register. One could argue that the right thing to do in such cases is to still register the resource and just don't provide it at build-time (in order to not include it in the binary). However, this becomes pretty tedious when a library tries to access a resource by trying different names and/or different paths. E.g. a library might try to load files matching a pattern like
Maybe. A regex/wildcard/glob would work-around the issue by essentially registering all FWIW
Correct, but the intention is to ultimately make |
Beta Was this translation helpful? Give feedback.
-
|
@zakkak I am also considering to relax the current setting. I feel we have two major culprits here:
I am not in favor of adding a special flag to make things work (for debugging it is OK), as this moves Native Image away from the Java semantics. My plan is to experiment with these three approaches in the next 2 months and then we can decide what is easiest to support and use. @zakkak what are your thoughts? |
Beta Was this translation helpful? Give feedback.
-
That indeed works well for cases where we expect the various resources to reside on a single folder (which is known at build-time), however there are cases where different folders are visited as well and to make matters worse these folders might be unknown at build-time. cc @radcortez
+1, I have also seen the pattern described in jobrunr/jobrunr#1101 (comment) used in more than one places as well. Users interested in a method, instead of explicitly trying to reflectively get the specific method, load the class and invoke
I guess there is a typo here and the exception would be thrown when the
Packages without wildcards might do the trick as well. It's not clear to me what this would do though? Would it allow us to reflectively access any Class in What we are more interested in is allowing any Class/Method under
Don't we already have that with
I was thinking about extending the {
"reflection": [
{
"type": "io.smallrye.config.MyClass",
"uncheckedReflectiveAccesses": true
}
],
"resources": [
{
"type": "io.smallrye.config.MyClass"
"uncheckedReflectiveAccesses": true
}
]
}which would result in ignoring unregistered accesses from
I am afraid 1 and 2 are going to be too expensive (in terms of bloat). |
Beta Was this translation helpful? Give feedback.
-
|
How about To load configuration, we use this API to search for optional resources, since it either returns an enum with the found resources or an empty one if no resources are found. Right now, if you try with In Quarkus, we discover these resources during build time, and byte code generate a programmatic representation of the files, so we don't need the resources. We leave the method call from the library to return an empty enumeration, which works fine. We could register the resources, but the library has semantics where additional resources can be added at runtime, which are searched of both classpath, and file system. Of course, during native image, you can't add resources in the classpath, but the library is not aware of that, and it searches both paths. In this case, we cannot register the resources, because they are unknown to us during the build. I understand the decision for APIs as |
Beta Was this translation helpful? Give feedback.
-
|
The awkwardness comes with a benefit that you know that the native execution will behave exactly the same as the JVM. I would always trade a few more entries in the JSON file (that are explicitly mentioned in the error), for hours spent in debugging with How many resource entries do you need to add in this case? Could you please share a link to the pattern that is used here? Have you tried using the resource patterns with |
Beta Was this translation helpful? Give feedback.
-
|
In our case, we support the resource as a runtime configuration: For example, you can add Obviously, the classpath lookup will not discover any resources in native mode, but this is fine. I can work around the error by providing a substitution, but I feel that |
Beta Was this translation helpful? Give feedback.
-
Regarding bloat, I find that the biggest problem here are not the application classes, but the JDK classes. In a simple
Yes it would include all of these types to be available for
We do, exactly, just the default exception for
When you boil it down, this is a workaround for a lack of Maybe we could throw This way libraries would be able to handle this and we don't need extra JSON. In the future if a
|
Beta Was this translation helpful? Give feedback.
-
In theory I agree, but this approach assumes that libraries will be willing to include code specifically for handling GraalVM native-image compilation, which is not always the case. Metadata (like substitutions) allow us to work around this and support libraries that are not willing to include GraalVM specific code. |
Beta Was this translation helpful? Give feedback.
-
We could make a flag for this, but I would leave it as experimental for the time the third-party libraries don't adopt the new semantics. |
Beta Was this translation helpful? Give feedback.
-
|
Have you considered branching the logic of the original code on the We created an enum around the possible values for convenience. |
Beta Was this translation helpful? Give feedback.
-
|
@nreid260 if I get this right, you suggest adding a check to guard resource accesses, e.g. That's indeed an option when you have write access (or can convince the maintainers to merge this kind of change), i.e. this approach assumes that libraries will be willing to include code specifically for handling GraalVM native-image compilation, which is not always the case. As a result, in some cases we would need to introduce substitutions to apply this. |
Beta Was this translation helpful? Give feedback.
-
|
Yeah this is the basic idea. It's fairly natural to me that some codepaths which work for JVM and need to be changed for native. For libraries you don't control, you could do a little bytecode rewriting (e.g. with ASM) to insert the needed logic. Of course this takes effort, but possibly less effort than other options. I also wonder if library authors would be sympathetic to simpler resource handling logic even outside the context of graalification. They might support args like "skip loading external config files" or whatever. |
Beta Was this translation helpful? Give feedback.
-
|
Yes, I full agree with the argument here. I would always strive to have a simple solution that is the same both with native and on the JVM. For example, in radcortez's example above it would be nice to look only on the file path and skip resources lookup altogether. But if this is not possible for some reason, we can always make a condition on |
Beta Was this translation helpful? Give feedback.
-
|
We discussed this with @radcortez and it's still problematic. Imagine the following scenario: The library allows users to load arbitrary files either from the classpath or the filesystem. The user defines the file name through a property called In JVM mode everything works as expected. The user sets In native mode the value of Substituting the logic and avoiding the classpath lookup works around the issue, but behaves differently than the JVM mode, since in native mode the classpath is never searched. That sounds fine since even in the current state that allows searching in the classpath the behavior would be different if the resource was not registered. The problem arises if an application developer using the library registers some resources, e.g. So far the only backwards compatible approach we could come up is to substitute the library code to introduce a try-catch that will handle the |
Beta Was this translation helpful? Give feedback.
-
|
In general, to make an informed decision it would be good to give examples of how many entries relative to all metadata entries need to be added for the project to work. If the number of entries is relatively small compared to the size of the project (e.g. <1-2% of all entries), I feel the benefits of strict semantics outweigh the cost of maintaining the extra entries (especially because the missing entries are pretty easy to find). |
Beta Was this translation helpful? Give feedback.
-
|
It looks like there are cases that the main issue is not the number of entries but the difficulty of figuring out what that entries should be. One example is given in #10264 (reply in thread) and I just stumbled upon another interesting case, this time in the JDK itself. The issue in this case is not related to resources, but constructors. In Are there any suggestions on how to approach this issue? |
Beta Was this translation helpful? Give feedback.
-
|
After a few iterations we have concluded the following:
In the example you provided with resources, if there is a test that passes a system property that leads to reading the resource in many different places, I feel it is OK that this test also provides the metadata for those places. The end users will eventually not need to do it, but the frameworks will have to go this extra step. If really cumbersome, one can always use wildcards in |
Beta Was this translation helpful? Give feedback.
-
|
@vjovanov can you please clarify what functionality of |
Beta Was this translation helpful? Give feedback.
-
|
Based on our experience in the community, the things that will not be promoted to default are special exceptions for End users will always be able to enable the strict mode to keep their metadata strict or to debug when the applications don't work. Framework and library authors should always use the strict mode to make sure all metadata is accounted for. |
Beta Was this translation helpful? Give feedback.
After a few iterations we have concluded the following:
--exact-reachability-metadatawill remain very strict in order to guarantee that all entries are covered by the metadata. This includes resource lookups andClass.forNameon non-existing classes. This flag should be used by the frameworks and libraries to make sure that they have covered all possible entries in the code and that their users don't have to do it.Most of the functionality
--exact-reachability-metadatacurrently covers will be promoted to the default behavior (e.g., throwing aMissingReflectionRegistrationErrorwhen callingClass.getDeclaredMethods()on an unregistered class). With this, the end user will not care a…