Confusing duplicates in AnnotationParser.addParsedClass() #2072
Labels
Stale
For auto-closed stale issues and pull requests
Comments
|
Okay, I take that back. For some reason, Eclipse / m2e adds the JRE twice to the classpath. Not sure what is going on there. It might still be a good idea to skip indentical JARs; the classloader is also only loading the first one. |
|
It might still be a good idea to skip indentical JARs... it is kind of annoying anyway. Same in 9.4.9 |
|
@phax can you give an example, in 9.4.9, where you see identical JARs? |
|
Sure. See the attached file |
|
@phax can you please do a run with dumpAfterStart=true and post the log file: I want to see your server classpath, webapp classpath and the containerIncludeJarPattern settings. So far as I can see in your last post, these are not duplicate jars, but the same class included in multiple jars on a classpath. |
|
Example for a duplicate are e.g. the JavaFX things: dumpAfterStart is what? System property or ???? |
|
@phax if you have a start.d directory, look in server.init and uncomment jetty.server.dumpAfterStart=false and change it to true; if you've got an amalgamated start.ini, find the same line and change it there and rerun. If you're running this embedded, then call setDumpAfterStart(true) on your Server instance. And yes, I see the duplicates that seem to all come from scanning the jdk, twice. It shouldn't be scanned at all, so I assume that you've got a too-permissive containerIncludeJarPattern like "*", and also have the jdk jars on the classpath twice (are you running this from eclipse?). |
|
Thanks, I just manually called |
|
And yes you are correct on the containerIncludePattern. Based on previous issues, I have the following in my code: // http://www.eclipse.org/jetty/documentation/9.3.x/configuring-webapps.html#container-include-jar-pattern
// https://github.com/eclipse/jetty.project/issues/680
aWebAppCtx.setAttribute (WebInfConfiguration.CONTAINER_JAR_PATTERN, ".*\\.jar$|.*/classes/.*"); |
|
@phax many of those seem to be from your |
|
@phax, yes, well there's your problem: your containerIncludeJarPattern includes everything in the whole container class hierarchy, and as you see the jre jars are on there twice, once on the AppClassLoader and once on the ExtClassLoader. If you refine your containerIncludeJarPattern to only include the jars you want scanned you will save yourself a bunch of time at startup. |
|
@phax if you're running from within Eclipse, it seems to populate the ExtClassLoader with lib/ext jars as well as put them on the AppClassLoader. |
|
@joakime No, the JDK is "Standard Windows 64 Bit JDK" - no mods at all @janbartel It seems to be an m2e launcher problem as stated by @digulla originally Thanks for your help guys!!! |
|
Interesting ... I do not have |
|
Here's my "tree /F" from that JDK folder if it helps: Windows 10, version 1709, Build 16299.371 with local administrator rights |
|
i get same issue on my product. I get it when run from eclipse also standalone, so isn't by m2e i get two pattern of warning: pattern 2 |
|
@hieplq it looks like you have a hybrid environment, some of it on normal JVM classpath / classloader, and some of it using OSGi. pattern 1 with better formatting pattern 2 with better formatting Those are both 2 different URI locations (hence the warning). Your scenario looks like an improper use of OSGi. |
|
i add "-verbose:class" to get info of class loader |
|
bytecode scanning doesn't load the class, it iterates through the entire classloader and opens each *.class as an inputstream for bytecode scanning. |
|
"Try connecting to your JVM via JMX and explore your classloaders" please help me what's tool or document or keyword to search |
|
@janbartel perhaps there could be some additional logging about classloaders? or better filtering for osgi use case? |
|
i see jetty scan from two path
|
|
@hieplq we scan from the container classpath ONLY those jars that have a pattern that matches the pattern specified by the ContainerIncludeJarPattern as documented here: http://www.eclipse.org/jetty/documentation/9.4.12.v20180830/using-annotations.html#jars-scanned-for-annotations Can you please do a server dump (server.setDumpAfterStart(true), or the equivalent with the jetty.server.dumpAfterStart=true jetty property. |
|
my ContainerIncludeJarPattern=./[^/]servlet-api-[^/].jar|./javax.servlet.jsp.jstl-..jar|./org.apache.taglibs.taglibs-standard-impl-.*.jar my server dump: |
|
I've tried just this from a Linux machine with Eclipse 2019.06 (4.12). jetty-9.4.8.v20171121 With this version, rt.jar isn't twice on the classpath. So it's either fixed in Eclipse m2e somewhere or it only happens on Windows. Can someone else confirm? |
above code will consider osgi bundle as jar on WEB-INF/lib and add to jar list
super.parseWebInfLib(context, parser);
if (fragAndRequiredBundles != null)
{
//scan the required bundles
for (Bundle requiredBundle : fragAndRequiredBundles)
{
//skip bundles that have been uninstalled since we discovered them
if (requiredBundle.getState() == Bundle.UNINSTALLED)
continue;
if (requiredBundle.getHeaders().get(Constants.FRAGMENT_HOST) == null)
{
//a bundle indeed:
parseRequiredBundle(context, oparser, webbundle, requiredBundle);
_webInfLibStats.increment();
}
}
}super.parseWebInfLib(context, parser) parse jar find on step 1 (include Osgi bundle) |
|
work-around for above duplicate, i can add below configuration to org.eclipse.jetty.deploy.DeploymentManager <Call name="setContextAttribute">
<Arg>org.eclipse.jetty.server.webapp.WebInfIncludeJarPattern</Arg>
<Arg>^((?!/plugins/).)*$</Arg>
</Call>but i still get duplicate by bellow code on org.eclipse.jetty.osgi.annotations.AnnotationConfiguration.parseWebInfLib if (!context.getMetaData().getWebInfJars().contains(bundleRes))
{
context.getMetaData().addWebInfJar(bundleRes);
}on this public void parseWebInfLib(WebAppContext context, org.eclipse.jetty.annotations.AnnotationParser parser)
throws Exception
{
AnnotationParser oparser = (AnnotationParser)parser;
if (_webInfLibStats == null)
_webInfLibStats = new CounterStatistic();
Bundle webbundle = (Bundle)context.getAttribute(OSGiWebappConstants.JETTY_OSGI_BUNDLE);
@SuppressWarnings("unchecked")
Set<Bundle> fragAndRequiredBundles = (Set<Bundle>)context.getAttribute(OSGiWebInfConfiguration.FRAGMENT_AND_REQUIRED_BUNDLES);
if (fragAndRequiredBundles != null)
{
//index and scan fragments
for (Bundle bundle : fragAndRequiredBundles)
{
//skip bundles that have been uninstalled since we discovered them
if (bundle.getState() == Bundle.UNINSTALLED)
continue;
Resource bundleRes = oparser.indexBundle(bundle);
if (!context.getMetaData().getWebInfJars().contains(bundleRes))
{
context.getMetaData().addWebInfJar(bundleRes);
}
if (bundle.getHeaders().get(Constants.FRAGMENT_HOST) != null)
{
//a fragment indeed:
parseFragmentBundle(context, oparser, webbundle, bundle);
_webInfLibStats.increment();
}
}
}
//scan ourselves
oparser.indexBundle(webbundle);
parseWebBundle(context, oparser, webbundle);
_webInfLibStats.increment();
//scan the WEB-INF/lib
super.parseWebInfLib(context, parser);
if (fragAndRequiredBundles != null)
{
//scan the required bundles
for (Bundle requiredBundle : fragAndRequiredBundles)
{
//skip bundles that have been uninstalled since we discovered them
if (requiredBundle.getState() == Bundle.UNINSTALLED)
continue;
if (requiredBundle.getHeaders().get(Constants.FRAGMENT_HOST) == null)
{
//a bundle indeed:
parseRequiredBundle(context, oparser, webbundle, requiredBundle);
_webInfLibStats.increment();
}
}
}
} |
|
In the osgi tests that we regularly run on jetty, we don't see any duplication of classes. Can you provide more details about your deployment that would explain the difference? |
|
found it can you add a Require-Bundle to web project and test? Require-Bundle: org.eclipse.jetty.apache-jsp;bundle-version="0.0.0" |
|
I see. The solution is not to add jsp to the Required Bundles: this effectively puts it onto the webapp's classpath, which is incorrect, even in non-osgi jetty. The correct thing to do is to deploy the jetty-osgi-boot and jetty-osgi-boot-jsp bundles into your osgi container, and they will form the equivalent of the server's classpath. |
|
you are correct about org.eclipse.jetty.apache-jsp, i can remove it from Required Bundles but i has many bundles still need to use Required Bundles, example zk library (a library for build up web ui from java) i get org.eclipse.jetty.apache-jsp for easy setup reproduce issue |
|
back to this block of code so for loop after public void parseWebInfLib(WebAppContext context, org.eclipse.jetty.annotations.AnnotationParser parser)
throws Exception
{
AnnotationParser oparser = (AnnotationParser)parser;
if (_webInfLibStats == null)
_webInfLibStats = new CounterStatistic();
Bundle webbundle = (Bundle)context.getAttribute(OSGiWebappConstants.JETTY_OSGI_BUNDLE);
@SuppressWarnings("unchecked")
Set<Bundle> fragAndRequiredBundles = (Set<Bundle>)context.getAttribute(OSGiWebInfConfiguration.FRAGMENT_AND_REQUIRED_BUNDLES);
if (fragAndRequiredBundles != null)
{
//index and scan fragments
for (Bundle bundle : fragAndRequiredBundles)
{
//skip bundles that have been uninstalled since we discovered them
if (bundle.getState() == Bundle.UNINSTALLED)
continue;
Resource bundleRes = oparser.indexBundle(bundle);
if (!context.getMetaData().getWebInfJars().contains(bundleRes))
{
context.getMetaData().addWebInfJar(bundleRes);
}
if (bundle.getHeaders().get(Constants.FRAGMENT_HOST) != null)
{
//a fragment indeed:
parseFragmentBundle(context, oparser, webbundle, bundle);
_webInfLibStats.increment();
}
}
}
//scan ourselves
oparser.indexBundle(webbundle);
parseWebBundle(context, oparser, webbundle);
_webInfLibStats.increment();
//scan the WEB-INF/lib
super.parseWebInfLib(context, parser);
if (fragAndRequiredBundles != null)
{
//scan the required bundles
for (Bundle requiredBundle : fragAndRequiredBundles)
{
//skip bundles that have been uninstalled since we discovered them
if (requiredBundle.getState() == Bundle.UNINSTALLED)
continue;
if (requiredBundle.getHeaders().get(Constants.FRAGMENT_HOST) == null)
{
//a bundle indeed:
parseRequiredBundle(context, oparser, webbundle, requiredBundle);
_webInfLibStats.increment();
}
}
}
} |
|
For the bundles that are named in your Require-Bundle statements, are these also build into your war? That is, are they also in WEB-INF/lib? Because that's the only way I've been able to generate duplicate warnings. |
|
no. my product same like eclipse. all bundles deploy on plugins folder |
|
i setup product use bndtool and also can reproduce this issue with Require-Bundle |
|
@hieplq I really can't reproduce a problem using jetty-9.4.30. Can you please post the manifest of your webbundle, and the manifests of the bundles that are being scanned twice please? |
hieplq
added a commit
to hieplq/vn.cxf.client.demo
that referenced
this issue
Jun 25, 2020
|
hi @janbartel i'm create a test project here https://github.com/hieplq/vn.cxf.client.demo
|
|
This issue has been automatically marked as stale because it has been a full year without activity. It will be closed if no further activity occurs. Thank you for your contributions. |
|
@hieplq sorry, your last post slipped through the cracks - blame covid! I've tried to follow your repro steps, but at step 3 I get: I don't use gradle, so I'm stuck unless you can help? |
|
@hieplq the code area you referenced in your previous post cannot be the problem for |
|
thank @janbartel i will check build issue of vn.cxf.client.demo also output of DefaultFileLocatorHelper.getBundleInstallLocation() and report you soon |
janbartel
added a commit
that referenced
this issue
Jul 1, 2021
Signed-off-by: Jan Bartel <janb@webtide.com>
|
@hieplq I've prepared a tentative fix in this branch: https://github.com/eclipse/jetty.project/tree/jetty-10.0.x-2072-dup-osgi-annotation-parsing It would be great if you could run it in your environment and check it works for you. |
|
hi @janbartel
debug info: detail log g! Jul 10, 2021 8:49:04 AM org.apache.aries.spifly.BaseActivator log |
|
run bellow command to redo error git clone -b jetty-10 git@github.com:hieplq/vn.cxf.client.demo.git |
|
i'm fixed Unable to load class org.eclipse.jetty.servlet.listener.ELContextCleaner confirmed: jetty 10.0.6 resolved issue without your patch output log Welcome to Apache Felix Gogo g! Jul 10, 2021 4:28:27 PM org.apache.aries.spifly.BaseActivator log |
|
This issue has been automatically marked as stale because it has been a |
github-actions
bot
added
the
Stale
|
This issue has been closed due to it having no activity. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
The method
addParsedClass()ofAnnotationParserreports duplicates on the classpath.The current behavior is confusing because it reports true and invalid duplicates. After upgrading from Jetty 9.3.9 to 9.4.8, I got a ton of outputs. I give just a few examples here (out of ~ 10'000):
Correct duplicate. I want to see those because they indicate duplicate classes on the classpath.
It's harmless since these are interfaces which didn't change for a long time. But then I saw this:
Nice, two versions of the same framework. That needs to be fixed. So this warning is useful.
But I also get roughly 9'000 (90%) of these:
These are shown for every class in the JRE (I saw
jre/lib/ext/access-bridge-64.jar,jre/lib/ext/cldrdata.jar,jre/lib/ext/jfxrt.jar,jre/lib/ext/sunec.jar,jre/lib/ext/nashorn.jar, and more)I'm 99.9% sure that I didn't add the JRE twice to the classpath.
I think the fastest way to fix this is to use a Set (or LinkedHashSet) instead of a List or use a loop instead of
existing.addAll(list);and to omit same locations. But that depends on whether the behavior is a bug (see comments in #1819 about different threads adding classes twice).Maybe you can check and remove duplicate JARs on the classpath first, or first build a list of all resources, check that for duplicates and then start scanning them.
Jetty v9.4.8
This is related to the fix made in #1819
The text was updated successfully, but these errors were encountered: