JPMS Support

[jmcc0nn3ll]

This can be a root issue for ongoing efforts for Jetty to support JPMS features in Java.

Jetty has had OSGi support for nigh on a decade and it is not something we can simply abandon in lieu of java adding on a module management system. While that community has made strides in integrating well with the JPMS mechanics at the point of this thread creation it has largely been up in the air as to what that should mean for the Jetty project itself.

Jetty 9.4.x supports the usage of Java 9 with the module system disabled and it remains to be seen what Jetty 10 will do with it. Lets have some discussion on this thread about that and maybe start getting this broken down a bit.

@guw We would welcome your input here as well as a historical user of jetty and osgi, your plans regarding jpms and what you might expect from jetty regarding it.

We also have @PashaTurok with #2189 as the first user requesting this support and there is some info in that issue.

Additionally, to set expectations this is not something that will happen overnight, especially since it seems that it will involve a fairly substantial review of the overall artifact layout of Jetty.

[PavelTurk]

@jmcc0nn3ll Could you explain

fairly substantial review of the overall artifact layout of Jetty.

If Jetty supports OSGi - then there are already modules (OSGi bundles). JPMS is like OSGi with, I would say, three main differences - no module lifecycle, nor package versioning, no extra framework between modules and JVM. For all our projects we used principle - existing OSGI bundle becomes JPMS module.

[jmcc0nn3ll]

In a perfect world Artifact would equal Module but they serve different purposes. We worked through that in OSGI through copious amounts of carnage in manifest files. It remains to be seen what effect that has with jpms but it would be simplistic to think there is a trivial path forward where Artifact = OSGI Module = JPMS Module.

[gregw]

To correctly use a module system we really need use cases from users such as @pashaturok. Questions that we need to consider include:

• why isn't the auto module handling of jetty sufficient? What are the features that it does not provide.
• should jetty be one big module or many module? Should we match OSGI?
• how are dependencies such as the servlet API to be handled? Should we make them modules or are others doing so?
• what about 3rd party libraries like JSP? Do we wait until they are modules? So module them? Or make our own modules if them?

[guw]

Since I've been called out, here are my two cents. :)

IMHO, mapping OSGi bundles 1:1 to Java modules does not make sense from a Jetty perspective. When working with Jetty in the OSGi world, I always found myself using at least a minimum set of multiple Jetty bundles (jars). The future Jetty layout should be re-thought. My personal preference clearly goes towards less individual, small pieces, less dependencies, less complexity.

FWIW, I think the currently Maven structure is more a reflection of the Jetty architecture then of the Jetty distribution, i.e. it exists to support architecture boundaries within the Jetty code base and not because of Jetty's distribution layout. Those jars have then been mapped to OSGi bundles 1:1. I don't think any other strategy has been applied when creating OSGi bundles.

Before defining Jetty modules, the use cases have to be defined. I doubt that there will be a one size fits all model. For example, the modules provided by Jetty might fit only a certain percentage of use cases. Consumers of Jetty should not be afraid of building their own modules out of Jetty classes. I understand this is difficult currently as there is no convenient tooling yet.

There is currently also no answer to how EE next is going to deal with and define modules. For example, will there be a Servlet spec module? Are application servers supposed to depend on the Servlet spec module or are they supposed to provide the Servlet API as part of an application server module?

You also have to keep in mind, that dependencies between Java modules are hard wired to a module name, i.e. a web application with a dependency on a "Jetty" module will likely not run in Tomcat.

Last but not least, there is another aspect of JPMS that is part of JPMS support but currently completely undefined: layers. Layers become important when dealing with conflicting module dependencies. From a Jetty (and EE next) perspective, this could be web application loading. But then again, it's very, very unclear currently, if a future "application server" will still host multiple different web applications or if any such concept is discarded.

[PavelTurk]

@guw About layers I gave my variant here #2189 . I thought and came to conclusion that using separate layer for every site is the best solution - all modules of every site are independent (we can have N sites) and besides it is possible to create and destroy layers dynamically. It is very interesting what others think about such approach.
Note - I use term "site" but not "web application" as I consider the following situation- site includes two modules - front.war (web application) and logic.jar (that provides JPMS services to web application). Obviously, that logic.jar I want to use and for non web applications. By the way, when I tried to create layer with .war archive JPMS didn't want to do it - as I understand it wants only .jar

[guw]

@PashaTurok Clearly, the concept of "site" seems very specific in the context of a specific use case. Thus, it needs to be defined as well, i.e. is site a reachable endpoint or does it resemble something that we today know as "war"? But then again, in a service scenario ("micro service" deployment), layers may not be wanted at all - not even a single one.

Thus, I think before deciding about what to implement, I'd like to see a discussion first about what is needed and what is reasonable.

[PavelTurk]

@guw I think now it is not very important how to name it. If you don't like site, lets call it web-resource or web-x, or domain or xxx.web. What important is to understand that this web-resource consists not only of web application archive (.war) but also of additional JPMS modules that, as I understand, and it is very important, must be outside .war.

[guw]

@PashaTurok I'm not hooked on a specific name. Again, the concept of defining layers which combine modules outside of war with a war seems one of the many possibilities. As with class loaders, there is some "inherited" visibility that is given. However, I cannot decide whether Jetty should implement all possible options now or wait for some discussion to happen in the EE4J context.

FWIW, I think it makes sense to start small and allow consumers defining their own composition of JPMS modules including Jetty. This is similar to running Jetty inside an OSGi container as a bundle. If there is an agreement on how to slice the modules then this could be implemented. Any consumer can then define it's own loader as well as layer structure.

[PavelTurk]

Could anyone say when this issue will approximately be resolved? In what version?

[WalkerWatch]

@PashaTurok We've not landed on any specific date or version for this yet. As @jmcc0nn3ll said, this is not a small change and will require significant effort to get up and running. Additionally, as @gregw pointed out, there are several questions we still need answers to in the long run. I know these are not things you can answer but will have to be dealt with regardless.

[PavelTurk]

@WalkerWatch Thank you for your answer. I know that it is not small change, but I don't see any movement, that's why I asked about time. Or maybe I am wrong, maybe the issue is discussed somewhere else. To tell the truth I would like to see JPMS support in the nearest future as Java 9 released more then half a year ago.

[sbordet]

"As of today, JPMS is a pain for library authors"
http://blog.joda.org/2018/03/jpms-negative-benefits.html

[PavelTurk]

@sbordet Thank you for the provided link. The red line is The problem is that other projects will depend on your library. By not adopting JPMS at all, or with serious delay, you block those projects from progressing in their modularization.

[sbordet]

@PashaTurok there are numerous red lines. From having to require Java 9+ as a baseline (can't do), to keep Android compatibility (a must), to get the work done (lots of work, negative benefits).
You see your red line, we see ours. It will be done, eventually. Feel free to produce a pull request. Thanks.

[PavelTurk]

@sbordet Thank you for your comment. I agree that you see your red lines and I see mine.

About your suggestion to produce a pull request I can say, that I suggested to make jpms modules as 1:1 to osgi modules as we did in all our products. But my suggestion was not accepted. I know nothing about jetty architecture and its structure and obvious it will take me a lot of time to start to do something. This is the first. The second I, as thousands users of jetty, have to spend time on my own projects and don't have time even for it. That's why I can't take part in jetty development.

In my posts I try to express the user needs about JPMS support feature and try to understand approximate time about this feature. For example, if someone says it will be resolved in 5-7 years I will understand that I should look for another solution. I think, I've expressed my idea. Thanks.

[jmcc0nn3ll]

Currently, the entire JavaEE project is going through the process of migrating to the Eclipse Foundation and being rebranded JakartaEE which is the new future of all of those specifications (Gunner referred to this above as EE4J). Once that lengthy process is off the ground they will be re-releasing the current specifications under the new letterhead so making sure we deal with that as elegantly as possible is probably one of the larger priorities we have. Coupled with finalizing the websocket refactoring, supporting the accelerated releases of Java itself and pushing other elements in the project towards the release of Jetty 10 all vying for priority the fact that you still remain the only person at the time of comment that has outright asked for this support may be an indication that the support you are seeking may not land in the short term. Now if we start getting a slew of people on this thread or another, be it end user, other projects or a client, expressing frustration with lack of support then that would be clear indication we need to move faster on the issue. As it stands there isn't a fire here and I don't see us rushing to support JPMS in the short term. Do we want to support it? Yes, absolutely. Are we feeling a massive need that it has to be in the next month or two? Not particularly.

Anyway, as you said "The problem is that other projects will depend on your library. By not adopting JPMS at all, or with serious delay, you block those projects from progressing in their modularization.". We want to hear from those projects and what their expectations are in how they want to work with Jetty through JPMS. If you have been in contact with them expressing similar concerns then by all means send them our way or point us at their having issues.

[jpe42]

+1 in the hopes there are others out there... Was looking for a quick embedded WebSocket server to run in my tests and I get a slew of module jetty.* reads package from both module A and B:

Error:java: module jetty.server reads package javax.websocket from both javax.websocket.client.api and javax.websocket.api

Also, really hoping to see more core/base libraries move so that running compact jlink builds are possible. Currently jlink doesn't allow automatic module names.

[sbordet]

@jamespedwards42 can you detail how you would use a JPMS Jetty (e.g. just embed it in your application, use with jlink, etc.) ?
What is it that using JPMS will give you that you cannot do it with normal classpath ?

As said previously, knowing how people would use a JPMS'd Jetty will help us drive the effort.

I ask also because I went to conferences and spoke to expert on the matter (book authors and conference speakers), and we struggle to find a reason to move to JPMS modules.
We also could not find any project of the size/structure of Jetty that is doing the effort.

The latest report from Maven Central shows that basically nobody is moving to modules - apart a couple of logging frameworks.

Also, the JAXB module that will be removed in JDK 11 has a module-info.java, but the whole thing is broken - so having a module-info.java in the jar does not mean that the module is usable.

Honestly, does not look good for JPMS modules for applications.
They are good for the JDK, but that's about it.

I can be convinced otherwise, but I need some strong evidence or a killer use case, something where it would be clearly evident that a JPMS Jetty far outweigh its classpath counterpart.

[PavelTurk]

Honestly, does not look good for JPMS modules for applications. They are good for the JDK, but that's about it.

Sorry, I can't agree with that. It seems to me, that you never used OSGi and didn't see the advantages of modular frameworks. JPMS is like OSGi but, of course, with some differences. You can take a look how many libraries are OSGi compliant. We can even say, that it is difficult to find a library that is not for OSGi.

By the way, Jetty is also OSGi compliant. Following your logic, there is no sense in it.

I see several reasons, why there are not so many projects that moved to JPMS: 1) a lot of projects can work as auto modules without any problems (Jetty, unfortunately can't). 2) not all developers learned this technology yet 3) this is new technology and not much time has passed.

[jpe42]

@sbordet I felt inclined to post my desire for modules because of the following statement by @jmcc0nn3ll

Now if we start getting a slew of people on this thread or another, be it end user, other projects or a client, expressing frustration with lack of support then that would be clear indication we need to move faster on the issue.

Yes, I would like be able to deploy minimal runtimes compiled by jlink that include jetty as a webserver. However, I don't think there is much of an objective argument there, unless it is for a client-side/GUI application or some kind of IOT application. Maybe there is an objective argument there around security and having a reduced attack surface, but I'm not sure, no expertise there.

If you want to try out incubator modules then you need to have your project build modularized. I've been building an application against the new HttpClient, jdk.incubator.httpclient, and now it will all be ready to go for JDK 11 with the module java.net.http.

I think it would encourage better development practices across the entire ecosystem because developers would be forced to take a look at all of the dependencies they are bringing in. Hopefully it would encourage developers to program against interfaces and use the ServiceLoader. For example, why do core libraries have dependencies on Gson or Jackson, etc? Develop against an interface and load your JsonSerializer service at runtime. You shouldn't force your users into Jackson. Allow them to choose the Jackson module at runtime. Go look at libraries put out by respected organizations such as Amazon and even Google for interacting with database services, it is crazy that they have these forced dependencies.

I agree that larger projects such as jetty should not be the first movers, but it would be nice if they would help put the pressure on their core dependencies.

[jpe42]

If anyone else is trying to get Jetty working as an automatic module. The dependency setup discussed in this blog post worked for me: https://itnext.io/writing-a-web-socket-server-with-embedded-jetty-46fe9ab1c435

[sbordet]

@jamespedwards42 thanks for your feedback 👍

[sbordet]

@jamespedwards42 wrote:

Develop against an interface and load your JsonSerializer service at runtime.

It is not that simple, though, and it's a common case in Jetty too.

Say for example that you want to format your JSON with nice indents when you serialize an object.
How do you tell your JsonSerializer interface this?

You may add a setter method on it (setPrettyPrint(boolean)), but it quickly becomes a fat interface with potentially a lot of setters when you only need String serialize(Object).
And these setters can only be those common to all implementations (or - yuck - throw/ignore when called and not supported).

Perhaps a builder? Well, it must be a Jackson-specific builder, though; so we are back at depending from a specific library directly.

How about a single configure(Map<String, Object> settings) method in the JsonSerializer interface? That works, but you lose your type safety.

What about XML/Properties file, à la JPA? Again type safety lost.

CDI-style annotations? Isn't it similar to setters case, with the complication of combinatorial explosions of different configurations and custom annotations?

Many Jetty modules have a similar dilemma: we would like to hide certain implementation classes, but how can we expose their configurability without exporting the implementation class as well?

Ok so we just export everything in the JPMS module file. But how's that any different than the classpath then? Sure, jlink. And that's it.

That's why we want to collect from the community how they intend to use modules.

The line that we'll draw when we hide certain classes and export others can only be moved in one direction (that of exporting more classes) and may potentially break people that downgrade their Jetty version, for whatever reason (tried Jetty version N, does not work for me, downgrade to N-1, can't compile anymore).
If we decide to not draw the line and export everything (e.g. automatic modules), we can never go back.

[panga]

Hi @sbordet ,

IMHO you are overreacting about JPMS, it is not broken, it is already been evaluated (takes time to companies and developers evaluate) and it is already started to be used by enterprise products.

I know that JPMS is not very well supported yet by many popular libraries, but that doesn't mean it's broken, actually, the core libraries like Jetty should support it in order to dependent projects migrate to JPMS.

Besides the logical resolution of classpath hell (now we will call it module hell), there are more reasons to migrate to JPMS as described here: https://medium.com/criciumadev/java-10-migration-5d853f5b5f7e

One thing to take into the consideration is that many many enterprises run its applications in Docker containers as uber/fat jars with all dependencies in a single JVM like my company does with hundreds of Java applications.

To be honest, Jetty is the only major servlet provider we use that doesn't support basic JPMS yet.

We can run Spring Boot 2 applications with Undertow or Tomcat without any issue using JPMS very smoothly (no classpath), so they are the recommended choice now.

This becomes the sweet spot for JPMS, where we can leverage of jlink to create a custom JRE with only needed modules, so we don't need to load Swing/AWT classes for example, and make the application bootstrap faster and reduce memory consumption.

One thing I want to emphasise is that JPMS support can come in phases, no need to have full compliance.

I have a list to start:

1. Add Automatic Module Names into all artifacts (Assign stable Java 9 Automatic-Module-Name for all published artifacts #2127), so we don't have module names conflicts.
2. Ensure packages aren't duplicated across artifacts, so we don't have module exports conflicts.
3. Refactor Jetty classloader/jar/war/etc. detection to be aware of JPMS and scan files accordingly.
4. Update dependencies to be Java 9+ compatible (ASM, etc.)
5. Create a getting started documentation to list supported features, getting started and know issues.

After fixing all the items on this list, Jetty will be JPMS compatible and can be used in Java modular projects, while still have compatibility with older JDKs (1.7, etc.) because no module descriptor is required.

I want to offer some contribution to help with this issue. Probably testing and documenting, as I don't know the Jetty architecture or code base.

[gregw]

@panga We are certainly interested in taking reasonable steps to ensure we don't clash with JPMS, even if we are not yet ready to fully embrace it.

However, the concern is that if we take a wrong step, then we make be committed to a wrong path. See @sbordet 's concern about going down the automatic module name path. If we do that, are we then committed to not change any of our APIs because they are now part of the public API of our modules?

[panga]

@gregw,

I understand you guys are concerned about API breaking changes because of JPMS, but there's a fact that you need to keep in mind:

Every JAR has already a Automatic Module Name generated by JVM (not stable & weird names) with all exported packages, just like it works today in JDK < 9.

So, what prevents Jetty to be JPMS compatible, making automatic module names stable, update thirdparty dependencies and be JPMS aware when doing some stuff like classloading/Unsafe/Reflection ? (remember that all packages are already exported by default).

The changes described above absolutely doesn't break anything on existing API and doesn't even need a major version to be released.

If first class JPMS support is really needed in future (JDK 11+), you want to embrace it, rethink of architecture and modules, private/public packages, etc. Them probably a major version would be released with JDK 9+ requirements and possibly with breaking changes.

That's not what I'm asking to do here or in near future.

[sbordet]

@panga, what use or benefit do you have by running jars as automodules (either via manifest or via JVM) that you cannot have with classpath?

Would be great if you can explain with a clear example "I want to do this, and I cannot do it unless I use modules".
Have not found a single one so far.

Breaking everybody when we'll decide to go with proper modules is not something that people will like, especially when there's no way back.

Not to mention that we will need to change all Jetty module names since there is no versioning.

Your blog entry is like many others that show how to move to modules, like many others show how incredibly complicated that is (well, just follow 10-15 bullets of instructions and dodge all the quirks/bugs present in libraries that you need), and like many others the main question remains "what benefits do I have"? The answer is, like many others, zero.

You cannot use jlink, you cannot leverage modules for hiding implementation classes from libraries, I have not yet seen proof of any reduced jvm footprint or faster jvm starts. I'll be interested if you have pointers to real data, possibly not "hello world" examples, compared with proper classpath usage.

Thanks!

[panga]

what use or benefit do you have by running jars as automodules (either via manifest or via JVM) that you cannot have with classpath?

You can't deal with the fact that JPMS is there and can be used and will be used, whatever is the reason.

Breaking everybody when we'll decide to go with proper modules is not something that people will like, especially when there's no way back.
Not to mention that we will need to change all Jetty module names since there is no versioning.

There's no breaking if you support both classpath and modulepath with multi-release jars.

Your blog entry is like many others that show how to move to modules, like many others show how incredibly complicated that is (well, just follow 10-15 bullets of instructions and dodge all the quirks/bugs present in libraries that you need), and like many others the main question remains "what benefits do I have"? The answer is, like many others, zero.

The workarounds are temporary while thirdparty dependencies doesn't act properly to this call https://dzone.com/articles/automatic-module-name-calling-all-java-library-maintainers

Also, you can see the benefits by yourself executing the sample REST + JPA from my sample tutorial, and even this little application has perceptible benefits in memory and startup.

Please see this presentation:
https://www.slideshare.net/leonardozanivan/qconsp-2018-java-module-system

You cannot use jlink, you cannot leverage modules for hiding implementation classes from libraries, I have not yet seen proof of any reduced jvm footprint or faster jvm starts. I'll be interested if you have pointers to real data, possibly not "hello world" examples, compared with proper classpath usage.

The assumption that jlink can't be used is wrong, I'm already using it my own Java projects and with this maven plugin (https://github.com/moditect/moditect) I can fix thirdparty dependencies modules that are slow to update themselves.

[PavelTurk]

@sbordet Can you say if you solved the problem with 2018-09-25 19:53:23.904:WARN:oatud.DigesterFactory:main: The XML schema [XMLSchema.dtd] could not be found. This is very likely to break XML validation if XML validation is enabled.?

[sbordet]

@nicolaiparlog thanks for the detailed update, all clear now. The --show-module-resolution issue was my fault - I was issuing it on the wrong JVM. Now I can see automatic modules are resolved, similar to your example. Thanks!

[sbordet]

@PashaTurok yes it's solved.

[lukehutch]

Since there was some discussion on multi-release jars earlier: @PashaTurok if you're still planning on using ClassGraph, you might be interested to know that I added support for multi-release jar scanning in version 4.2.3. If any META-INF/versions/X directories exist (for integers X greater than or equal to 9 and less than or equal to the running JVM version), then the directory X with the highest value less than or equal to the JVM version is used as the package root for scanning the jar.

[PavelTurk]

@lukehutch Ok. Thank you. I will take a look.

[joakime]

@lukehutch took a look at your multi-release stuff, why didn't you just use the jvm built-in zipfs FileSystem?

[lukehutch]

@joakime because ClassGraph is supposed to be backwards-compatible all the way back to JDK7, so every call I have to make to APIs only available in JDK 8+ needs to be done through reflection, which is cumbersome. Also, I put a lot of work into highly parallelizing ZipFile scanning, and using zipfs (probably) can't achieve the same level of scalability. (ZipFile has a global per-instance lock around it -- I'm not sure if zipfs uses that, or reimplements unzipping from scratch, but it is unlikely that the Java libraries implemented the unzip code two separate times, so I'm guessing ClassGraph's unzip code is much faster than zipfs, because it creates one ZipFile instance per thread, which is the only way I am aware of that zipfile access can be parallelized in Java using the standard APIs.)

That said, if you're using JPMS, which is highly likely if you're using multi-release jars, the ClassGraph parallel ZipFile code is never even called, because ClassGraph calls the JPMS API through reflection, which internally uses zipfs. The JPMS API automatically handles multi-release masking (and removing the META-INF/versions/X prefix from resource paths), so ClassGraph doesn't need to do that for the module path scanning case either.

So really the code I put in place for multi-release jars is for a very specific usecase: including a multi-release jar on the traditional classpath (rather than the module path) when running on JDK 9+.

[PavelTurk]

@lukehutch You suggested to replace

URL url = new URL("jar:" + res.getClasspathElementURL() + "!/" + res.getPath());

with:

URL url = new URL("jar:" + res.getURL());

Unfortunately it didn't work.
My code gives such result:

jar:file:/home/Pavel/Web%20Server/bin/./../jar/com.noname.temp.core-1.0.0.jar!/META-INF/tags/tags.tld

Your code this:

jar:file:/home/Pavel/Web%20Server/bin/./../jar/com.noname.temp.core-1.0.0.jar!META-INF/tags/tags.tld

See the difference before META-INF.

[PavelTurk]

@sbordet I have two layers. On one I have jetty, on the second I have two modules - war module and jar module. Jar module contains tld files and tag implementations classes. Tags from jar module I use in war module.

This is stack:

2018-10-10 19:54:15:325 [qtp1062359567-39] ERROR com.noname.jar.core.mvc.JspView - There was an error processing file /dynamic/jsp/common.jsp
org.apache.jasper.JasperException: /dynamic/jsp/common.jsp (line: [19], column: [8]) Unable to load tag handler class [com.noname.jar.core.tags.IncludeAreaTag] for tag [t:includeArea]
	at org.mortbay.apache.jasper@8.5.24.2/org.apache.jasper.compiler.DefaultErrorHandler.jspError(DefaultErrorHandler.java:41)
	at org.mortbay.apache.jasper@8.5.24.2/org.apache.jasper.compiler.ErrorDispatcher.dispatch(ErrorDispatcher.java:291)
	at org.mortbay.apache.jasper@8.5.24.2/org.apache.jasper.compiler.ErrorDispatcher.jspError(ErrorDispatcher.java:97)
	at org.mortbay.apache.jasper@8.5.24.2/org.apache.jasper.compiler.Parser.parseCustomTag(Parser.java:1244)
	at org.mortbay.apache.jasper@8.5.24.2/org.apache.jasper.compiler.Parser.parseElements(Parser.java:1473)
	at org.mortbay.apache.jasper@8.5.24.2/org.apache.jasper.compiler.Parser.parse(Parser.java:145)
	at org.mortbay.apache.jasper@8.5.24.2/org.apache.jasper.compiler.ParserController.doParse(ParserController.java:244)
	at org.mortbay.apache.jasper@8.5.24.2/org.apache.jasper.compiler.ParserController.parse(ParserController.java:105)
	at org.mortbay.apache.jasper@8.5.24.2/org.apache.jasper.compiler.Compiler.generateJava(Compiler.java:203)
	at org.mortbay.apache.jasper@8.5.24.2/org.apache.jasper.compiler.Compiler.compile(Compiler.java:374)
	at org.mortbay.apache.jasper@8.5.24.2/org.apache.jasper.compiler.Compiler.compile(Compiler.java:351)
	at org.mortbay.apache.jasper@8.5.24.2/org.apache.jasper.compiler.Compiler.compile(Compiler.java:335)
	at org.mortbay.apache.jasper@8.5.24.2/org.apache.jasper.JspCompilationContext.compile(JspCompilationContext.java:595)
	at org.mortbay.apache.jasper@8.5.24.2/org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:368)
	at org.mortbay.apache.jasper@8.5.24.2/org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:386)
	at org.mortbay.apache.jasper@8.5.24.2/org.apache.jasper.servlet.JspServlet.service(JspServlet.java:330)
	at org.eclipse.jetty.apache.jsp@9.4.13-SNAPSHOT/org.eclipse.jetty.jsp.JettyJspServlet.service(JettyJspServlet.java:112)
	at javax.servlet.api@4.0.1/javax.servlet.http.HttpServlet.service(HttpServlet.java:750)
	at org.eclipse.jetty.servlet@9.4.13-SNAPSHOT/org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:865)
	at org.eclipse.jetty.servlet@9.4.13-SNAPSHOT/org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:542)
	at org.eclipse.jetty.server@9.4.13-SNAPSHOT/org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:146)
	at org.eclipse.jetty.security@9.4.13-SNAPSHOT/org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:566)
	at org.eclipse.jetty.server@9.4.13-SNAPSHOT/org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
	at org.eclipse.jetty.server@9.4.13-SNAPSHOT/org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:257)
	at org.eclipse.jetty.server@9.4.13-SNAPSHOT/org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1595)
	at org.eclipse.jetty.server@9.4.13-SNAPSHOT/org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255)
	at org.eclipse.jetty.server@9.4.13-SNAPSHOT/org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1340)
	at org.eclipse.jetty.server@9.4.13-SNAPSHOT/org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:203)
	at org.eclipse.jetty.servlet@9.4.13-SNAPSHOT/org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:480)
	at org.eclipse.jetty.server@9.4.13-SNAPSHOT/org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1564)
	at org.eclipse.jetty.server@9.4.13-SNAPSHOT/org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:201)
	at org.eclipse.jetty.server@9.4.13-SNAPSHOT/org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1242)
	at org.eclipse.jetty.server@9.4.13-SNAPSHOT/org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:144)
	at org.eclipse.jetty.server@9.4.13-SNAPSHOT/org.eclipse.jetty.server.Dispatcher.include(Dispatcher.java:129)
	at сom.noname.jar.core/com.noname.jar.core.mvc.JspView.renderFile(JspView.java:136)
	at сom.noname.jar.core/com.noname.jar.core.mvc.JspView.render(JspView.java:94)
	at сom.noname.jar.core/com.noname.jar.core.AbstractFrontController.processDynamicRequest(AbstractFrontController.java:194)
	at сom.noname.jar.core/com.noname.jar.core.AbstractFrontController.processRequest(AbstractFrontController.java:168)
	at сom.noname.jar.core/com.noname.jar.core.AbstractFrontController.doGet(AbstractFrontController.java:236)
	at javax.servlet.api@4.0.1/javax.servlet.http.HttpServlet.service(HttpServlet.java:645)
	at javax.servlet.api@4.0.1/javax.servlet.http.HttpServlet.service(HttpServlet.java:750)
	at org.eclipse.jetty.servlet@9.4.13-SNAPSHOT/org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:857)
	at org.eclipse.jetty.servlet@9.4.13-SNAPSHOT/org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:542)
	at org.eclipse.jetty.server@9.4.13-SNAPSHOT/org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:146)
	at org.eclipse.jetty.security@9.4.13-SNAPSHOT/org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)
	at org.eclipse.jetty.server@9.4.13-SNAPSHOT/org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
	at org.eclipse.jetty.server@9.4.13-SNAPSHOT/org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:257)
	at org.eclipse.jetty.server@9.4.13-SNAPSHOT/org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1595)
	at org.eclipse.jetty.server@9.4.13-SNAPSHOT/org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255)
	at org.eclipse.jetty.server@9.4.13-SNAPSHOT/org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1340)
	at org.eclipse.jetty.server@9.4.13-SNAPSHOT/org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:203)
	at org.eclipse.jetty.servlet@9.4.13-SNAPSHOT/org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:480)
	at org.eclipse.jetty.server@9.4.13-SNAPSHOT/org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1564)
	at org.eclipse.jetty.server@9.4.13-SNAPSHOT/org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:201)
	at org.eclipse.jetty.server@9.4.13-SNAPSHOT/org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1242)
	at org.eclipse.jetty.server@9.4.13-SNAPSHOT/org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:144)
	at org.eclipse.jetty.server@9.4.13-SNAPSHOT/org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
	at org.eclipse.jetty.server@9.4.13-SNAPSHOT/org.eclipse.jetty.server.Server.handle(Server.java:503)
	at org.eclipse.jetty.server@9.4.13-SNAPSHOT/org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:364)
	at org.eclipse.jetty.server@9.4.13-SNAPSHOT/org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:260)
	at org.eclipse.jetty.io@9.4.13-SNAPSHOT/org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:305)
	at org.eclipse.jetty.io@9.4.13-SNAPSHOT/org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103)
	at org.eclipse.jetty.io@9.4.13-SNAPSHOT/org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:118)
	at org.eclipse.jetty.util@9.4.13-SNAPSHOT/org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:765)
	at org.eclipse.jetty.util@9.4.13-SNAPSHOT/org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:683)
	at java.base/java.lang.Thread.run(Thread.java:834)

At the same time methods protected Class<?> foundClass(final String name, URL url) and protected Class<?> findClass​(java.lang.String name) of my ClassLoader (extends WebAppClassLoader) which I set via webapp.setClassLoader are not used. Could you say how to explain it? (I use the newest jetty jpms code).

[PavelTurk]

@sbordet About schemas

Note,
INFO org.eclipse.jetty.server.Server - jetty-9.4.z-SNAPSHOT; built: 2018-10-10T15:07:33.377Z; git: bd3eeea; jvm 11+28

2018-10-10 20:23:03:876 [RMI TCP Connection(2)-127.0.0.1] INFO org.eclipse.jetty.server.Server - jetty-9.4.z-SNAPSHOT; built: 2018-10-10T15:07:33.377Z; git: bd3eeeaa3f123e4c559e80ce21b5ab0ecbfb2f9c; jvm 11+28
2018-10-10 20:23:04:371 [RMI TCP Connection(2)-127.0.0.1] INFO org.eclipse.jetty.annotations.AnnotationConfiguration - Scanning elapsed time=58ms
2018-10-10 20:23:04:442 [RMI TCP Connection(2)-127.0.0.1] WARN org.apache.tomcat.util.descriptor.DigesterFactory - The XML schema [XMLSchema.dtd] could not be found. This is very likely to break XML validation if XML validation is enabled.
2018-10-10 20:23:04:442 [RMI TCP Connection(2)-127.0.0.1] WARN org.apache.tomcat.util.descriptor.DigesterFactory - The XML schema [datatypes.dtd] could not be found. This is very likely to break XML validation if XML validation is enabled.
2018-10-10 20:23:04:442 [RMI TCP Connection(2)-127.0.0.1] WARN org.apache.tomcat.util.descriptor.DigesterFactory - The XML schema [xml.xsd] could not be found. This is very likely to break XML validation if XML validation is enabled.
2018-10-10 20:23:04:443 [RMI TCP Connection(2)-127.0.0.1] WARN org.apache.tomcat.util.descriptor.DigesterFactory - The XML schema [web-app_2_2.dtd] could not be found. This is very likely to break XML validation if XML validation is enabled.
2018-10-10 20:23:04:443 [RMI TCP Connection(2)-127.0.0.1] WARN org.apache.tomcat.util.descriptor.DigesterFactory - The XML schema [web-app_2_3.dtd] could not be found. This is very likely to break XML validation if XML validation is enabled.
2018-10-10 20:23:04:444 [RMI TCP Connection(2)-127.0.0.1] WARN org.apache.tomcat.util.descriptor.DigesterFactory - The XML schema [j2ee_web_services_1_1.xsd] could not be found. This is very likely to break XML validation if XML validation is enabled.
2018-10-10 20:23:04:444 [RMI TCP Connection(2)-127.0.0.1] WARN org.apache.tomcat.util.descriptor.DigesterFactory - The XML schema [j2ee_web_services_client_1_1.xsd] could not be found. This is very likely to break XML validation if XML validation is enabled.
2018-10-10 20:23:04:444 [RMI TCP Connection(2)-127.0.0.1] WARN org.apache.tomcat.util.descriptor.DigesterFactory - The XML schema [web-app_2_4.xsd] could not be found. This is very likely to break XML validation if XML validation is enabled.
2018-10-10 20:23:04:444 [RMI TCP Connection(2)-127.0.0.1] WARN org.apache.tomcat.util.descriptor.DigesterFactory - The XML schema [j2ee_1_4.xsd] could not be found. This is very likely to break XML validation if XML validation is enabled.
2018-10-10 20:23:04:445 [RMI TCP Connection(2)-127.0.0.1] WARN org.apache.tomcat.util.descriptor.DigesterFactory - The XML schema [web-app_2_5.xsd] could not be found. This is very likely to break XML validation if XML validation is enabled.
2018-10-10 20:23:04:445 [RMI TCP Connection(2)-127.0.0.1] WARN org.apache.tomcat.util.descriptor.DigesterFactory - The XML schema [javaee_5.xsd] could not be found. This is very likely to break XML validation if XML validation is enabled.
2018-10-10 20:23:04:446 [RMI TCP Connection(2)-127.0.0.1] WARN org.apache.tomcat.util.descriptor.DigesterFactory - The XML schema [javaee_web_services_1_2.xsd] could not be found. This is very likely to break XML validation if XML validation is enabled.
2018-10-10 20:23:04:446 [RMI TCP Connection(2)-127.0.0.1] WARN org.apache.tomcat.util.descriptor.DigesterFactory - The XML schema [javaee_web_services_client_1_2.xsd] could not be found. This is very likely to break XML validation if XML validation is enabled.
2018-10-10 20:23:04:446 [RMI TCP Connection(2)-127.0.0.1] WARN org.apache.tomcat.util.descriptor.DigesterFactory - The XML schema [web-app_3_0.xsd] could not be found. This is very likely to break XML validation if XML validation is enabled.
2018-10-10 20:23:04:446 [RMI TCP Connection(2)-127.0.0.1] WARN org.apache.tomcat.util.descriptor.DigesterFactory - The XML schema [web-fragment_3_0.xsd] could not be found. This is very likely to break XML validation if XML validation is enabled.
2018-10-10 20:23:04:447 [RMI TCP Connection(2)-127.0.0.1] WARN org.apache.tomcat.util.descriptor.DigesterFactory - The XML schema [web-common_3_0.xsd] could not be found. This is very likely to break XML validation if XML validation is enabled.
2018-10-10 20:23:04:447 [RMI TCP Connection(2)-127.0.0.1] WARN org.apache.tomcat.util.descriptor.DigesterFactory - The XML schema [javaee_6.xsd] could not be found. This is very likely to break XML validation if XML validation is enabled.
2018-10-10 20:23:04:448 [RMI TCP Connection(2)-127.0.0.1] WARN org.apache.tomcat.util.descriptor.DigesterFactory - The XML schema [javaee_web_services_1_3.xsd] could not be found. This is very likely to break XML validation if XML validation is enabled.
2018-10-10 20:23:04:448 [RMI TCP Connection(2)-127.0.0.1] WARN org.apache.tomcat.util.descriptor.DigesterFactory - The XML schema [javaee_web_services_client_1_3.xsd] could not be found. This is very likely to break XML validation if XML validation is enabled.
2018-10-10 20:23:04:448 [RMI TCP Connection(2)-127.0.0.1] WARN org.apache.tomcat.util.descriptor.DigesterFactory - The XML schema [web-app_3_1.xsd] could not be found. This is very likely to break XML validation if XML validation is enabled.
2018-10-10 20:23:04:448 [RMI TCP Connection(2)-127.0.0.1] WARN org.apache.tomcat.util.descriptor.DigesterFactory - The XML schema [web-fragment_3_1.xsd] could not be found. This is very likely to break XML validation if XML validation is enabled.
2018-10-10 20:23:04:449 [RMI TCP Connection(2)-127.0.0.1] WARN org.apache.tomcat.util.descriptor.DigesterFactory - The XML schema [web-common_3_1.xsd] could not be found. This is very likely to break XML validation if XML validation is enabled.
2018-10-10 20:23:04:449 [RMI TCP Connection(2)-127.0.0.1] WARN org.apache.tomcat.util.descriptor.DigesterFactory - The XML schema [javaee_7.xsd] could not be found. This is very likely to break XML validation if XML validation is enabled.
2018-10-10 20:23:04:449 [RMI TCP Connection(2)-127.0.0.1] WARN org.apache.tomcat.util.descriptor.DigesterFactory - The XML schema [javaee_web_services_1_4.xsd] could not be found. This is very likely to break XML validation if XML validation is enabled.
2018-10-10 20:23:04:449 [RMI TCP Connection(2)-127.0.0.1] WARN org.apache.tomcat.util.descriptor.DigesterFactory - The XML schema [javaee_web_services_client_1_4.xsd] could not be found. This is very likely to break XML validation if XML validation is enabled.

Maybe I don't use some module? These are my modules:

	<module file="jetty-servlet-9.4.13-SNAPSHOT.jar"/>
	<module file="jetty-servlets-9.4.13-SNAPSHOT.jar"/>
	<module file="jetty-util-9.4.13-SNAPSHOT.jar"/>
	<module file="jetty-webapp-9.4.13-SNAPSHOT.jar"/>
	<module file="jetty-xml-9.4.13-SNAPSHOT.jar"/>
	<module file="jetty-http-9.4.13-SNAPSHOT.jar"/>
	<module file="jetty-io-9.4.13-SNAPSHOT.jar"/>
	<module file="jetty-security-9.4.13-SNAPSHOT.jar"/>
	<module file="jetty-server-9.4.13-SNAPSHOT.jar"/>
	<module file="jetty-annotations-9.4.13-SNAPSHOT.jar"/>
	<module file="jetty-jndi-9.4.13-SNAPSHOT.jar"/>
	<module file="jetty-plus-9.4.13-SNAPSHOT.jar"/>
	<module file="jetty-schemas-3.1.jar"/>
	<module file="org.eclipse.jetty.apache-jsp-9.4.13-SNAPSHOT.jar"/>
	<module file="javax.servlet-api-4.0.1.jar"/>
	<module file="asm-6.2.jar"/>
	<module file="asm-analysis-6.2.jar"/>
	<module file="asm-commons-6.2.jar"/>
	<module file="asm-tree-6.2.jar"/>
	<module file="classgraph-4.2.2-SNAPSHOT.jar"/>
	<module file="javax.annotation-api-1.2.jar"/>
	<module file="org.eclipse.jdt.ecj-3.12.3.jar"/>
	<module file="org.mortbay.jasper.apache-el-8.5.24.2.jar"/>
	<module file="apache-jsp-8.5.24.2.jar"/>
	<module file="org.apache.taglibs.taglibs-standard-impl-1.2.5.jar"/>
	<module file="org.apache.taglibs.taglibs-standard-spec-1.2.5.jar"/>

[lukehutch]

See the difference before META-INF

@PashaTurok Yes, getURL() used to put "!/" between the jar name and the path within the jar, but now it just puts "!", because I noticed somewhere in the Java standard API omitted the leading slash in the path. What are you passing this URL to, and how does it complain when the slash is missing?

[PavelTurk]

@lukehutch When I use your variant I get:

java.net.MalformedURLException: no !/ in spec
	at java.base/java.net.URL.<init>(URL.java:661)
	at java.base/java.net.URL.<init>(URL.java:523)
	at java.base/java.net.URL.<init>(URL.java:470)
        ....
Caused by: java.lang.NullPointerException: no !/ in spec
	at java.base/sun.net.www.protocol.jar.Handler.parseAbsoluteSpec(Handler.java:168)
	at java.base/sun.net.www.protocol.jar.Handler.parseURL(Handler.java:150)
	at java.base/java.net.URL.<init>(URL.java:656)
	... 25 more

[lukehutch]

@PashaTurok interesting -- these URLs work fine without the "jar:" scheme prefix, which is why I didn't run into this before. The exception is only reported with that scheme prefix. Fixed in 4.2.11.

[sbordet]

#2934 has been merged into mainline jetty-9.4.x.

[PavelTurk]

@sbordet Could you say if you answer two my last questions? Or I should ask them somewhere else? I mean about tag class loading and schemas. Note - I am using code from jetty-9.4.x master and get the same results as I described above.

[sbordet]

@PashaTurok jetty-9.4.x has support for modules if you use it standalone.

For embedded usage, like you are doing, you have to do a lot of manual stuff yourself.

For the schemas, you have to patch the javax.servlet jar with jetty-schemas jar, see https://github.com/eclipse/jetty.project/blob/jetty-9.4.x/jetty-server/src/main/config/modules/server.mod.

For the TLDs, we already discussed this and it's all about letting Jetty scan classes at startup.
I thought you resolved this with ClassGraph.
We do the scanning here.

Now, you have a totally different setup, you use ModuleLayer, so I cannot help you with your code - only tell you what we have done in Jetty.

[PavelTurk]

@sbordet Thank you very much for your help. Only two moments : 1) how to explain this patch-module: servlet.api=lib/jetty-schemas-3.1.jar in terms of java and JPMS but not Jetty? 2) I totally don't agree with you that Jetty must not support ModuleLayer. And the explanation is very simple - Layers are part of JPMS specs. If we say about Jetty JPMS support we must say also about layers support. But if Jetty JPMS support == Jetty JPMS modularization (within boot layer) then layers of course are not the issue.

Maybe I should open issue JPMS ModuleLayer support :)?

[sbordet]

patch-module is a JPMS concept.

@PashaTurok you have stated multiple times what you think, and we understand it. You have already opened an issue about deploying a web application as a module layer, and we cannot do it in 9.4.x. Will be done in 10.x. There we will decide if it is also convenient that the server is in a module layer.
We also need to coordinate with the JakartaEE effort.