Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Clarify location of spec artifacts #221

Closed
arthurdm opened this issue May 19, 2018 · 50 comments
Closed

Clarify location of spec artifacts #221

arthurdm opened this issue May 19, 2018 · 50 comments

Comments

@arthurdm
Copy link
Member

arthurdm commented May 19, 2018

This issue will take a look at the spec, decide what is the appropriate design, and further clarify the location of the various artifacts, such as the OpenAPI static file / snippet, and property files.

@OndroMih
Copy link

It's always been confusing where the location of config files is in a WAR file - JPA persistence.xml is supposed to be in classes/META-INF but CDI's beans.xml should be in WEB-INF/beans.xml.

Though, I haven't seen a solution where a file is expected in the root META-INF folder in case of a WAR file as the MP Open API spec now mandates. In maven projects, META-INF is usually placed in src/main/resources which maps to WEB-INF/classes in the WAR. In order to map to root META-INF folder, the folder would have to be in src/main/webapp/META-INF next to WEB-INF, potentially leading to 2 META-INF folders in case the same WAR uses e.g. persistence.xml from JPA.

I don't mind too much but MP Specs should be consistent and support same options. All the following options meet that:

  1. Open API defines in a WAR file META-INF should be located in WEB-INF/classes/META-INFand obsoletes supporting META-INF in the root of a WAR file
  2. MP Config adds support for META-INF in the root of a WAR file to match the current requirements of Open API
  3. Both Config and OpenAPI add support for config files in WEB-INF instead of META-INF in a WAR file (similar to the CDI spec which expects beans.xml in a WEB-INF for WARs) - I've raised Support config files in WEB-INF for a WAR file microprofile-config#362 to add support for this in MP Config
  4. Both Config and OpenAPI add support for all locations (root WEB-INF, root META-INF, WEB-INF/classes/META-INF) so that users can place their config in any of them without too much thinking

From the above, my preferred solution is 3. or if MP Config wouldn't agree then option 1.

The option 2. is supported by OpenAPI now but I don't think it's the best solution and I discourage from adopting it for MP Config and rather make it obsolete for OpenAPI.

@arthurdm
Copy link
Member Author

arthurdm commented May 19, 2018

thanks a lot for outlining these different alternatives @OndrejM - it helps to see them all in one place.

Part of the motivation for the root/META-INF location was to make it simple to augment an existing WAR (with either a OAS3 doc or a properties file that configured the API scanner) so that it produced the resulting /openapi endpoint and enabled this app to participate in the "API Economy". In this case we're talking about legacy/older applications where the developers don't want to (or can't) touch anything inside /classes.

Perhaps going forward in the new 1.1 spec the solution 1 you outlined is less confusing though, so I will ask other members of the community to weigh in as well.

@leochr @EricWittmann @mrglavas

@arjantijms
Copy link

@arthurdm

In this case we're talking about legacy/older applications where the developers don't want to (or can't) touch anything inside /classes.

I wonder, is that a real scenario? If you can touch [web root]/META-INF, why wouldn't you be able to touch [web root]/WEB-INF/classes or even simpler [web root]/WEB-INF/?

@arjantijms
Copy link

From #222

MP Open API currently specifies [war root]/META-INF for certain files to be placed as well as a MP Config properties file.

[war root]/META-INF is however a highly unusual location, and doesn't align with any existing conventions in either MicroProfile, Java EE, or related technologies.

In fact, the usage of this folder is so alien for war archives that I think we should consider it to be a spec bug and therefor should best deprecate it.

I'd like to propose to use the [war root]/WEB-INF folder for the MP Open API specific files, and do not put any requirements on the configuration file used by MP Config (just specify the properties, but leave it to MP Config where they ultimately originate from)

I though this issue (#221) was different, since this one just asks for clarification, while #222 specifically asks for deprecation, but if everything can be handled here that's fine too.

@arthurdm
Copy link
Member Author

thanks @arjantijms - this issue will decide which way we go forward in 1.1 and explicitly clarify that decision in the spec.

@arthurdm
Copy link
Member Author

arthurdm commented May 19, 2018

@arjantijms / @OndrejM - one such example is Tomcat's usage of context.xml, which sets per-web-app configuration at the <root>/META-INF/context.xml.

@arjantijms
Copy link

@arjantijms / @OndrejM - one such example is Tomcat's usage of context.xml, which sets per-web-app configuration at the /META-INF/context.xml.

That one is however not a configuration file that belongs to a spec implemented by Tomcat and/or a spec module bundled in WEB-INF/lib.

context.xml is more akin to domain.xml in Payara and standalone.xml in JBoss. You typically put that file outside the archive (in a server folder). Tomcat in fact treats it like that, as in it copies context.xml from the war using low level code to the usual location outside in the server, but only if there isn't already a context.xml there.

So although I do appreciate you found one example, this is a highly exceptional one really.

@arthurdm
Copy link
Member Author

arthurdm commented May 20, 2018

So although I do appreciate you found one example, this is a highly exceptional one really.

It demonstrates there's precedence for using <root>/META-INF beyond EE specs. WebSphere Liberty has also used <root>/META-INF for its Swagger documents and <root>/META-INF/stub for its partial Swagger documents.

Also, the fact that Java SE uses <root>/META-INF/services for jars can also be applied to app server's FAT jars, which is also relevant to the domain in question here - a lot of the MP OpenAPI properties have to do with something akin to a service reference (i.e. Filter, ModelReader, etc).

So - although <root>/META-INF is not defined in a WAR spec, there are a few examples above where it does make sense to consider as a valid location.

@arjantijms
Copy link

I'll check the liberty example to see if they actually mean [war root]/META-INF and not "a" META-INF that's found on the classpath of the web module.

I don't think Tomcat is a valid example really. It doesn't process or read context.xml from META-INF, but practically the war is extracted, the file is copied to its usual location (if there's no file there already), and only from there is the file read and processed. Liberty might be the only actual example, if it indeed does that.

Also, the fact that Java SE uses /META-INF/services for jars can also be applied to app server's FAT jars, which is also relevant to the domain in question here - a lot of the MP OpenAPI properties have to do with something akin to a service reference (i.e. Filter, ModelReader, etc).

I don't quite agree with that. The /META-INF/services for jars is well known and often used in wars, and that's via WEB-INF/lib/[some jar] basically each and every spec in Java EE and MP too makes use of that.

there are a few examples above where it does make sense to consider as a valid location.

The fact that it's possible and even that there might be some (weak?) precedence, does beg the question whether we -should-?

Obviously it has proven to be possible for JAX-RS to use @Context instead of @Inject, but is it wise and is the best for our customers and user? Do we choose things so that they are consistent with the rest of MP and Java EE so that it's easy and logical for user to use, or do we just choose something arbitrarily because we technically can, and just require the user to remember all the different ways?

@kenfinnigan
Copy link

One thing I'd like to make sure is that OpenAPI doesn't define only locations that are specific to WAR files, as JAR is still a perfectly valid deployment package type

@arthurdm
Copy link
Member Author

+1 @kenfinnigan

I think a lot of the arguments in this thread are geared solely for WAR, which is one of the deployment patterns, but not the only one. Having the OAS3 doc in the <root>/META-INF allows it to apply to any deployment pattern derived from jar (WAR, OSGi app, Jar, etc), which is one of the reasons why Liberty chose that location even for the Swagger document support back in 2016.

Since the OAS3 doc is in that location, the MP OpenAPI spec allows the flexibility to put the OpenAPI config in that directory too - in addition to (not replacing) the other valid MP Config locations.

@arjantijms
Copy link

arjantijms commented May 21, 2018 via email

@arjantijms
Copy link

arjantijms commented May 21, 2018 via email

@kenfinnigan
Copy link

No, I mean any old JAR.

MicroProfile does not dictate the packaging format as WAR or JAR, it could be either, and certainly some implementations support one or the other or both.

My concern was that it seemed to be pointing towards a WAR specific location being mentioned in the spec, which is I said that it needs to work for WAR or JAR structures

@arthurdm arthurdm self-assigned this Jun 5, 2018
@arthurdm
Copy link
Member Author

arthurdm commented Jun 5, 2018

Consensus was reached today at the hangout that the current behaviour is correct. I will assign this to myself to clarify in the spec.

@arjantijms
Copy link

MicroProfile does not dictate the packaging format as WAR or JAR, it could be either, and certainly some implementations support one or the other or both.

I wonder if that should not be an overal concern then, since the other MP specs as far as I know don't really say anything about that, do they?

My concern was that it seemed to be pointing towards a WAR specific location being mentioned in the spec, which is I said that it needs to work for WAR or JAR structures

Which is really simple to solve, by using the META-INF folder that's in the class path. That will basically resolve automatically to whatever physical location is valid for that in whatever archive.

Consensus was reached today at the hangout that the current behaviour is correct.

I wonder, who was present in that hangout and based on which arguments was that consensus reached? If it were the same people who originally okay'ed that to be in the spec in the first place, it's perhaps not a surprise they reached consensus about being correct themselves.

I feel it might be better to bring this to the broader attention of an architecture board in MP.

@OndroMih
Copy link

OndroMih commented Jun 5, 2018

@kenfinnigan @arthurdm, you're missing a few very important points:

  • for any JAR, saying that files should be in META-INF on the classpath means that they are in root/META-INF, so no point in making this explicit
  • for WAR files, if files are in root/META-INF then they are public resources accessible by an HTTP client (outside world), which is a security risk

Therefore specifying the location as "META-INF" on classpath is more correct and works for both JAR and WAR files. For JAR files, it would be also valid to specify that "META-INF" is in root, but this wouldn't work for WAR files.

I'm not suggesting we need to treat WAR files in a special way but we should specify a location that doesn't break WAR files. Specifying location as "META-INF" on classpath doesn't break WAR files while specifying the location as "META-INF" in the root of the package does because it exposes files to the outside world.

It would be convenient if WAR files used "WEB-INF" location as an alternative, but it's not necessary if you don't want to mention WAR files in the spec.

@OndroMih
Copy link

OndroMih commented Jun 5, 2018

As a result, an alternative location in <classpath>/META-INF should be supported, which works in both JAR and WAR files. Later, the location in <package-root>/META-INF should be deprecated. It wouldn't affect standard JAR files because there both locations are the same.

This section of the spec needs to be modified:

...inside the application module’s (i.e. WAR artifact) META-INF folder

Should be

inside META-INF folder on the classpath, alternatively inside the application module’s (i.e. WAR artifact), but this is deprecated and discouraged.

And the PR #220 should be accepted, or better, a new test case should be created for the new location, while the old one deprecated.

@arthurdm
Copy link
Member Author

arthurdm commented Jun 6, 2018

thanks for the comments guys. The conclusion from the hangout today is that:

  • static OAS3 files, if any, go into <root>/META-INF
  • MP OpenAPI properties can be specified anywhere that MP Config supports (so that can be within WEB-INF for WARs).
  • for convenience, MP OpenAPI properties can also be specified in <root>/META-INF

@arjantijms
Copy link

Thanks for the reference to the document. This clarifies that Ivan Junckes Filho from Tomitribe, Eric Wittmann from Red Hat, and yourself reach that consensus (I'm not sure who Marc S is though).

But since it was a video hangout, it doesn't reveal which arguments where used to reach that consensus. I'm still very curious why anyone would think <root>/META-INF would be a good place to store files, unless those files as @OndrejM mentioned are intended to be publicly available to the outside world (via an http client).

So would great if you could tell us a little about which actual arguments were used.

@arthurdm
Copy link
Member Author

arthurdm commented Jun 6, 2018

Marc S is from Red Hat, and Andy G from Tomitribe was also present (updated attendance now).

The arguments used were the same described along this thread - we took into consideration the arguments in favour and against the current solution, so we did have a fair conversation about it.

I will go ahead with a PR for the agreed direction, but if you still feel strongly against it, please join the next hangout on June 18, 10am EST and we can chat.

@OndroMih
Copy link

OndroMih commented Jul 12, 2018

I've raise eclipse/microprofile#46 for the Arch. board to address this globally.

To summarize the discussion here, the only argument for keeping the current state is mentioned in here

Part of the motivation for the root/META-INF location was to make it simple to augment an existing WAR (with either a OAS3 doc or a properties file that configured the API scanner) so that it produced the resulting /openapi endpoint and enabled this app to participate in the "API Economy". In this case we're talking about legacy/older applications where the developers don't want to (or can't) touch anything inside /classes.

I didn't find any other argument to support the decision, despite it was declared multiple times that the discussion in hangouts concluded with keeping the current state.

I can easily counter that argument. For JAR files, the situation doesn't change with having artifacts in META-INF on classpath as it's the same location as root/META-INF. For WAR files, it's as easy to augment an existing package by adding a file into root/WEB-INF/classes/META-INF as by adding it into root/META-INF. So there's really no point against expecting files on the classpath which would work for both JAR and WAR files. The only issue is that an augmentation tool would have to treat WAR files differently to JAR files, because the files should be in different locations.

Another (invalid) argument was that there's already a precendence in using root/META-INF. But all examples of this are vendor-specific (tomcat, liberty), while there's a strong precedence in existing specifications to use either META-INF on classpath (works in both JAR and WAR) or WEB-INF (in case of WAR). Adding another META-INF location for WARs causes confusion, as you can see in this StackOverflow question: https://stackoverflow.com/questions/17997731/maven-war-has-meta-inf-folder-in-two-places. The answer pinpoints the servlet spec, which on one hand allows root/META-INF and says it isn't visible to the outside world, on the other hand it states that this is reserved when using the standard Java archive tools and implies that it shouldn't be used in any other way. So the precedence for root/META-INF is much smaller than for other locations and this argument thus isn't valid at all.

On the other hand, there are many arguments for changing the expected location to META-INF on the classpath instead of in the root of a package:

  • using root/META-INF in WAR files isn't portable (this applies to other non-JAR package formats)
  • using root/META-INF in WAR results in multiple META-INF folders if other specs like JPA are used in the same app (also may be valid for other non-JAR package formats)
  • using root/META-INF in WAR leads to confusion for developers because they are used to META-INF in WEB-INF/classes/META-INF and maven projects use this location too. It's not natural to build a WAR file that contains custom files in root/META-INF (see again how this person is trying to package context.xml into root/META-INF in a maven project and fails)
  • the servlet spec discourages using META-INF for other purposes than files generated by packaging tools (see section 10.6 of the servlet spec)
  • although MP doesn't specify any support for WAR packaging, it doesn't specify support for JAR packaging either. It should be completely valid to execute classes from a folder without putting them to a JAR or WAR file. In that case, it's not often clear what is the root/META-INF location because the classpath is assembled by unknown means (by IDE mostly)

It's 5 valid arguments for changing the current behavior against 1 valid argument to keep it.

@arjantijms
Copy link

arjantijms commented Jul 12, 2018

@OndrejM @arthurdm

In addition to agreeing with Ondrej's points raised above, I would like to call special attention to this:

I didn't find any other argument to support the decision, despite it was declared multiple times that the discussion in hangouts concluded with keeping the current state.

So far the discussion has given a distinct feel of "we've chosen it before, maybe not thought too much about it, but now that it's chosen it has been chosen".

MicroProfile however promised to be fast paced, and part of this fast pace mean mistakes slip in. To counter this, it was also promised to be agile with correcting mistakes, as opposed to the Java EE process where things that slipped into a spec release can not really be changed anymore without moving the world and many years of waiting (typically 6 years at least).

I personally think there's very little discussion needed about [war root]/meta-inf being a mistake. We all know it's a mistake, and I think you do as well ;) We all make mistakes (I certainly made tons of them), so there's no shame at all in that.

I assume the issue is solely with the idea of having to correct something in a spec, not about it actually being a mistake. As such I think this should not be an issue at all, as the MP process allows for things to be corrected without too mush hassle.

@Emily-Jiang
Copy link
Member

For .war in CDI, beans.xml can be placed under WEB-INF\beans.xml or WEB-INF\classes\META-INF\beans.xml. If multiple files are found in both locations, an unportable behavior occurs. Both @struberg and I did not like it. Therefore, in MP Config, we simply said the microprofile-config.properties on the classpath will be loaded, which means WEB-INF\classes\META-NF\microprofile-config.properties for .war files or META-INF\microprofile-config.properties for .jar files.

It is unusual to load files from META-INF\ folder under .war. I think in MP we should adopt what MP config has done - classpath only, which is clear and neat. If any vendor wants to load files from different locations, they are free to do so.
@arthurdm are you considering the migration issues for the ones who used Open API v3 migrating to MP Open API? I think in Java, it is uncommon to expect people to place places under META-INF under .war though.

@pilhuhn
Copy link

pilhuhn commented Jul 13, 2018

And then .war/META-INF/*.properties seems to work across many/all(?) servers.

@rmannibucau
Copy link

@pilhuhn nop, .war/META-INF/*.properties is not portable at all and something impossible to use for vendors. .war/WEB-INF/classes/META-INF/*.properties or jar META-INF works everywhere.

@pilhuhn
Copy link

pilhuhn commented Jul 13, 2018

@rmannibucau I guess I was trying to make the distinction between 'de facto' and 'de jure'.

If .war/WEB-INF/classes/META-INF/*.properties� is the only sensible place, then to me the decision is clear. And I guess the specs that introduce such property files should also call that out explicitly.

@arthurdm
Copy link
Member Author

I think this thread has focused too much on the EE classpath, specially for a WAR file. It's important to note that the parent specification for MP OpenAPI is the OpenAPI v3 spec (OAS3), and static (YAML | JSON) OAS3 files come from an API-first design pattern, which is language agnostic (i.e. has nothing to do with Java, annotations, or classpaths). This is where customers design their API separately from their application and without any influence or dependency on what language will implement it - you can have polyglot microservices implementing the same OAS3 doc in HA fashion if you wanted.

So when figuring out the best place the store this static OAS3 doc we took in consideration the common denominator in all Java deployments, including EE, SE, Spring, etc, and JAR is that common artifact - reminding ourselves that WAR, EAR, WAB FAT Jar are all derived from JAR. This document doesn't need to be in the classpath, it just needs to be in a consistent location that vendors know where to look - without worrying about conflicts or changing its location if they change their programming model (i.e. if they move from Spring to EE, or vice versa).

As stated before, Liberty has been doing this for over 2 years now, and we have noticed that adopters of API-design first approach in the field usually have build-time tools which extract the OAS3 from Git, validate against the application (whichever that may be, JAR, WAR, whichever) to ensure the developer didn't break the API contact, and then insert that OAS3 static file into META-INF for the exposure of that API - all in a black box style, without knowledge or special code of classpaths or the type of application we just handled.

@rmannibucau
Copy link

@arthurdm well, actually no. MP is about embrassing EE and being reversed to JakartaEE so if you do something against EE like using vendor specific descriptors or locations like .war/META-INF then you block that process. Functionally there are well defined and known location to meet the same requirement as mentionned previously so no need to reinvent the wheel and loose users.

@arthurdm
Copy link
Member Author

EE ... .war/META-INF

Thanks for confirming my point. =)

@rmannibucau
Copy link

@arthurdm the most important part being the root spec of mp-openapi is mp so EE and it is an openapi integration in EE and not the opposite ;)

@arjantijms
Copy link

@arthurdm

Isn't your reasoning a variant of the age old:

"There might be a person, deep in the jungles of Japan, who might want to use [insert EE tech] with Python/Spring/Ruby/What have you"

In general that leads to very bland, and overly complex designs, for no good. (Python users would never want to use an API that's designed for Java). JSR 77 is a prime example.

I don't mean this in any way as offence, so please bear with me, but if you like to do work for OpenAPI (OAS3) or Spring, wouldn't it be a better idea to work with or for them directly instead of polluting/compromising the EE targeted MP specs?

@OndroMih
Copy link

I think we've heard a lot of opinions already. @arthurdm as a s pre c lead has his own opinion. But the role of spec lead is not to make ultimate decisions but drive collaboration and facilitate agreements. Arthur is failing in this respect. I'm sorry to say so, especially because I haven't met Arthur to have and I'm sure we'd have a friendly talk together.

At this stage, the only reasonable option is to have a wider discussion in the MP community and ultimately vote on a decision among all MP committers where everybody has an equal vote, including spec leads.

@arthurdm
Copy link
Member Author

arthurdm commented Jul 16, 2018

drive collaboration and facilitate agreements

@OndrejM - we have discussed this topic in three different MP OpenAPI hangouts, where I invited the opinions of all members - so definitely opinions were heard and discussed. Just ask @EricWittmann or @ivanjunckes or @msavy or @AndyGee or @leochr

I could have just ignored this issue, like it happens in many other specs (in or outside MP), but instead I gave it the appropriate attention - three hangouts, plus all the discussions here..which don't just reflect my opinions, it reflects the consensus reached at the various hangouts.

We actually even agreed today (as a team) to defer this topic to whatever decision comes out of eclipse/microprofile#46 - so I definitely don't see any problems with the way this particular issue was handled. We discussed, invited opinions, had chats, reached an agreement to move forward.

@EricWittmann
Copy link
Contributor

EricWittmann commented Jul 16, 2018

Ultimately no one on the MP OpenAPI hangout felt strongly enough about this issue one way or the other, so the agreement we reached was to defer to the decision of the arch. board. Also I noticed that the information included in eclipse/microprofile#46 is unclear (at least to me). So I'll update that in order for the board to make an informed decision.

@arjantijms
Copy link

@EricWittmann
[quote]Ultimately no one on the MP OpenAPI hangout felt strongly enough about this issue one way or the other[/quote]

But perhaps that was because all the people on that hangout were the same people who didn't care (or thought about it) in the first place?

@EricWittmann
Copy link
Contributor

Very likely. And without anyone else joining the call, I think a reasonable decision was made: abide by the decision reached on issue eclipse/microprofile#46

Disagree?

@arjantijms
Copy link

Disagree?

It's now probably for the best to wait for 46 then. Still it surprises me how those on the hangout would not feel strongly about this given all the arguments raised by so many different people.

@eclipse eclipse deleted a comment from arjantijms Jul 17, 2018
@eclipse eclipse deleted a comment from arjantijms Jul 17, 2018
@eclipse eclipse deleted a comment from arjantijms Jul 17, 2018
@eclipse eclipse deleted a comment from arjantijms Jul 17, 2018
@eclipse eclipse deleted a comment from arjantijms Jul 17, 2018
@eclipse eclipse deleted a comment from arjantijms Jul 17, 2018
@eclipse eclipse deleted a comment from arjantijms Jul 17, 2018
@eclipse eclipse deleted a comment from arjantijms Jul 17, 2018
@eclipse eclipse deleted a comment from arjantijms Jul 17, 2018
@OndroMih
Copy link

@arjantijms, it seems that your comment was add multiple times. Probably Github messed up, Github was messing up with me yesterday too. I deleted all the redundant comments.

@OndroMih
Copy link

I agree, @EricWittmann with moving the resolution to eclipse/microprofile#46 and many thanks for clarification in that issue, you described the problem very well!

Just one thing to add - it's not easy for all interested parties to join the hangout, especially at the same time. Therefore I wouldn't make a decision during any hangout if there's a wide dispute like this one. It's better to ask on the mailing list and let people vote. Or discuss this in gitter so that it's easier to join the discussion and also keep the history of the discussion.

Hangouts should be for better coordination and not for making decisions.

@arjantijms
Copy link

@OndrejM thanks, GitHub just has an outage in the middle of me posting that comment (https://twitter.com/githubstatus/status/1018911757245538304)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

8 participants