Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

quarkus-universe-bom too large for Azure Artifacts in 1.6.0.Final #10648

Closed
hacst opened this issue Jul 10, 2020 · 13 comments
Closed

quarkus-universe-bom too large for Azure Artifacts in 1.6.0.Final #10648

hacst opened this issue Jul 10, 2020 · 13 comments
Assignees
Labels
kind/bug Something isn't working

Comments

@hacst
Copy link
Contributor

hacst commented Jul 10, 2020

Describe the bug
This isn't an actual Quarkus bug but @aloubyansky recommended to open one anyways.

With 1.6.0.Final quarkus-universe-bom is now 402KB. This exceeds the maximum pom.xml size limit of Azure Artifacts Feeds which is 384KB. This means trying to download Quarkus through such a feeds (same use-case as a nexus proxy upstream) will fail. I do realize that this is on Azure Artifacts. Unfortunately it doesn't look like they will fix this anytime soon (https://developercommunity.visualstudio.com/idea/809379/remove-or-increase-the-hard-limit-of-384000-bytes.html).

I do not really expect Quarkus to work around this - though Azure Artifact users would definitely be thankful - but I thought I still should raise the issue somewhere in case others hit it.

The current workaround I'm thinking of implementing is to remove the unneeded whitespace from the pom. That leaves it 305KB which I should then be able to publish to the feed directly. Not a great solution and if I mess it up the version is burned. Also has to be re-done for every new version and might ultimately fail if the "minified" pom becomes too large. If the project wants to work around this I think one potential way would be to split the pom by (ab)using parent poms.

Expected behavior
Can use new Quarkus through an Azure Artifacts feed.

Actual behavior
Maven fails to download dependencies with:

Could not transfer artifact io.quarkus:quarkus-universe-bom:pom:1.6.0.Final from/to *** (***/maven/v1): Transfer failed for ***/maven/v1/io/quarkus/quarkus-universe-bom/1.6.0.Final/quarkus-universe-bom-1.6.0.Final.pom 400 Bad Request - pom.xml file is too large; the limit is 384000 bytes (DevOps Activity ID: 08718F7F-368C-4378-BE8B-2829908129D1)

To Reproduce
Steps to reproduce the behavior:

  1. Setup an azure artifact feed with upstream enabled
  2. Try to download the quarkus-universe-bom 1.6.0.Final from it

The feed I encountered this on was on an on-premise Azure DevOps Server 2019 though the issue should be the same for the cloud based Feeds.

@hacst hacst added the kind/bug Something isn't working label Jul 10, 2020
@aloubyansky aloubyansky self-assigned this Jul 11, 2020
@hacst
Copy link
Contributor Author

hacst commented Sep 29, 2020

@aloubyansky As feared with Quarkus 1.8.1.Final my workaround is no longer possible. The "minified" pom is now 389578 exceeding the limit of 384000. Do you have any other suggestions for a viable workaround?

@aloubyansky
Copy link
Member

No workaround on our side at this point, unfortunately. However, I didn't forget about this issue and we are re-working the way we compose the BOM. Now there is an actual generator that puts it all together and it seems feasible to introduce fragmentation that would help this case. Which, tbh, is not something we'd probably prefer in the first place and this issue is probably the only reason to consider it. I'll try to collect opinions on this.

@maxandersen
Copy link
Member

@hacst i'm not user of azure artifacts, but do I get it right that in your usecase here it is used as a proxy for your builds to run faster (or even just be able to build as external maven is cut-off?)

@maxandersen
Copy link
Member

I've pinged my MS contacts on this as it sounds totally arbitrary they have a limits on pom.xml sizes as you'll have .jars even larger...lets figure out why they have that limit and if can be increased at it just feels very long trying to work around this specific limit.

@gsmet
Copy link
Member

gsmet commented Sep 29, 2020 via email

@hacst
Copy link
Contributor Author

hacst commented Sep 29, 2020

@aloubyansky Thanks for your effort.
@maxandersen Correct. We try to follow best practice described by MS for a cooperate setup which is to use a proxy repository for maven access through an artifact feed and prevent any direct access to upstreams (maven central). Besides the speedup you mentioned this keeps builds working if the upstream has issues, if packages are removed or if there are general connectivity issues.

Microsoft would definitely be the only party able to really fix this issue. Everyone else can only work around. Unfortunately even if they fix it tomorrow it would likely take a at least half a year to reach their on-premise product we are using (better then never ofc.). I'll probably have to start stripping out parts of the pom we don't use right now to kick the can down the road (which is just awful).

@aloubyansky
Copy link
Member

@hacst have you tried using io.quarkus:quarkus-bom instead? Are you using extensions that aren't available in it?

@hacst
Copy link
Contributor Author

hacst commented Sep 29, 2020

@aloubyansky I haven't. It is only 200K so would work. What is the difference to the universe bom? I only ever saw universe mentioned in documentation.

@aloubyansky
Copy link
Member

quarkus-bom is limited to the extensions from the quarkus repo https://github.com/quarkusio/quarkus/tree/master/extensions which may already be more than enough for many apps. quarkus-universe-bom however is defined here https://github.com/quarkusio/quarkus-platform/blob/master/bom/pom.xml it import quarkus-bom plus external extensions such as camel, kogito, qpid, cassandra, hazelcast, debezium, etc.

@maxandersen
Copy link
Member

@hacst lets see what MS contacts says - and if doesn't happen you could also split up very crudly the file with a script.

as i assume their system is fine having N smaller poms ?

@hacst
Copy link
Contributor Author

hacst commented Sep 29, 2020

@aloubyansky Interesting. It will cause some confusion to switch this to something not in the docs but it is a much saner workaround than me messing with a released pom. If we encounter specific stuff missing we should still be able to "vendor" that part somehow (should hopefully be rare). Thanks a lot for making me aware of this.

@maxandersen The limit is per pom file. We have tons of other dependencies being pulled through the feed so if we hit some pom count limit it wouldn't be due to such a workaround ;)

@aloubyansky
Copy link
Member

I think this one should be out-of-date now. In addition, the universe BOM was deprecated in Quarkus 2.1 in favor of platform member-specific BOMs.

@maxandersen
Copy link
Member

Anyone stopping by its good to know that Microsoft fixed/patched their azure artifacts to now compress bigger pom files so it should now work. As long as you use a recent Azure Artifacts release.

see details at https://developercommunity.visualstudio.com/t/remove-or-increase-the-hard-limit-of-384000-bytes/809379#TPIN-N10535242

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
kind/bug Something isn't working
Projects
None yet
Development

No branches or pull requests

4 participants