Java packages #20444

NeQuissimus · 2016-11-15T20:11:46Z

Note: I am replacing #12770 with this.

I have taken most of the concepts from the mentioned PR and cleaned them up.
I have also added a full build of JUnit 4.12 and a sample project, which uses said JUnit version.

Most artifacts are binaries because an extremely simplistic project like the one I created requires hundreds of artifacts. Building all of those from source is (surprisingly) impossible. - The Java world for you :D

I am terrible at rebasing in git, so I suggest we squash upon PR merge. (I did things out of order and in weird chunks, so it would not be a straight rebase with squashing; unless we remove context)

Here is what I have done:

Build support

Essentially what was already in #12770 except much less copying because as it turns out, Maven does like symbolic links if you do it right :)
I have a way to fetch Maven Central binaries (fetchMaven) and a build helper (mavenbuild).
I have also added optional skipTests and quiet parameters (both of which default to true).

Why default skipTests to true?
Turns out, JUnit expects hamcrest-core on the classpath without defining it as an explicit runtime dependency. If somebody knows how to convince a pom.xml that depends on JUnit but not hamcrest-core to run tests in offline mode, please do provide feedback.

Plugins

I have added a large number of default Maven plugins.
Theses are needed to get Maven to do anything at all...
I have created mavenMinimal for this purpose, a list of plugins and their dependencies that are required no matter what you are building. This list of automatically included when using mavenbuild.

Libraries

So far, I have all the libraries that are needed to

Run Maven (3.3.9)
Build JUnit 4.12
Run tests using JUnit 4.12

Mirror

I have added a maven alias to the fetchurl mirrors. This is not strictly required but may help in the future if URLs change or another mirror can be added.

Useful stuff

JUnit 4.12
Maven-Hello v1.0 and v1.1 (sample project to prove things work, code on GitHub)

Collections

Generally, if you have one of the Maven libraries that you depend on, they eventually pull in all their brothers and sisters with the same versions numbers as well. So I created mavenLibs collections to make things easier.

SHA512

Sorry @copumpkin, I know we just talked about it but the number of SHA512 in my branch had already been in the triple digits, so I kept it this way. (Forgive me :D)

Maven

The Maven/Java environment is dirty, crazy and unorganized for the most part.
Try building a "Hello World" application and you need to pull in hundreds of dependencies.
Don't even think about building all of those from source (see #12770) for examples of circular dependencies and seemingly endless chains of dependencies that will feel like time travel into the beginnings of Java.
This is why the code in this PR looks the way it does and lists more than 3,500 new LoC.
Compared to the old PR I have already abstracted and massively simplified...

Future

I would ask for any feedback I can get here.
This has been a lot of work but if most people are against this, I won't be mad.
If we decide to include javaPackages into nixpkgs, we should definitely write some documentation, which is why I added the appropriate label.

Note sure what else to say...

mention-bot · 2016-11-15T20:11:49Z

@NeQuissimus, thanks for your PR! By analyzing the history of the files in this pull request, we identified @edolstra, @cmfwyp and @civodul to be potential reviewers.

NeQuissimus · 2016-11-15T20:12:58Z

/cc @copumpkin @shlevy @puffnfresh @bendlas @Mathnerd314 because they all had comments on the old PR

kamilchm · 2016-11-16T08:17:30Z

pkgs/development/java-modules/maven-minimal.nix

+      commonsLoggingApi_1_1
+      findbugsJsr305_2_0_1
+      googleCollections_1_0
+      junit_3_8_1


junit_3_8_1 + junit_3_8_2 - isn't that duplicate classes?

At runtime, yes... But they are both required as transitive dependencies...

strange, things like this could leads to https://en.wikipedia.org/wiki/Java_Classloader#JAR_hell and there are tools to work with it, like http://docs.jboss.org/tattletale/userguide/1.1/html/reports.html#multiplejarfiles
But it could be more an issue for maven itself than this derivation.
How did you generate the list of dependencies?

I removed my local ~/.m2 built the project using the "normal" way and then did a find ~/.m2/repository -name '*.jar' -o -name '*.pom'

IMO maven shouldn't use junit at runtime at all. On a first glance it could be test dependency of https://mvnrepository.com/artifact/org.apache.maven.surefire/maven-surefire-common/2.17
Anyway, it's ok if it doesn't break it.
We could try to optimize it later. It may be worth to dig into mvn dependency:tree and treat compile, test, provided scopes differently instead of listing ~/.m2 tree.

Agreed, you could strip certain dependencies if you choose to skip tests. However, that did not seem worth the effort right now. Most builds from source should want to not skip tests :)

kamilchm · 2016-11-16T08:24:03Z

Just an idea. It would be nice if fetchMaven could use attribute names from maven space, like:

fetchMaven rec {
  version = "2.4";
  artifactId = "maven-install-plugin";
  groupId = "org.apache.maven.plugins";
...

instead of:

fetchMaven rec {
  version = "2.4";
  baseName = "maven-install-plugin";
  package = "/org/apache/maven/plugins";
...

NeQuissimus · 2016-11-16T17:36:49Z

Re: groupId/artifactId

YES!

NeQuissimus · 2016-12-02T01:57:27Z

Ping :)

garbas · 2017-01-04T04:29:45Z

restarted travis builds

garbas · 2017-01-04T04:32:19Z

looks like travis fails

garbas · 2017-01-04T04:32:55Z

sorry i just notices, that this PR is meant for feedback. ignore me

copumpkin · 2017-02-13T13:50:32Z

👍

copumpkin · 2017-02-13T13:52:33Z

I say we merge it and iteratively improve on it. It fills in a big gap in nixpkgs today, and isn't going to hurt/conflict with any of the other package ecosystems by being merged.

globin · 2017-02-13T14:12:19Z

The travis failures definitely look valid.

NeQuissimus · 2017-02-13T14:41:39Z

I will take a look at this today or tomorrow (I hope). Seems that Maven Central is sensitive to the (unintentional) double slash all the sudden...

NeQuissimus · 2017-02-13T23:41:10Z

Found two typos, everything should be good to go now. Let's see what Travis thinks...

NeQuissimus · 2017-02-14T13:41:00Z

There we go, Travis is happy

copumpkin · 2017-02-14T14:00:37Z

One more question: do people always just ask for the pom and jar combination or are there use cases for only asking for one? I'm wondering if we could cut down on the hash noise a bit by making fetchMaven do a recursive hash on the two files it retrieves at the same time. Not sure if that would complicate other things, but copying and pasting long strings of junk is bad enough without having to do it twice for every package 😄

copumpkin · 2017-02-14T14:02:04Z

For example, I have a (yet unpushed, sorry) fetchIvy fetcher that will use ivy to retrieve a whole pile of dependencies at once, and just hash once per bundle of dependencies.

NeQuissimus · 2017-02-14T14:18:49Z

I found it actually depends on the artifact... Some are POM-only, most want JAR and POM but it is technically allowed to only use the JAR and leave the POM alone (i.e. if you want to avoid pulling transitive dependencies). By doing it this way, I just decided to give all the options :)

copumpkin · 2017-02-14T14:20:36Z

@NeQuissimus I guess I'm saying that you could still allow people to ask for all those combinations without requiring one hash per thing you ask for. It might not be worth it, but as a potentially avid user of this thing I'm a little concerned about the code's UX 😄

NeQuissimus · 2017-02-14T15:17:56Z

I am hoping to eventually have some sort of auto-generation. I mean, for now if you want to build your own Maven project, you will probably need to do so much by hand, it is going to get cumbersome real quick. This is essentially more of a PoC than anything, I suppose :)

spacekitteh · 2017-02-26T09:48:51Z

How's this going?

NeQuissimus · 2017-02-26T17:03:09Z

This is done from my POV

copumpkin · 2017-02-26T17:04:29Z

Fine by me!

copumpkin · 2017-02-26T17:06:08Z

pkgs/top-level/all-packages.nix

@@ -9761,6 +9761,10 @@ in

  leaps = callPackage ../development/tools/leaps { };

+  ### DEVELOPMENT / JAVA MODULES
+
+  javaPackages = recurseIntoAttrs (callPackage ./java-packages.nix { });


I actually think this should be callPackages rather than callPackage. The difference is that the former recurses (one level) into the inner attrset and makes the individual derivations there overridable. I think you'll find that in the current setup, override[Derivation] won't work on anything inside there.

That's going to mess with my lists. Is there a straight-forward List -> Set function?
callPackages wants things to be sets.

Hmm, I don't see the lists you're speaking of. Can you point them out? It looks like java-pacakges.nix results in a big attrset full of attributes it inherits from other .nix files?

This is what I get

~/dev/nixpkgs java_packages ✗ ▶ nix-build -A javaPackages.mavenHello_1_1 --show-trace error: while evaluating the attribute ‘buildPhase’ of the derivation ‘maven-hello-1.1’ at /home/nequi/dev/nixpkgs/pkgs/development/java-modules/build-maven-package.nix:10:10: while evaluating ‘optionalString’ at /home/nequi/dev/nixpkgs/lib/strings.nix:138:26, called from /home/nequi/dev/nixpkgs/pkgs/development/java-modules/build-maven-package.nix:24:7: while evaluating ‘unique’ at /home/nequi/dev/nixpkgs/lib/lists.nix:432:12, called from /home/nequi/dev/nixpkgs/pkgs/development/java-modules/build-maven-package.nix:12:14: while evaluating ‘flatten’ at /home/nequi/dev/nixpkgs/lib/lists.nix:90:13, called from /home/nequi/dev/nixpkgs/pkgs/development/java-modules/build-maven-package.nix:12:22: while evaluating the attribute ‘mavenMinimal’ at /home/nequi/dev/nixpkgs/pkgs/development/java-modules/maven-minimal.nix:13:3: while evaluating ‘flatten’ at /home/nequi/dev/nixpkgs/lib/lists.nix:90:13, called from /home/nequi/dev/nixpkgs/pkgs/development/java-modules/maven-minimal.nix:13:18: while evaluating ‘concatMap’ at /home/nequi/dev/nixpkgs/lib/lists.nix:79:18, called from /home/nequi/dev/nixpkgs/lib/lists.nix:92:10: while evaluating anonymous function at /home/nequi/dev/nixpkgs/lib/lists.nix:92:21, called from undefined position: while evaluating ‘flatten’ at /home/nequi/dev/nixpkgs/lib/lists.nix:90:13, called from /home/nequi/dev/nixpkgs/lib/lists.nix:92:24: while evaluating ‘mkAttrOverridable’ at /home/nequi/dev/nixpkgs/lib/customisation.nix:111:33, called from /home/nequi/dev/nixpkgs/lib/attrsets.nix:199:52: value is a list while a set was expected

In maven-minimal I build up a long list of default dependencies.

Also genAttrs if you want to specify a function to create attribute names

This is going to open up all kinds of weird little things. I won't get around to doing this in the near future... I had hoped to get this into the next release, but I will try to fix this first.

dang :\ maybe try to break it up into smaller, mergable chunks?

or merge it now as is, and then work on fixing it

This is pretty much the bare minimum to build anything with Maven :) If we can deal with callPackage instead of callPackages, let's merge...

copumpkin · 2017-02-28T01:04:43Z

Sure, fine by me. Thanks for the effort and patience!

spacekitteh · 2017-02-28T01:16:12Z

Don't forget to make an issue in the issue tracker btw!

sephalon · 2018-04-23T20:03:51Z

@NeQuissimus I'm trying to package an Ant-based project right now that needs a couple of JARs not shipped inside the tarball from Maven Central. Could you please give an example on how to add a Java package to the build inputs and modify the classpath?

NeQuissimus · 2018-04-23T21:29:56Z

You're going to need to write something similar to pkgs/development/java-modules/m2install.nix. I have not dealt with this stuff in a while, sorry.

NANASHI0X74 · 2020-12-10T14:14:56Z

looks like no one got round to doing all those improvements and fixes in the end, huh? 🙂

I'm currently wondering about wether to build an existing maven based project with nix. I'm fairly new to writing my own derivations and it's been quite a while since I've used java and I've also never really got to know the build systems in the java ecosystem, mostly I relied on my IDE doing the work for me back then.
Some documentation on how to use this to package maven projects would be nice :D
when is fetchmaven needed for example?
I'll just try to naively use mavenbuild on the repository like it's done with maven-hello.
Maybe I can write the documentation myself later if I succeed

NeQuissimus and others added 9 commits November 14, 2016 15:19

Build support

d849c0e

Partial junit_4_12

1f2e19b

Partial maven-hello

38e762c

Fixup

f23c445

Finish maven-hello / framework

8500c24

junit_4_12 part 2

5d54cfd

maven_hello: Add 1.1, add ability to choose skipping tests

0e10bfa

Complete hello world with test

63a46e0

Create mavenLibs

82c1ff5

NeQuissimus added 9.needs: changelog 9.needs: community feedback 9.needs: documentation labels Nov 15, 2016

kamilchm reviewed Nov 16, 2016

View reviewed changes

javaPackages: Use artifactId/groupId

8a7a44c

NeQuissimus force-pushed the java_packages branch from 91c5907 to 8a7a44c Compare February 13, 2017 23:40

NeQuissimus mentioned this pull request Feb 19, 2017

Packaging Gradle-based applications #17342

Closed

copumpkin reviewed Feb 26, 2017

View reviewed changes

copumpkin merged commit 1f77939 into NixOS:master Feb 28, 2017

NeQuissimus deleted the java_packages branch February 28, 2017 13:00

NeQuissimus mentioned this pull request Feb 28, 2017

Java packages: Use callPackages #23271

Open

tomodachi94 added the 6.topic: java Including JDK, tooling, other languages, other VMs label Nov 4, 2024

Java packages #20444

Java packages #20444

Conversation

NeQuissimus commented Nov 15, 2016 • edited Loading

mention-bot commented Nov 15, 2016

NeQuissimus commented Nov 15, 2016

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kamilchm Nov 16, 2016 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kamilchm commented Nov 16, 2016

NeQuissimus commented Nov 16, 2016

NeQuissimus commented Dec 2, 2016

garbas commented Jan 4, 2017

garbas commented Jan 4, 2017

garbas commented Jan 4, 2017

copumpkin commented Feb 13, 2017

copumpkin commented Feb 13, 2017

globin commented Feb 13, 2017

NeQuissimus commented Feb 13, 2017

NeQuissimus commented Feb 13, 2017

NeQuissimus commented Feb 14, 2017

copumpkin commented Feb 14, 2017

copumpkin commented Feb 14, 2017

NeQuissimus commented Feb 14, 2017

copumpkin commented Feb 14, 2017

NeQuissimus commented Feb 14, 2017

spacekitteh commented Feb 26, 2017

NeQuissimus commented Feb 26, 2017

copumpkin commented Feb 26, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

copumpkin commented Feb 28, 2017

spacekitteh commented Feb 28, 2017

sephalon commented Apr 23, 2018

NeQuissimus commented Apr 23, 2018

NANASHI0X74 commented Dec 10, 2020

NeQuissimus commented Nov 15, 2016 •

edited

Loading

kamilchm Nov 16, 2016 •

edited

Loading