New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Maven builds are not documented in nixpgs manual #19741

Open
cko opened this Issue Oct 20, 2016 · 19 comments

Comments

Projects
None yet
6 participants
@cko
Contributor

cko commented Oct 20, 2016

Many Java Projects are build with Maven. There are a couple of tools around that like mvn2nix and buildMaven which should be documented in the nixpkgs manual.

@Gerschtli

This comment has been minimized.

Contributor

Gerschtli commented Nov 21, 2017

Whats the state of this?
I'm asking because I can't figure out how to do it..

I executed mvn org.nixos.mvn2nix:mvn2nix-maven-plugin:mvn2nix and my default.nix looks like

with import <nixpkgs> { };
(buildMaven ./project-info.json).build

but I' m getting

[INFO] Scanning for projects...
[ERROR] [ERROR] Some problems were encountered while processing the POMs:
[FATAL] Non-resolvable parent POM for org.springframework:gs-rest-service:0.1.0: Cannot access spring-releases (https://repo.spring.io/libs-release) in offline mode and the artifact org.springframework.boot:spring-boot-starter-parent:pom:1.5.8.RELEASE has not been downloaded from it before. and 'parent.relativePath' points at wrong local POM @ line 10, column 13
 @ 
[ERROR] The build could not read 1 project -> [Help 1]
[ERROR]   
[ERROR]   The project org.springframework:gs-rest-service:0.1.0 (/tmp/nix-build-gs-rest-service-0.1.0.jar.drv-0/spring-rest-api/pom.xml) has 1 error
[ERROR]     Non-resolvable parent POM for org.springframework:gs-rest-service:0.1.0: Cannot access spring-releases (https://repo.spring.io/libs-release) in offline mode and the artifact org.springframework.boot:spring-boot-starter-parent:pom:1.5.8.RELEASE has not been downloaded from it before. and 'parent.relativePath' points at wrong local POM @ line 10, column 13 -> [Help 2]
[ERROR] 
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR] 
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/ProjectBuildingException
[ERROR] [Help 2] http://cwiki.apache.org/confluence/display/MAVEN/UnresolvableModelException
builder for ‘/nix/store/d7zmpfil3hhbx868jqsz9kvbx0ln4kxg-gs-rest-service-0.1.0.jar.drv’ failed with exit code 1
error: build of ‘/nix/store/d7zmpfil3hhbx868jqsz9kvbx0ln4kxg-gs-rest-service-0.1.0.jar.drv’ failed

on nix-build.

pom.xml :

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>

    <groupId>org.springframework</groupId>
    <artifactId>gs-rest-service</artifactId>
    <version>0.1.0</version>

    <parent>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-starter-parent</artifactId>
        <version>1.5.8.RELEASE</version>
    </parent>

    <dependencies>
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-web</artifactId>
        </dependency>
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-test</artifactId>
            <scope>test</scope>
        </dependency>
        <dependency>
            <groupId>com.jayway.jsonpath</groupId>
            <artifactId>json-path</artifactId>
            <scope>test</scope>
        </dependency>
        <dependency>
            <groupId>org.nixos.mvn2nix</groupId>
            <artifactId>mvn2nix-maven-plugin</artifactId>
            <version>1.2.0</version>
        </dependency>
    </dependencies>

    <properties>
        <java.version>1.8</java.version>
    </properties>


    <build>
        <plugins>
            <plugin>
                <groupId>org.springframework.boot</groupId>
                <artifactId>spring-boot-maven-plugin</artifactId>
            </plugin>
        </plugins>
    </build>

    <repositories>
        <repository>
            <id>spring-releases</id>
            <url>https://repo.spring.io/libs-release</url>
        </repository>
    </repositories>
    <pluginRepositories>
        <pluginRepository>
            <id>spring-releases</id>
            <url>https://repo.spring.io/libs-release</url>
        </pluginRepository>
    </pluginRepositories>
</project>

If my comment should go in a separate issue, I will move it ;)

@jerith666

This comment has been minimized.

Contributor

jerith666 commented Nov 22, 2017

Thanks for the reminder -- I'd run across this same problem and fixed it locally, but forgot to ever make a PR. Done now as NixOS/mvn2nix-maven-plugin#11.

However, that wasn't enough to get my test case working. I actually ended up abandoning mvn2nix and wrote my own shell script, which I ran on the local maven repo after building the project outside of nix:

#!/usr/bin/env bash

#run 'mvn -Dmaven.repo.local=dir package' (note: *not* dependency:go-offline, that doesn't reliably download everything)
#then 'repo-to-fetchMaven.sh dir > repo.nix'

echo "{fetchMaven} :";
echo "";
echo "let";
echo "";

deps="";

for f in $(find $1 -type f -not -name \*.sha1 -not -name _remote.repositories | sort); do
    groupId=$(echo $(dirname $(dirname $(dirname $f))) | sed "s|^$1||" | sed "s|/|.|g");
    fName=$(basename $f);
    version=$(basename $(dirname $f));
    fNameNoExt=$(basename -s .signature $(basename -s .pom $(basename -s .jar $fName)));
    if echo $fNameNoExt | grep ".-${version}-" > /dev/null; then
        classifier=$(echo $fNameNoExt | sed -E "s/^.*-${version}-//");
        cls="      suffix = \"-$classifier\";
";
    else
        cls="";
    fi;
    artifactId=$(echo $fNameNoExt | sed -E "s/-${version}(-.*)?$//");
    type=$(echo $fName | rev | cut -d . -f 1 | rev);
    sha=$(sha512sum $f | cut -d " " -f 1);
    pkg=$(echo $(dirname $f) | sed "s|^$1||" | sed "s|/|_|g" | sed "s|\.|_|g")_$type;
    cat <<EOF
    $pkg = fetchMaven {
      groupId = "$groupId";
      artifactId = "$artifactId";
$cls      version = "$version";
      type = "$type";
      sha512 = "$sha";
    };
EOF
    deps="$deps $pkg";
done;

echo "";
echo "in";
echo "  [$deps ]";
echo "";
@jerith666

This comment has been minimized.

Contributor

jerith666 commented Nov 22, 2017

I also made this minor change to make buildMaven a bit more flexible, but haven't thought through whether it's ready for upstream or not: jerith666@3f09402.

@Gerschtli

This comment has been minimized.

Contributor

Gerschtli commented Nov 22, 2017

@jerith666 Looks cool, but how would a default.nix look like for such a maven project?

@jerith666

This comment has been minimized.

Contributor

jerith666 commented Nov 23, 2017

Here's what I came up with for apache commons-lang:

$ cat default.nix
with import <nixpkgs> {};

{} :

let

  repoInfo = import ./repo.nix {
    fetchMaven=javaPackages.fetchMaven;
  };

in

javaPackages.mavenbuild rec{
  name = "commons-lang";
  version = "3.5";

  src = fetchgit {
    url = https://git-wip-us.apache.org/repos/asf/commons-lang.git;
    rev = "36f98d87b24c2f542b02abbf6ec1ee742f1b158b"; #sha1 of LANG_3_5 tag;
    sha256 = "16wx2jg8pcfb5mg28pwyihv3gkhgnr3gj2146kps2lngskbpxwpk";
  };

  mavenDeps = repoInfo;

  quiet = false;
  skipTests = false;

  meta = {};

  m2Path = "/org/apache/commons/commons-lang3/${version}";
}
@Gerschtli

This comment has been minimized.

Contributor

Gerschtli commented Nov 23, 2017

What should I do, if the hashes don't match?

trying http://repo1.maven.org/maven2/com/vaadin/external/google/android-json/0.0.20131108.vaadin1/android-json-0.0.20131108.vaadin1.pom
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  2786  100  2786    0     0   2786      0  0:00:01 --:--:--  0:00:01 66333
output path ‘/nix/store/ny0fq46nsg56p5519r29cyhrr8f2x321-android-json-0.0.20131108.vaadin1.pom’ has sha512 hash ‘0z2675wz1axx7vnrvkjr981jnjsk004w2yz3mcf4dl1wkjb0qlmizlmmrm1ccgjm69l1j1vk77mx3bwsmsd8c57z4cch0b4yh75w32p’ when ‘1i0cghq7b89b23a61jl1cq78ci9axvhk21c6g15mldn6kvl3vd14jkp0qd196cdvk921m8q5fpjjxmd98w7yc34d0f23k6d7gakamrb’ was expected
cannot build derivation ‘/nix/store/11hkzx63ybb1kzv69r969z57n87xwnvl-android-json-0.0.20131108.vaadin1.drv’: 1 dependencies couldn't be built

default.nix:

with import <nixpkgs> { };

let
  repoInfo = import ./repo.nix {
    inherit (javaPackages) fetchMaven;
  };
in

javaPackages.mavenbuild rec {
  name = "spring-rest-api";

  src = ./.;

  mavenDeps = repoInfo;

  quiet = false;
  skipTests = true;

  meta = { };

  m2Path = "/de/tobias-happ/spring-rest-api";
}
@jerith666

This comment has been minimized.

Contributor

jerith666 commented Nov 24, 2017

I don't know. I get a third value for the hash:

$ curl -O http://repo1.maven.org/maven2/com/vaadin/external/google/android-json/0.0.20131108.vaadin1/android-json-0.0.20131108.vaadin1.pom

$ sha512sum android-json-0.0.20131108.vaadin1.pom 
570c5e0ef46401c818f9a730d474d57c8d5ecf993bc84093a9f231166aae95fe5829064b4e1e68238ed5f1bde00480a9a59501a52ce7ce769fde55f8bc1c233e  android-json-0.0.20131108.vaadin1.pom
@Gerschtli

This comment has been minimized.

Contributor

Gerschtli commented Nov 24, 2017

Very strange.. :/

@jerith666 jerith666 referenced this issue Dec 16, 2017

Closed

clojure: 1.8.0 -> 1.9.0.273, new tools! #32695

4 of 8 tasks complete
@jerith666

This comment has been minimized.

Contributor

jerith666 commented Dec 21, 2017

@yegortimoshenko said this over in #32695:

Reimplementing native Maven fetcher is a bad idea: your fetcher won't handle classifiers, Maven repositories other than Central, exclusions, custom destFileName or outputDirectory, stripVersion flag, and probably a whole bunch of other corner cases I couldn't come up with right away: https://maven.apache.org/plugins/maven-dependency-plugin/usage.html

It does handle classifiers.

It does not handle maven repositories other than Central. Not sure how often that comes up in practice for open source projects; for corporate closed-source projects, it's probably most frequently the case that everything flows through an internal repo manager (nexus or artifactory); that's handled easily enough IMO, by letting https://github.com/NixOS/nixpkgs/blob/master/pkgs/development/java-modules/m2install.nix#L10 be parameterized.

I'm not sure what the other three things are. If they're options to the maven dependency plugin, I'm not sure they really matter, since the goal is to create a valid normally-formed local repo, not download maven artifacts in a one-off fashion.

@yegortimoshenko

This comment has been minimized.

Member

yegortimoshenko commented Dec 21, 2017

It does not handle maven repositories other than Central. Not sure how often that comes up in practice for open source projects

It will come up very often: http://mvnrepository.com/repos. Sonatype Releases end up in Central, but most other repositories don't. Clojure projects use Clojars primarily. Or, just from yesterday, Hadoop: #32893. Check their POM: https://github.com/apache/hadoop/blob/release-3.0.0-RC1/pom.xml#L61

I'm not sure what the other three things are. If they're options to the maven dependency plugin, I'm not sure they really matter, since the goal is to create a valid normally-formed local repo, not download maven artifacts in a one-off fashion.

The rest specify how to name jars. If POM uses any of these options, this script will fail to generate proper Nix to fetch jars in the first place.

@jerith666

This comment has been minimized.

Contributor

jerith666 commented Dec 21, 2017

Okay, good to know that repos other than central are pretty common -- thanks for the concrete examples!

The rest specify how to name jars. If pom.xml uses these and they are not handled, it just won't build.

I did a quick google for maven destFileName, maven outputDirectory, and maven stripVersion -- in all three cases, google is pointing me at the dependency plugin. But it would seem to me that, in order to fetch artifacts and construct a local maven repo, all we care about are the maven coordinates (5 things: GAV + packaging & classifier) ... what am I missing? Do you have an example pom.xml that uses these features?

@yegortimoshenko

This comment has been minimized.

Member

yegortimoshenko commented Dec 21, 2017

Sure:

However, I was wrong about these options: my understanding was that such artifcats are fetched straight from Maven and mutate name of the artifact inside the local repo, while actually they are just copies.

It's down to remote repositories, then. I think there is an ephemeral _remote.repositories file that could be used to determine which repository should be used.

@jerith666

This comment has been minimized.

Contributor

jerith666 commented Dec 25, 2017

hmm, interesting; apparently there is also a _maven.repositories file; the difference is as yet unclear, but these are good leads. ref: https://stackoverflow.com/questions/16866978/maven-cant-find-my-local-artifacts.

@jerith666

This comment has been minimized.

Contributor

jerith666 commented Dec 25, 2017

An earlier version of your comment also mentioned systemPath e.g. https://github.com/apache/hadoop/blob/release-3.0.0-RC1/hadoop-common-project/hadoop-annotations/pom.xml#L52. This is a bit of a separate issue since these artifacts by definition are not obtained by maven and don't reside in the local repo. But still an important consideration for full maven support, so I don't want to lose track of the issue.

@yegortimoshenko

This comment has been minimized.

Member

yegortimoshenko commented Dec 25, 2017

This is a bit of a separate issue since these artifacts by definition are not obtained by maven and don't reside in the local repo.

That's why I've dropped that comment.

You've changed my mind and now I think this is not only feasible approach, but better than what I've proposed. Thank you for your patience :-)

@jerith666

This comment has been minimized.

Contributor

jerith666 commented Dec 29, 2017

Glad to hear it! :) The manual pre-build + shell script I have here obviously leaves several things to be desired -- I think it's worth contacting the maven folks to see if (a) dependency:go-offline can be made to work more reliably or (b) they have any other ideas.

@yegortimoshenko

This comment has been minimized.

Member

yegortimoshenko commented Dec 29, 2017

I don't think there is a way to make dependency:go-offline work any more reliably: the problem is that some plugins don't declare their dependencies but instead inject them at runtime.

I've had the same problem with Leiningen (Clojure build tool based on Maven): technomancy/leiningen#2055

@jerith666

This comment has been minimized.

Contributor

jerith666 commented Feb 5, 2018

Noting @icetan's https://github.com/icetan/mavenix as another approach to investigate. Cursory glance looks like it's built on top of dependency:go-offline, but I don't claim to have grokked it completely.

@asymmetric

This comment has been minimized.

Contributor

asymmetric commented Jul 26, 2018

@jerith666 I'm giving your script a try. I've seen a couple of issues that I'm currently fixing:

  • It sometimes produces duplicates in the repo.nix file - not sure why they're there, so maybe the script should just check for them
  • The groupId always started with a ., which produced an invalid URL - I've changed it to groupId=$(echo $(cut -d / -f 2));

I'll let you know how building K goes.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment