New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add profile for skipping test compilation/resolution #713

Closed
ryan-williams opened this Issue Jun 4, 2015 · 6 comments

Comments

Projects
None yet
3 participants
@ryan-williams
Member

ryan-williams commented Jun 4, 2015

Would it be feasible to support a Maven profile that removes or omits test dependencies? I want to be able to build/install ADAM without resolving/building/installing the tests (while developing/iterating).

I have a workflow where I am mvn installing custom versions of ADAM (version "numbers" are actually git commit SHAs, using the codehaus versions-maven-plugin) in my local Maven cache and then depending on them in my Pageant POM.

I have patches to both ADAM and Spark that I need to use and this seems like the best way to do it.

I'm interested in maximally streamlining the process of building/installing ADAM when I make changes to ADAM, so that I can link against the newly-built version in other projects.

Right now, I run mvn install -DskipTests to install custom ADAM jars in my local Maven cache.

  • (cf. the install-sha script in my fork, which uses -Plocal profile that cuts out the distribution module and the javadoc plugin because I don't care about those).

While this skips running of tests, it still resolves, compiles, and installs the tests / test JARs. This makes my process take noticeably longer, and feels wasteful/wrong on principle.

Attempting to skip compiling (and installing) the tests:

mvn install -Dmaven.test.skip -DskipTests

results in errors like:

[ERROR] Failed to execute goal on project adam-apis_2.10: Could not resolve dependencies for project org.bdgenomics.adam:adam-apis_2.10:jar:e3e7e41: Could not find artifact org.bdgenomics.adam:adam-core_2.10:jar:tests:e3e7e41 in spark-rc (https://repository.apache.org/content/repositories/orgapachespark-1112) -> [Help 1]

…because adam-apis depends on adam-core's tests JAR, even when it won't build/use any tests.

I was able to work-around this in Pageant:

I tried to do a similar thing in ADAM but dealing with the <module>s took me out of my Maven depth quickly; I don't really understand how the <version>-less dependency declarations in the modules' POMs relate to their corresponding definitions in the parent POM.

Anyone have thoughts on how this could be done, and whether it should?

@ryan-williams

This comment has been minimized.

Show comment
Hide comment
@ryan-williams

ryan-williams Mar 15, 2016

Member

Bumping this as I still can't locally mvn install a -SNAPSHOT and sometimes want to test/use a forked version's functionality in a downstream application.

Member

ryan-williams commented Mar 15, 2016

Bumping this as I still can't locally mvn install a -SNAPSHOT and sometimes want to test/use a forked version's functionality in a downstream application.

@fnothaft

This comment has been minimized.

Show comment
Hide comment
@fnothaft

fnothaft Mar 15, 2016

Member

Fascinating! Totally missed this the first time around. I typically just run with -DskipTests. @heuermh can you take a look? If it's not hard to patch, might be reasonable to add a profile.

Member

fnothaft commented Mar 15, 2016

Fascinating! Totally missed this the first time around. I typically just run with -DskipTests. @heuermh can you take a look? If it's not hard to patch, might be reasonable to add a profile.

@ryan-williams

This comment has been minimized.

Show comment
Hide comment
@ryan-williams

ryan-williams Mar 15, 2016

Member

hm mvn install -DskipTests seems to have just worked for me. I'm very confused by that; pretty sure that'd failed previously.

Member

ryan-williams commented Mar 15, 2016

hm mvn install -DskipTests seems to have just worked for me. I'm very confused by that; pretty sure that'd failed previously.

@ryan-williams

This comment has been minimized.

Show comment
Hide comment
@ryan-williams

ryan-williams Mar 18, 2016

Member

OK I figured out the issue: mvn install -DskipTests works when you are on a SNAPSHOT version, because it can resolve test JARs that ADAM's existing SNAPSHOT-uploading infrastructure is uploading to Maven.

If you change the <version> of ADAM to be something new/unique (e.g. I'm currently using a git SHA 5bd2d7e of a branch I'm on), then this issue appears: it can't find the tests jar so it won't install, even when passing -DskipTests and/or -Dmaven.test.skip.

Maybe this is specific to my workflow, but what I'm trying to do seems pretty reasonable and I don't see any way to work around it; it seems like a weird bootstrapping problem: I can only mvn install ADAM core for versions where someone has already uploaded a distribution (including test jars) to Maven.

More about my use case:
I want to use a custom version instead of 0.19.1-SNAPSHOT because when I build Pageant it keeps fetching the latest 0.19.1-SNAPSHOT from Maven, overwriting the fork I'd purposely installed, due to Maven snapshot semantics, I guess? This seems to imply that I can tell it not to clobber snapshots; maybe I'll try that.

The other workaround is just to keep mvn installing 0.19.1-SNAPSHOT each day after I inadvertently clobber the one I'd installed the previous day.

Member

ryan-williams commented Mar 18, 2016

OK I figured out the issue: mvn install -DskipTests works when you are on a SNAPSHOT version, because it can resolve test JARs that ADAM's existing SNAPSHOT-uploading infrastructure is uploading to Maven.

If you change the <version> of ADAM to be something new/unique (e.g. I'm currently using a git SHA 5bd2d7e of a branch I'm on), then this issue appears: it can't find the tests jar so it won't install, even when passing -DskipTests and/or -Dmaven.test.skip.

Maybe this is specific to my workflow, but what I'm trying to do seems pretty reasonable and I don't see any way to work around it; it seems like a weird bootstrapping problem: I can only mvn install ADAM core for versions where someone has already uploaded a distribution (including test jars) to Maven.

More about my use case:
I want to use a custom version instead of 0.19.1-SNAPSHOT because when I build Pageant it keeps fetching the latest 0.19.1-SNAPSHOT from Maven, overwriting the fork I'd purposely installed, due to Maven snapshot semantics, I guess? This seems to imply that I can tell it not to clobber snapshots; maybe I'll try that.

The other workaround is just to keep mvn installing 0.19.1-SNAPSHOT each day after I inadvertently clobber the one I'd installed the previous day.

@heuermh

This comment has been minimized.

Show comment
Hide comment
@heuermh

heuermh Mar 18, 2016

Member

If you look in the Sonatype snapshots repository

https://oss.sonatype.org/content/repositories/snapshots/org/bdgenomics/adam/adam-core_2.10/0.19.1-SNAPSHOT/

you will see timestamped snapshot builds, e.g. adam-core_2.10-0.19.1-20160315.112717-201.jar. You can use these as dependencies, although they won't necessarily be there forever. The same goes for snapshot versions you mvn install to your local repository.

Another workaround is to tell Pageant to use offline mode mvn -o ... so that it won't check with remote snapshot repositories.

Member

heuermh commented Mar 18, 2016

If you look in the Sonatype snapshots repository

https://oss.sonatype.org/content/repositories/snapshots/org/bdgenomics/adam/adam-core_2.10/0.19.1-SNAPSHOT/

you will see timestamped snapshot builds, e.g. adam-core_2.10-0.19.1-20160315.112717-201.jar. You can use these as dependencies, although they won't necessarily be there forever. The same goes for snapshot versions you mvn install to your local repository.

Another workaround is to tell Pageant to use offline mode mvn -o ... so that it won't check with remote snapshot repositories.

@fnothaft

This comment has been minimized.

Show comment
Hide comment
@fnothaft

fnothaft Mar 3, 2017

Member

I'm going to close this as won't fix for now. Please reopen if this is still a pain point, @ryan-williams.

Member

fnothaft commented Mar 3, 2017

I'm going to close this as won't fix for now. Please reopen if this is still a pain point, @ryan-williams.

@fnothaft fnothaft closed this Mar 3, 2017

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment