5288 Introducing proper deps management and docs about it. #5289

poikilotherm · 2018-11-07T15:24:55Z

~~As suggested by @pdurbin on IRC (logs), this PR is primarily about the issues found in #5274.~~ This also paves some of the way to #5288, but lets work in smaller chunks.

Please see my comment below. Splitted this up so it does not break AWS on current Glassfish 4.1 environments.

Related Issues

connects to Dependency Housekeeping and how developers will benefit from it #5288

Pull Request Checklist

Unit tests completed
Integration tests: None
Deployment requirements, SQL updates, Solr updates, etc.: None
Documentation completed
Merged latest from "develop" branch and resolved conflicts

I could use a hand with testing the S3 stuff in some EC2 or so. Don't have this at hand and a deployment with Minio is currently quite an amount of work.

coveralls · 2018-11-07T15:33:42Z

Coverage decreased (-0.0009%) to 16.215% when pulling 96ec306 on poikilotherm:5288-dependency-housekeeping into 0ee43fa on IQSS:develop.

matthew-a-dunlap

This is great! Both that you were able to deal with our huge aws dependency and the documentation on maven.

A few thoughts:

Maybe we should put a note in the top of our pom telling folks to read this doc section before doing edits.
I think the "direct dependency" rules section should be its own section. The background info is pretty vital to understanding this part of maven (and I've definitely learned things reading it) but if folks need to refresh themselves its better if its easier to get to.
In the rules discussing implicit usage, etc, its probably worth pointing to the helpful tools so folks know what they can do.
I feel the lists in Transitive dependencies & managing transitive dependencies overlap somewhat. I'm not sure we need to list all the ways maven can handle them in a list, maybe we can instead add the ways we want to our rules and then discuss the bad ways in a section of text?

Maybe I've missed this in the stories, but I'm curious what you see as the remaining tech debt in our dependency management. We should do our own investigation into maven but if you had some high level thoughts it would help us move forward.

pdurbin

I'm approving this but I'd be happy to fix a typo here and there once @dlmurphy is done making commits. Or we can merge it and fix the typos later.

dlmurphy · 2018-11-15T22:56:13Z

I'm finished making commits, all looks good to me!

poikilotherm · 2018-11-16T07:50:33Z

doc/sphinx-guides/source/developers/documentation.rst

 `adding options to the make call <https://groups.google.com/forum/#!topic/sphinx-users/yXgNey_0M3I>`_.

-On Mac and Linux this should "just work", on Windows this is currently untested. The worst thing that might happen
-is a warning and missing images in your local documentation build.
+This has been tested and works on Mac, Linux, and Windows. If you have not properly configured GraphViz, then the worst thing that might happen is a warning and missing images in your local documentation build.


Great! Good to know this works on a Windows box 😄

poikilotherm · 2018-11-16T07:51:20Z

doc/sphinx-guides/source/developers/dependencies.rst

-          in the model. You might end up using legacy or repackaged classes because of a wrong scope.
-.. [#ide2] This is going to bite back in modern IDEs when importing classes from transitive dependencies by "autocompletion accident".
+
+.. rubric:: Footnotes


Thanks for adding this - I tried to find other examples, but didn't find one.

poikilotherm · 2018-11-16T08:00:53Z

@matthew-a-dunlap

Maybe we should put a note in the top of our pom telling folks to read this doc section before doing edits.

Sure, good idea! Maybe that's even better than my explanation comments. Do feel like adding a commit for this?

I think the "direct dependency" rules section should be its own section. The background info is pretty vital to understanding this part of maven (and I've definitely learned things reading it) but if folks need to refresh themselves its better if its easier to get to.

I am not opinionated on this. @dlmurphy what do you think?

In the rules discussing implicit usage, etc, its probably worth pointing to the helpful tools so folks know what they can do.

Sure, good idea. Feel free to add a commit 👍

I feel the lists in Transitive dependencies & managing transitive dependencies overlap somewhat. I'm not sure we need to list all the ways maven can handle them in a list, maybe we can instead add the ways we want to our rules and then discuss the bad ways in a section of text?

The exclude option is currently in use within the real POM, but I wasn't in the mood to add more docs about that, as IMHO this is last resort stuff. IMHO a developer might enjoy a short overview, which is not easy to get from the detailed blogs and docs out there (took me a while to sort this out... 😉 )

I'm curious what you see as the remaining tech debt in our dependency management.

Let's discuss that in #5288, alright? Maybe the things I collected over there are an inspiration already 😉

pdurbin · 2018-11-16T12:35:57Z

Do feel like adding a commit for this?

Done in 5b49f50 (in pom.xml direct people to new dependencies page)

poikilotherm · 2018-11-21T13:58:45Z

As @kcondon said that this would have broken current setups based on Glassfish 4.1, I once more changed this PR to only address the dep management stuff. Thus reassigning this to #5288 again and hope this can be merged then.

EDIT: Please be aware that I force pushed things! I removed the commit that dealt with the TrueZIP stuff and changed the AWS dep, so this is now just about introducing dep management.

matthew-a-dunlap

This looks good to me, thanks for splitting off the AWS work

…ncy convergence problems. People unaware of direct and transitive dependencies and how to manage them are kindly requested to have a look at the Maven docs and tutorials: * https://maven.apache.org/guides/introduction/introduction-to-dependency-mechanism.html * https://www.davidjhay.com/maven-dependency-management * https://maven.apache.org/enforcer/maven-enforcer-plugin/index.html Docs about this will follow in the Dataverse dev guide, too.

As JaCoCo has been a direct dependency in *compile* scope. At a few places accidentially some classes from transitive dependencies from JaCoCo have been used instead of the classes used commonly within the codebase. Removing the JaCoCo direct dependency as only the plugin is used at build/test time. It is sufficient for Maven to have the plugin included with its version number - this is handled as a transitive dependency during the build internally by Maven. Hopefully this created no harm, but better fix this now.

Needed because the transitive dependency from AWS has been used before. Thus this relates to IQSS#5274. As best practise, code should not rely on those deps but add a direct dependency. For the sake of working on IQSS#4260 (Java EE 8) at some point in the future, please remember to refactor the code using Jackson (and Gson) and remove it in favor of Java EE 8 native JSON-B and JSON-P support.

…follow. Added a note about GraphViz requirement for building the docs.

Added that GraphViz works with Windows.

Made some typo fixes and such. Looks good!

Fixed a syntax problem that was throwing up sphinx errors.

poikilotherm force-pushed the 5288-dependency-housekeeping branch 2 times, most recently from e00b23c to 0990d83 Compare November 14, 2018 15:40

poikilotherm force-pushed the 5288-dependency-housekeeping branch 4 times, most recently from 2760fad to b5c12b6 Compare November 14, 2018 21:16

poikilotherm changed the title ~~WIP - 5288 Dependency Housekeeping~~ 5274 Stripping down AWS, solving dependency conflicts Nov 14, 2018

poikilotherm mentioned this pull request Nov 14, 2018

Stripping AWS Mail component with TrueZIP blocks packaging for docker images #5274

Closed

pdurbin requested a review from dlmurphy November 14, 2018 21:54

poikilotherm force-pushed the 5288-dependency-housekeeping branch 2 times, most recently from 23eb913 to 83031f3 Compare November 15, 2018 09:10

poikilotherm mentioned this pull request Nov 15, 2018

As a developer I don't want outages of http://maven.geotoolkit.org to prevent me from compiling a war file #5312

Closed

matthew-a-dunlap reviewed Nov 15, 2018

View reviewed changes

pdurbin approved these changes Nov 15, 2018

View reviewed changes

poikilotherm commented Nov 16, 2018

View reviewed changes

poikilotherm force-pushed the 5288-dependency-housekeeping branch 3 times, most recently from 53e055f to 17648f2 Compare November 21, 2018 13:57

poikilotherm changed the title ~~5274 Stripping down AWS, solving dependency conflicts~~ 5288 Introducing proper deps management and docs about it. Nov 21, 2018

This was referenced Nov 21, 2018

Dependency Housekeeping and how developers will benefit from it #5288

Closed

Flyway: Add the database migrator in order to make database migration happen automatically. #5344

Closed

matthew-a-dunlap approved these changes Nov 27, 2018

View reviewed changes

poikilotherm added 2 commits November 29, 2018 15:11

Add proper checksums so Maven is happier (needed when running with -C).

ae40cb7

poikilotherm and others added 8 commits November 29, 2018 15:11

Add developer documentation about dependency management and rules to …

ac9d775

…follow. Added a note about GraphViz requirement for building the docs.

Add some notes about repo ordering, as requested in IQSS#5312

0fb15c0

Updating Documentation page

2955d98

Added that GraphViz works with Windows.

Review of Dependencies page

bab2eda

Made some typo fixes and such. Looks good!

Footnote formatting fix

b09af1f

Fixed a syntax problem that was throwing up sphinx errors.

direct pom.xml hackers to dependencies page IQSS#5274

96ec306

poikilotherm force-pushed the 5288-dependency-housekeeping branch from 17648f2 to 96ec306 Compare November 29, 2018 14:11

kcondon merged commit 3275681 into IQSS:develop Nov 30, 2018

poikilotherm mentioned this pull request Dec 3, 2018

As a documentation writer, I'd like to create images with inline markup. #5365

Closed

pdurbin added this to the 4.10 - Additional Data Transfer Options milestone Dec 19, 2018

pdurbin mentioned this pull request Jan 23, 2019

As a developer, I want to see code coverage from Netbeans #5483

Closed

poikilotherm referenced this pull request in smillidge/dataverse Jun 2, 2019

5736 clean dependencies

ba42682

poikilotherm deleted the 5288-dependency-housekeeping branch September 13, 2019 07:18

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

5288 Introducing proper deps management and docs about it. #5289

5288 Introducing proper deps management and docs about it. #5289

poikilotherm commented Nov 7, 2018 •

edited

Loading

coveralls commented Nov 7, 2018 •

edited

Loading

matthew-a-dunlap left a comment

pdurbin left a comment

dlmurphy commented Nov 15, 2018

poikilotherm Nov 16, 2018

poikilotherm Nov 16, 2018

poikilotherm commented Nov 16, 2018

pdurbin commented Nov 16, 2018

poikilotherm commented Nov 21, 2018 •

edited

Loading

matthew-a-dunlap left a comment

5288 Introducing proper deps management and docs about it. #5289

5288 Introducing proper deps management and docs about it. #5289

Conversation

poikilotherm commented Nov 7, 2018 • edited Loading

Related Issues

Pull Request Checklist

coveralls commented Nov 7, 2018 • edited Loading

matthew-a-dunlap left a comment

Choose a reason for hiding this comment

pdurbin left a comment

Choose a reason for hiding this comment

dlmurphy commented Nov 15, 2018

poikilotherm Nov 16, 2018

Choose a reason for hiding this comment

poikilotherm Nov 16, 2018

Choose a reason for hiding this comment

poikilotherm commented Nov 16, 2018

pdurbin commented Nov 16, 2018

poikilotherm commented Nov 21, 2018 • edited Loading

matthew-a-dunlap left a comment

Choose a reason for hiding this comment

poikilotherm commented Nov 7, 2018 •

edited

Loading

coveralls commented Nov 7, 2018 •

edited

Loading

poikilotherm commented Nov 21, 2018 •

edited

Loading