Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Dependency updates #38

Merged
merged 8 commits into from
Jun 27, 2022
Merged

Dependency updates #38

merged 8 commits into from
Jun 27, 2022

Conversation

dgault
Copy link
Member

@dgault dgault commented Jun 9, 2022

Following on from conversation with @sbesson and @jburel, trying to remove some of the dependencies. Goal of this PR will be to update the below areas:

1 - formats API bumped to latest 6.10.0, this should be straight forward
2 - commons-io removed, only area impacted will be the list of used files. PR aims to keep the reproduce the same list
3 - initially remove netty-nio-client dependency, this looks safe and straight forward

As a follow up I will continue trying to strip out and remove the remaining amazon dependencies which essentially means removing the s3 functionality for the time being.

@sbesson sbesson requested review from sbesson and jburel June 13, 2022 15:41
@jburel
Copy link
Member

jburel commented Jun 13, 2022

Building this PR in ome/openmicroscopy#6324 (I used my branch since this branch is not built on top of main and the version is old).
It now fails with new error

		::::::::::::::::::::::::::::::::::::::::::::::
		::              FAILED DOWNLOADS            ::
		:: ^ see resolution messages for details  ^ ::
		::::::::::::::::::::::::::::::::::::::::::::::
		:: org.objenesis#objenesis;2.5.1!objenesis.jar
		:: joda-time#joda-time;2.2!joda-time.jar
		:: com.google.guava#guava;29.0-jre!guava.jar(bundle)
		:: org.checkerframework#checker-qual;2.11.1!checker-qual.jar
		:: com.google.errorprone#error_prone_annotations;2.3.4!error_prone_annotations.jar
		:: ch.qos.logback#logback-core;1.2.0!logback-core.jar
		::::::::::::::::::::::::::::::::::::::::::::::

@sbesson
Copy link
Member

sbesson commented Jun 14, 2022

Have you established that removing/commenting https://github.com/ome/openmicroscopy/blob/f73008bc34d37f6034a21095c55eb2a8b6075e44/ivy.xml#L21 suffices to fix the build issues? i.e. is this component the only source of issues?

@jburel
Copy link
Member

jburel commented Jun 14, 2022

I have rebuilt using only ome/openmicroscopy#6323 and excluding the ZarrReader and it builds fine

@dgault
Copy link
Member Author

dgault commented Jun 14, 2022

Heres the dependency tree with this PR, other than logback the rest of these are related to jzarr:

[INFO] ome:OMEZarrReader:jar:0.1.6-SNAPSHOT
[INFO] +- com.bc.zarr:jzarr:jar:0.3.4:compile
[INFO] |  +- edu.ucar:cdm-core:jar:5.3.3:compile
[INFO] |  |  +- edu.ucar:udunits:jar:5.3.3:compile
[INFO] |  |  +- edu.ucar:httpservices:jar:5.3.3:compile
[INFO] |  |  |  +- org.apache.httpcomponents:httpclient:jar:4.5.9:compile
[INFO] |  |  |  |  +- commons-logging:commons-logging:jar:1.2:compile
[INFO] |  |  |  |  \- commons-codec:commons-codec:jar:1.11:compile
[INFO] |  |  |  \- org.apache.httpcomponents:httpmime:jar:4.5.9:compile
[INFO] |  |  +- joda-time:joda-time:jar:2.10.3:compile
[INFO] |  |  +- org.jdom:jdom2:jar:2.0.6:compile
[INFO] |  |  +- com.google.code.findbugs:jsr305:jar:3.0.2:compile
[INFO] |  |  +- com.google.protobuf:protobuf-java:jar:3.9.1:compile
[INFO] |  |  +- com.google.guava:guava:jar:28.0-jre:compile
[INFO] |  |  |  +- com.google.guava:failureaccess:jar:1.0.1:compile
[INFO] |  |  |  +- com.google.guava:listenablefuture:jar:9999.0-empty-to-avoid-conflict-with-guava:compile
[INFO] |  |  |  +- org.checkerframework:checker-qual:jar:2.8.1:compile
[INFO] |  |  |  +- com.google.errorprone:error_prone_annotations:jar:2.3.2:compile
[INFO] |  |  |  +- com.google.j2objc:j2objc-annotations:jar:1.3:compile
[INFO] |  |  |  \- org.codehaus.mojo:animal-sniffer-annotations:jar:1.17:compile
[INFO] |  |  +- org.apache.httpcomponents:httpcore:jar:4.4.11:compile
[INFO] |  |  \- com.google.re2j:re2j:jar:1.3:compile
[INFO] |  +- com.fasterxml.jackson.core:jackson-databind:jar:2.11.3:compile
[INFO] |  |  +- com.fasterxml.jackson.core:jackson-annotations:jar:2.11.3:compile
[INFO] |  |  \- com.fasterxml.jackson.core:jackson-core:jar:2.11.3:compile
[INFO] |  \- org.lasersonlab:jblosc:jar:1.0.1:compile
[INFO] |     \- net.java.dev.jna:jna:jar:4.2.2:compile
[INFO] +- org.mockito:mockito-inline:jar:3.7.7:compile
[INFO] |  \- org.mockito:mockito-core:jar:3.7.7:compile
[INFO] |     +- net.bytebuddy:byte-buddy:jar:1.10.19:compile
[INFO] |     +- net.bytebuddy:byte-buddy-agent:jar:1.10.19:compile
[INFO] |     \- org.objenesis:objenesis:jar:3.1:compile
[INFO] +- org.testng:testng:jar:6.8:compile
[INFO] |  +- junit:junit:jar:4.10:compile
[INFO] |  |  \- org.hamcrest:hamcrest-core:jar:1.1:compile
[INFO] |  +- org.beanshell:bsh:jar:2.0b4:compile
[INFO] |  +- com.beust:jcommander:jar:1.27:compile
[INFO] |  \- org.yaml:snakeyaml:jar:1.6:compile
[INFO] +- xalan:serializer:jar:2.7.2:runtime
[INFO] |  \- xml-apis:xml-apis:jar:1.3.04:runtime
[INFO] +- xalan:xalan:jar:2.7.2:runtime
[INFO] \- ome:formats-api:jar:6.10.0:compile
[INFO]    +- org.openmicroscopy:ome-common:jar:6.0.9:compile
[INFO]    |  +- io.minio:minio:jar:5.0.2:compile
[INFO]    |  |  +- com.google.http-client:google-http-client-xml:jar:1.20.0:compile
[INFO]    |  |  |  +- com.google.http-client:google-http-client:jar:1.20.0:compile
[INFO]    |  |  |  \- xpp3:xpp3:jar:1.1.4c:compile
[INFO]    |  |  +- com.squareup.okhttp3:okhttp:jar:3.7.0:compile
[INFO]    |  |  \- com.squareup.okio:okio:jar:1.12.0:compile
[INFO]    |  +- com.esotericsoftware:kryo:jar:4.0.2:compile
[INFO]    |  |  +- com.esotericsoftware:reflectasm:jar:1.11.3:compile
[INFO]    |  |  |  \- org.ow2.asm:asm:jar:5.0.4:compile
[INFO]    |  |  \- com.esotericsoftware:minlog:jar:1.3.0:compile
[INFO]    |  +- ch.qos.logback:logback-core:jar:1.2.9:compile
[INFO]    |  \- ch.qos.logback:logback-classic:jar:1.2.0:compile
[INFO]    +- org.openmicroscopy:ome-xml:jar:6.3.1:compile
[INFO]    |  \- org.openmicroscopy:specification:jar:6.3.1:compile
[INFO]    +- org.openmicroscopy:ome-codecs:jar:0.3.2:compile
[INFO]    |  \- org.openmicroscopy:ome-jai:jar:0.1.3:compile
[INFO]    \- org.slf4j:slf4j-api:jar:1.7.6:compile

@sbesson
Copy link
Member

sbesson commented Jun 14, 2022

Using % mvn dependency:tree -Dverbose=true to include the conflicting dependency (resolved by the build system)

[INFO] ome:OMEZarrReader:jar:0.2.1-SNAPSHOT
[INFO] +- com.bc.zarr:jzarr:jar:0.3.4:compile
[INFO] |  +- edu.ucar:cdm-core:jar:5.3.3:compile
[INFO] |  |  +- edu.ucar:udunits:jar:5.3.3:compile
[INFO] |  |  |  +- (com.google.code.findbugs:jsr305:jar:3.0.2:compile - omitted for duplicate)
[INFO] |  |  |  \- (org.slf4j:slf4j-api:jar:1.7.28:compile - omitted for duplicate)
[INFO] |  |  +- edu.ucar:httpservices:jar:5.3.3:compile
[INFO] |  |  |  +- (com.google.guava:guava:jar:28.0-jre:compile - omitted for duplicate)
[INFO] |  |  |  +- (com.google.code.findbugs:jsr305:jar:3.0.2:compile - omitted for duplicate)
[INFO] |  |  |  +- org.apache.httpcomponents:httpclient:jar:4.5.9:compile
[INFO] |  |  |  |  +- (org.apache.httpcomponents:httpcore:jar:4.4.11:compile - omitted for duplicate)
[INFO] |  |  |  |  +- commons-logging:commons-logging:jar:1.2:compile
[INFO] |  |  |  |  \- commons-codec:commons-codec:jar:1.11:compile
[INFO] |  |  |  +- (org.apache.httpcomponents:httpcore:jar:4.4.11:compile - omitted for duplicate)
[INFO] |  |  |  +- org.apache.httpcomponents:httpmime:jar:4.5.9:compile
[INFO] |  |  |  |  \- (org.apache.httpcomponents:httpclient:jar:4.5.9:compile - omitted for duplicate)
[INFO] |  |  |  +- (com.google.re2j:re2j:jar:1.3:compile - omitted for duplicate)
[INFO] |  |  |  \- (org.slf4j:slf4j-api:jar:1.7.28:compile - omitted for duplicate)
[INFO] |  |  +- joda-time:joda-time:jar:2.10.3:compile
[INFO] |  |  +- org.jdom:jdom2:jar:2.0.6:compile
[INFO] |  |  +- com.google.code.findbugs:jsr305:jar:3.0.2:compile
[INFO] |  |  +- com.google.protobuf:protobuf-java:jar:3.9.1:compile
[INFO] |  |  +- com.google.guava:guava:jar:28.0-jre:compile
[INFO] |  |  |  +- com.google.guava:failureaccess:jar:1.0.1:compile
[INFO] |  |  |  +- com.google.guava:listenablefuture:jar:9999.0-empty-to-avoid-conflict-with-guava:compile
[INFO] |  |  |  +- (com.google.code.findbugs:jsr305:jar:3.0.2:compile - omitted for duplicate)
[INFO] |  |  |  +- org.checkerframework:checker-qual:jar:2.8.1:compile
[INFO] |  |  |  +- com.google.errorprone:error_prone_annotations:jar:2.3.2:compile
[INFO] |  |  |  +- com.google.j2objc:j2objc-annotations:jar:1.3:compile
[INFO] |  |  |  \- org.codehaus.mojo:animal-sniffer-annotations:jar:1.17:compile
[INFO] |  |  +- (com.beust:jcommander:jar:1.78:compile - omitted for conflict with 1.27)
[INFO] |  |  +- org.apache.httpcomponents:httpcore:jar:4.4.11:compile
[INFO] |  |  +- com.google.re2j:re2j:jar:1.3:compile
[INFO] |  |  \- (org.slf4j:slf4j-api:jar:1.7.28:compile - omitted for conflict with 1.7.6)
[INFO] |  +- com.fasterxml.jackson.core:jackson-databind:jar:2.11.3:compile
[INFO] |  |  +- com.fasterxml.jackson.core:jackson-annotations:jar:2.11.3:compile
[INFO] |  |  \- com.fasterxml.jackson.core:jackson-core:jar:2.11.3:compile
[INFO] |  \- org.lasersonlab:jblosc:jar:1.0.1:compile
[INFO] |     +- (junit:junit:jar:4.12:compile - omitted for conflict with 4.10)
[INFO] |     \- net.java.dev.jna:jna:jar:4.2.2:compile
[INFO] +- org.mockito:mockito-inline:jar:3.7.7:compile
[INFO] |  \- org.mockito:mockito-core:jar:3.7.7:compile
[INFO] |     +- net.bytebuddy:byte-buddy:jar:1.10.19:compile
[INFO] |     +- net.bytebuddy:byte-buddy-agent:jar:1.10.19:compile
[INFO] |     \- org.objenesis:objenesis:jar:3.1:compile
[INFO] +- org.testng:testng:jar:6.8:compile
[INFO] |  +- junit:junit:jar:4.10:compile
[INFO] |  |  \- org.hamcrest:hamcrest-core:jar:1.1:compile
[INFO] |  +- org.beanshell:bsh:jar:2.0b4:compile
[INFO] |  +- com.beust:jcommander:jar:1.27:compile
[INFO] |  \- org.yaml:snakeyaml:jar:1.6:compile
[INFO] +- xalan:serializer:jar:2.7.2:runtime
[INFO] |  \- xml-apis:xml-apis:jar:1.3.04:runtime
[INFO] +- xalan:xalan:jar:2.7.2:runtime
[INFO] |  \- (xalan:serializer:jar:2.7.2:runtime - omitted for duplicate)
[INFO] \- ome:formats-api:jar:6.10.0:compile
[INFO]    +- org.openmicroscopy:ome-common:jar:6.0.9:compile
[INFO]    |  +- io.minio:minio:jar:5.0.2:compile
[INFO]    |  |  +- com.google.http-client:google-http-client-xml:jar:1.20.0:compile
[INFO]    |  |  |  +- com.google.http-client:google-http-client:jar:1.20.0:compile
[INFO]    |  |  |  |  \- (org.apache.httpcomponents:httpclient:jar:4.0.1:compile - omitted for conflict with 4.5.9)
[INFO]    |  |  |  \- xpp3:xpp3:jar:1.1.4c:compile
[INFO]    |  |  +- (com.google.guava:guava:jar:25.1-jre:compile - omitted for conflict with 28.0-jre)
[INFO]    |  |  +- com.squareup.okhttp3:okhttp:jar:3.7.0:compile
[INFO]    |  |  |  \- (com.squareup.okio:okio:jar:1.12.0:compile - omitted for duplicate)
[INFO]    |  |  +- com.squareup.okio:okio:jar:1.12.0:compile
[INFO]    |  |  +- (joda-time:joda-time:jar:2.7:compile - omitted for conflict with 2.10.3)
[INFO]    |  |  +- (com.fasterxml.jackson.core:jackson-annotations:jar:2.9.6:compile - omitted for conflict with 2.11.3)
[INFO]    |  |  +- (com.fasterxml.jackson.core:jackson-core:jar:2.9.6:compile - omitted for conflict with 2.11.3)
[INFO]    |  |  \- (com.fasterxml.jackson.core:jackson-databind:jar:2.9.6:compile - omitted for conflict with 2.11.3)
[INFO]    |  +- com.esotericsoftware:kryo:jar:4.0.2:compile
[INFO]    |  |  +- com.esotericsoftware:reflectasm:jar:1.11.3:compile
[INFO]    |  |  |  \- org.ow2.asm:asm:jar:5.0.4:compile
[INFO]    |  |  +- com.esotericsoftware:minlog:jar:1.3.0:compile
[INFO]    |  |  \- (org.sis:objenesis:jar:2.5.1:compile - omitted for conflict with 3.1)
[INFO]    |  +- (joda-time:joda-time:jar:2.2:compile - omitted for conflict with 2.10.3)
[INFO]    |  +- (com.google.guava:guava:jar:29.0-jre:compile - omitted for conflict with 28.0-jre)
[INFO]    |  +- (org.slf4j:slf4j-api:jar:1.7.6:compile - omitted for conflict with 1.7.28)
[INFO]    |  +- ch.qos.logback:logback-core:jar:1.2.9:compile
[INFO]    |  +- ch.qos.logback:logback-classic:jar:1.2.0:compile
[INFO]    |  |  +- (ch.qos.logback:logback-core:jar:1.2.0:compile - omitted for conflict with 1.2.9)
[INFO]    |  |  \- (org.slf4j:slf4j-api:jar:1.7.22:compile - omitted for conflict with 1.7.28)
[INFO]    |  +- (xalan:serializer:jar:2.7.2:runtime - omitted for duplicate)
[INFO]    |  \- (xalan:xalan:jar:2.7.2:runtime - omitted for duplicate)
[INFO]    +- org.openmicroscopy:ome-xml:jar:6.3.1:compile
[INFO]    |  +- (org.openmicroscopy:ome-common:jar:6.0.9:compile - omitted for duplicate)
[INFO]    |  +- org.openmicroscopy:specification:jar:6.3.1:compile
[INFO]    |  +- (org.slf4j:slf4j-api:jar:1.7.6:compile - omitted for conflict with 1.7.28)
[INFO]    |  \- (joda-time:joda-time:jar:2.2:compile - omitted for conflict with 2.10.3)
[INFO]    +- org.openmicroscopy:ome-codecs:jar:0.3.2:compile
[INFO]    |  +- org.openmicroscopy:ome-jai:jar:0.1.3:compile
[INFO]    |  \- (org.openmicroscopy:ome-common:jar:6.0.9:compile - omitted for duplicate)
[INFO]    +- org.slf4j:slf4j-api:jar:1.7.6:compile
[INFO]    +- (xalan:serializer:jar:2.7.2:runtime - omitted for duplicate)
[INFO]    \- (xalan:xalan:jar:2.7.2:runtime - omitted for duplicate)

So most of the dependencies which fail to resolve in the OMERO build system are defined at the low-level ome-common component and conflict with other components also defined in the the core formats-* components like cdm-core.

It's still not fully clear to me why this is an issue with this repository but not with formats-gpl which effectively pulls the same transitive dependencies and exposes them through the downstream omero-* components. My best guess it that the Ivy conflict resolution fails to resolve the dependency tree injected in https://github.com/ome/openmicroscopy/blob/develop/ivy.xml#L21.

As things stand, we have evidence that although our coupled CI workflow is sufficient to include the development ZarrReader into a development OMERO.server, this causes serious issue when trying to build OMERO in isolation. In addition to the impact on a developer workfow, a more critical impact is that release builds are likely to fail for similar reasons that caused ZarrReader to be excluded in the past. So solving this is a blocker to the inclusion of this reader into an OMERO.server release.

Trying to come up with a list of options to explore/discuss:

  • investigate more specifically what causes the dependency resolution to fail and fing either how to update the POM of this component and/or the Ivy settings file in OMERO to use the current injection of ZarrReader
  • review the low-level Bio-Formats dependencies to unify versions and minimize conflicts so that Maven, Gradle & Ivy resolve dependencies identically
  • update the OMERO build system to drop Ivy as the dependency resolver and use another system e.g. Gradle more compatible with the rest of the stack
  • migrate this reader back into the core Bio-Formats components (see also discussion in Integration with Bio-Formats #32) and release it through omero-model etc
  • migrate this reader as an omero-* component using the Gradle build system

@jburel
Copy link
Member

jburel commented Jun 23, 2022

using this PR in openmicroscopy build fails with

		::::::::::::::::::::::::::::::::::::::::::::::
		::              FAILED DOWNLOADS            ::
		:: ^ see resolution messages for details  ^ ::
		::::::::::::::::::::::::::::::::::::::::::::::
		:: org.objenesis#objenesis;2.5.1!objenesis.jar
		::::::::::::::::::::::::::::::::::::::::::::::

https://github.com/ome/openmicroscopy/runs/7022030936?check_suite_focus=true

@jburel
Copy link
Member

jburel commented Jun 23, 2022

This time

		::::::::::::::::::::::::::::::::::::::::::::::
		::              FAILED DOWNLOADS            ::
		:: ^ see resolution messages for details  ^ ::
		::::::::::::::::::::::::::::::::::::::::::::::
		:: junit#junit;4.12!junit.jar
		:: org.hamcrest#hamcrest-core;1.3!hamcrest-core.jar
		::::::::::::::::::::::::::::::::::::::::::::::

If I remember correctly a bump to testng should fix that one

@jburel
Copy link
Member

jburel commented Jun 23, 2022

Copy link
Member

@sbesson sbesson left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One question about overriding vs exclusing the objenesis dependdency.

Otherwise, this seems to fix the downstream build as per #38 (comment) and potentiall unlock the release. Should we try to get this merged and released (proposing 0.3.0 as per the S3 disabling) and try to consume the released version in the openmicroscopy build?

pom.xml Outdated
<dependency>
<groupId>org.objenesis</groupId>
<artifactId>objenesis</artifactId>
<version>2.5.1</version>
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is the version shipped in ome-common (via kryo 4.0.2). From mvn dependency:tree -Dverbose it looks like it is conflicting with the version from org.mockito:mockito-inline:3.7.7.

Declaring it in the POM should suffice although I assume an alternative would be to add and exclude in the mockito dependency?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

From a quick discussion with @dgault and @jburel, a possible simpler and more effective option would simply be to declare mockito and testng as test dependencies (via scope). This might suffice to exclude these from the openmicroscopy Ivy resolution and prevent the copying of unnecessary JARs into the lib folder

@jburel
Copy link
Member

jburel commented Jun 23, 2022

Copy link
Member

@sbesson sbesson left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Barring the removal of the S3 functionality (which will likely need to be reviewed at a later date), the proposed change fix some dependency mismatch in the dependency declaration, classify test-only dependency and remove the usage of commons-io.

I did some minimal testing in our nightly CI server importing a multi-file single image, a WSI and a plate and everything worked as expected.

Under these assumptions, proposing to get this merged, tag the component as 0.3.0 and then bump the version in openmicroscopy in preparation of the release if no objection

@sbesson sbesson merged commit 3205f7a into ome:main Jun 27, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

4 participants