Pekko Connectors 1.1.x dependency upgrades

[pjfanning]

We were conservative with dependency upgrades in Pekko Connectors 1.0.x to make it easier for users coming from Alpakka to uptake Pekko Connectors.
The aim with Pekko Connectors 1.1.x is to use newer dependencies where feasible.

Some exceptions have come up

• Hbase testing is broken so it makes it difficult to upgrade its connector dependencies - hbase CI build is disabled #61
• SQS connector doesn't work with latest AWS lib - Update sqs to 2.23.17 #441
• Latest versions of Spring only support Java 17. We still support Java 8. https://github.com/apache/incubator-pekko-connectors/blob/4223ac20b3eec15fd3a2020b86397c0bcf262056/.scala-steward.conf
  • I ran a test locally with the latest jars and using Java 17 and the tests pass
• ActiveMQ libs now have Java 11 minimum
• Some test libs like mockito only support Java 11 in their latest releases
• Latest protobuf and related grpc releases cause us issues - google-pub-sub-grpc, google-cloud-bigquery-storage: Bump pubsub protos and protobuf-java dep #277 and others
  • I've tested locally and using pekko-grpc 1.1 snapshots - which have newer protobuf dependencies - means that you can upgrade some of the protobuf linked jars - i.e. pekko-grpc 1.0 compatibility is preventing us from upgrading some protobuf linked jars, like google-cloud-pubsub
• couchbase client v3 causes us compile problems - Update java-client to 3.5.2 #400
• latest kudu jars seem to be restructured in a way that breaks our build
• solr 9 causes us compile issues - Update solr-solrj, solr-test-framework to 9.4.1 #379
• hoverfly upgrade issues
• we have started using slf4jv2 in pekko and pekko http 1.1 but some changes need to be made in pekko connectors to support it

[mdedetrich]

As I see it, I envisage pekko connectors 1.1.x having multiple milestones as we slowly enable the latest versions of the sdk's/apis that we wrap over

[raboof]

We should probably make sure those known problems have GitHub issues, and exclude them from scala-steward with a reference to those issues, else we'll keep running into them :).

I think in pekko-connectors we should move a bit faster with breaking changes, both by upgrading to newer major versions of dependencies and by requiring newer Java versions. If that means creating more major versions than 'usual' I'm OK with that. A first step towards that would be #443.

[btomala]

Sorry, my bad the dependencies is on jwt-json-common which is used by jwt-circe

[pjfanning]

Have you tried just updating the version of jwt-json-common by adding a dependency to it in your build? I am not aware of any code changes that we had to make to facilitate upgrading the jwt-json-common jar.

[btomala]

I tried but I'm getting version conflicts

[error]     * com.github.jwt-scala:jwt-json-common_2.13:10.0.0 (early-semver) is selected over 7.1.5
[error]             +- com.github.jwt-scala:jwt-circe_2.13:10.0.0         (depends on 10.0.0)
[error]             +- org.apache.pekko:pekko-connectors-google-common_2.13:1.0.2 (depends on 7.1.5)

I managed to downgrade circe in my project. Thanks

[pjfanning]

Sbt has mechanisms to get it not to fail when there are versions mismatched. The easiest way is to add explicit dependencies with a selected version on the transitive dependencies that you have trouble with. You can also use dependencyOverrides.

[mdedetrich]

Overriding dependencies in this way may not be enough if the version of Circe is old enough since Circe doesn't guarantee binary compatibility in its releases (and this has caused a lot of pain in the Scala ecosystem).