Update to Java 17 and Kryo 5

[hannesj]

Summary

Update to Java 17. I needed to update a couple of libraries, in order not to have compering files of some of the dependencies.

• Added UnmodifiableCollectionsSerializer from external source Compatibility with JDK 17 magro/kryo-serializers#131 (comment)
• Added JavaImmutableMapSerializer, as the Kryo built-in serializer sometimes returned the wrong type.

Changelog

The changelog file
is generated from the pull-request title, make sure the title describe the feature or issue fixed.
To exclude the PR from the changelog add [changelog skip] in the title.

[leonardehrenfried]

Can these be remove once we move to Kryo 5?

[hannesj]

It can be removed when issue magro/kryo-serializers#131 is fixed. The PR contains an update to Kryo 5

[t2gran]

This looks great. I only had one question, se above.

[leonardehrenfried]

Doesn't Kryo 5 have built-in support for List.of()? https://github.com/EsotericSoftware/kryo/blob/master/src/com/esotericsoftware/kryo/serializers/ImmutableCollectionsSerializers.java

Or are they broken in the same was as the map one?

[hannesj]

Yes, they will return a wrong type when serializing an empty Set/List/Map, thus failing the tests which compare the loaded graph. I'll create an issue in kryo about that

[hannesj]

https://github.com/EsotericSoftware/kryo/blob/master/src/com/esotericsoftware/kryo/serializers/ImmutableCollectionsSerializers.java#L51-L53

and

https://github.com/EsotericSoftware/kryo/blob/master/src/com/esotericsoftware/kryo/serializers/CollectionSerializer.java#L216-L219

[leonardehrenfried]

Ok, thanks for the explanation. Can you cc me when you create the ticket?

[abyrd]

Is this intended to be merged into dev-2.x before or after the release? I ask because there's already a 2.1-rc branch and this PR seems like a major change to introduce immediately before a release. Actually I may be misinterpreting the branching strategy because it's different than what we've used in the past. It looks like development is continuing on dev-2.x and changes intended for the release are being cherry-picked over to 2.1-rc.

[hannesj]

This is intended for 2.2, so after the release has been cut

[t2gran]

@abyrd I already updated the version number in dev-2.x to 2.2.0-SNAPSHOT. There is a stream of PRs constantly beeing merged and I did not want everyone to change their habit of merging into dev-2.x, so we branched of 2.1-rc at the point we thought was the best place, and then we have cherry-picked a few PRs/commits from dev-2.x after that. I plan to merge the master branch back into dev-2.x when the release is done, so we get all change (also the one done locally in 2.1-rc) in the dev-2.1. It is a bit messy - for me - but cleaner for every one else.

[theigl]

Hi @hannesj. I'm the maintainer of Kryo and just stumbled across this issue. Are you sure that the immutable collections serializers really return the wrong type at times? I just wrote a a test-case that asserts on the result type of the collection and it passes for all sizes.

[leonardehrenfried]

Hi @theigl, Hannes isn't contributing a lot anymore.

However, I can try removing our custom serializers and run the test suite again. Maybe these things have been fixed in the meantime.

Thanks for chiming in!

[theigl]

@leonardehrenfried: Thanks, that would be great. I'd really like to know if there is some hidden bug in those serializers that I'm currently not aware of.

[leonardehrenfried]

So I just tried it by removing our custom serializers and using the built-in Kyro ones: leonardehrenfried@591a54d

(Is this setup correct?)

Unfortunately, after this change our test reports that we expect at SetN but we receive a HashSet instead.

Classes are not the same: SetN vs HashSet
Comparison stack (outermost first):
    class org.opentripplanner.routing.graph.Graph
    private final java.util.Map org.opentripplanner.routing.graph.Graph.vertices
    class java.util.concurrent.ConcurrentHashMap
    EN:NSR:Quay:122003
    class org.opentripplanner.street.model.vertex.TransitStopVertex
    private final org.opentripplanner.transit.model.site.RegularStop org.opentripplanner.street.model.vertex.TransitStopVertex.stop
    class org.opentripplanner.transit.model.site.RegularStop
    private final java.util.Set org.opentripplanner.transit.model.site.RegularStop.fareZones

Something is amiss here. It could very well be that our setup is wrong.

[leonardehrenfried]

I wrote some more tests and I can confirm that with plain Kryo the type is correctly deserialized to SetN not HashMap. Presumably something is wrong with our setup.

Can you spot an error in our KryoBuilder? https://github.com/opentripplanner/OpenTripPlanner/blob/dev-2.x/src/main/java/org/opentripplanner/routing/graph/kryosupport/KryoBuilder.java

[theigl]

Everything looks alright in your builder. I'll try to get to the bottom of this next week. Your branch should be enough for me to write a reproducer.

Thanks!

[leonardehrenfried]

Great. If you want to run the test yourself it's called GraphSerializationTest.