Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add KamonRemoteInstrument #917

Merged
merged 12 commits into from
Jan 20, 2021

Conversation

SimunKaracic
Copy link
Contributor

changes

Added KamonRemoteInstrument, refactored part of BinaryPropagation (new code is in storeContextInBuffer, which I'll refactor, but wanted to get something up and running ASAP).

problems

As of this commit, context propagation looks ok, but there's a warning when deserializing a HandshakeReq.

I can't find what's causing the warning.
I have also tried removing the existing Remoting instrumentation, but nothing changed.

Pertinent logs (with DEBUG):

[2020-12-15 15:54:06,266] [DEBUG] [akka.remote.artery.Decoder] [] [ClusterSystem-akka.actor.default-dispatcher-5] - Decoded message but unable to record hits for compression as no remoteAddress known. No association yet?
[2020-12-15 15:54:06,266] [DEBUG] [akka.remote.artery.Decoder] [] [ClusterSystem-akka.actor.default-dispatcher-5] - Decoded message but unable to record hits for compression as no remoteAddress known. No association yet?
[2020-12-15 15:54:06,266] [DEBUG] [akka.remote.artery.Decoder] [] [ClusterSystem-akka.actor.default-dispatcher-5] - Decoded message but unable to record hits for compression as no remoteAddress known. No association yet?
[2020-12-15 15:54:06,269] [DEBUG] [akka.remote.artery.Decoder] [] [ClusterSystem-akka.actor.default-dispatcher-5] - Decoded message but unable to record hits for compression as no remoteAddress known. No association yet?
[2020-12-15 15:54:06,407] [DEBUG] [akka.remote.artery.RemoteInstruments] [] [ClusterSystem-akka.actor.default-dispatcher-5] - Skipping local RemoteInstrument 8 that has no matching data in the message
[2020-12-15 15:54:06,407] [DEBUG] [akka.remote.artery.RemoteInstruments] [] [ClusterSystem-akka.actor.default-dispatcher-5] - Skipping local RemoteInstrument 8 that has no matching data in the message
[2020-12-15 15:54:06,408] [DEBUG] [akka.remote.artery.RemoteInstruments] [] [ClusterSystem-akka.actor.default-dispatcher-5] - Skipping local RemoteInstrument 8 that has no matching data in the message
[2020-12-15 15:54:06,408] [DEBUG] [akka.remote.artery.RemoteInstruments] [] [ClusterSystem-akka.actor.default-dispatcher-5] - Skipping local RemoteInstrument 8 that has no matching data in the message
[2020-12-15 15:54:06,408] [WARN] [akka.remote.artery.Deserializer] [] [ClusterSystem-akka.actor.default-dispatcher-5] - Failed to deserialize message from [unknown] with serializer id [17] and manifest [d]. java.lang.IllegalArgumentException
[2020-12-15 15:54:06,408] [WARN] [akka.remote.artery.Deserializer] [] [ClusterSystem-akka.actor.default-dispatcher-5] - Failed to deserialize message from [unknown] with serializer id [17] and manifest [d]. java.lang.IllegalArgumentException
[2020-12-15 15:54:06,408] [WARN] [akka.remote.artery.Deserializer] [] [ClusterSystem-akka.actor.default-dispatcher-5] - Failed to deserialize message from [unknown] with serializer id [17] and manifest [d]. java.lang.IllegalArgumentException
[2020-12-15 15:54:06,418] [DEBUG] [akka.serialization.Serialization(akka://ClusterSystem)] [] [ClusterSystem-akka.actor.default-dispatcher-5] - Using serializer [akka.remote.serialization.ArteryMessageSerializer] for message [akka.remote.artery.OutboundHandshake$HandshakeRsp]

[2020-12-15 15:54:06,407] [DEBUG] [akka.remote.artery.RemoteInstruments] [] [ClusterSystem-akka.actor.default-dispatcher-5] - Skipping local RemoteInstrument 8 that has no matching data in the message is expected, since our RemoteInstrument won't put anything in the metadata when the context is empty.

This is the same warning that was being triggered when Kamon wasn't on all nodes in a cluster, and is now present when Kamon is running on both nodes :/

The warning only appears on the side that's receiving the message, and it shows up 4 times when a node tries to join the cluster. The nodes continue communicating without a problem (that I can see :/) after that.

The warning is not present in 2.1.9 when all nodes have Kamon. So it's something triggered by my changes, or their interaction with existing instrumentation :)

Not sure how to tackle this. Any ideas?

As of this commit, context propagation looks ok, but there are deserialization
errors happening on HandshakeReq
@SimunKaracic SimunKaracic force-pushed the akka-cluster-partial-kamon branch 2 times, most recently from 401585c to 4a5d681 Compare January 19, 2021 09:02
@danischroeter
Copy link
Contributor

nice! will this fix #767 ?

@SimunKaracic
Copy link
Contributor Author

SimunKaracic commented Jan 19, 2021

Yup, and it's gonna be released soon :)

P.S. also fixes #680 and #833

val `Akka-2.5-version` = "2.5.26"
val `Akka-2.6-version` = "2.6.0"
// latest is 2.6.10
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

latest is 2.6.11 ;)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

They released a new one! How dare they!

It is now in the akka module, since it must always be enabled
when that module is enabled.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants