Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix(actuator): add coroutine id to help debug #985

Merged
merged 1 commit into from
Apr 8, 2020

Conversation

lorin
Copy link
Contributor

@lorin lorin commented Apr 8, 2020

Create an ad-hoc coroutine id to help gather related log messages. This is a temporary change to make it easier to go through the log messages for debugging #951.

Context: I'm seeing different thread names that seem to be associated with the same coroutine. Since there's no guarantee of 1:1 mapping between threads and coroutines, I don't know if it's the same coroutine or a different one, this change will make it easier to figure that out.

val id = resource.id
val plugin = handlers.supporting(resource.kind)

if (actuationPauser.isPaused(resource)) {
log.debug("Actuation for resource {} is paused, skipping checks", id)
log.debug("Actuation for resource {} is paused, skipping checks [$coid]", id)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One thought - if you want to group all the messages that have a coid in them together, you might want each log message to be [coid: $coid] so it's easy to search for all of them. If you don't want to do that, then ignore me.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

My plan is to find the "Resource check for $id failed [$coid]" log message (which always appears when instances go bad), and then just search for the coid in that message to get the remaining relevant messages.

@luispollo
Copy link
Contributor

I think you can just get the coroutine IDs automatically added to the thread names by turning on -Dkotlinx.coroutines.debug in the JVM:
https://kotlinlang.org/docs/reference/coroutines/coroutine-context-and-dispatchers.html#debugging-coroutines-and-threads

Also, just to make sure you're aware, the behavior of coroutines sharing threads can be changed, if we feel like that's an approach worth pursuing as a fix: https://kotlinlang.org/docs/reference/coroutines/coroutine-context-and-dispatchers.html#dispatchers-and-threads

@lorin
Copy link
Contributor Author

lorin commented Apr 8, 2020

@luispollo Thanks, I'm going to enable coroutine ids. However, even with the ability to turn on coroutine ids, I'd like to have unique ids for each invocation in the log to help me debug this. I may play with the coroutine / thread behavior settings at some point, but I want to collect a little more info first.

Create an ad-hoc coroutine id to help make gather related log messages.
This is a temporary change to make it easier to go through the log
messages for debugging spinnaker#951.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants