Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Implement Error Bingo mapping for nextsteps generation #193

Merged
merged 4 commits into from
Aug 24, 2023
Merged

Conversation

jon-funk
Copy link
Contributor

@jon-funk jon-funk commented Aug 23, 2023

  • Adds a troubleshooting lookup keyword that can "suggest" nextsteps based on a error bingo board, called RW.NextSteps.Suggest
  • Adds RW.NextSteps.Format keyword to allow for quick injection of live object values/names into nextsteps message before issue submission to reduce robot bloat
  • Board is based on a maintained static yaml mapping file, specific to each system/platform (eg: 1 for Kubernetes, 1 for ECS, 1 for Google Cloudrun, etc)
  • Adds RW.CLI.Run Bash File keyword

See https://github.com/runwhen-contrib/rw-cli-codecollection/blob/err-bingo/codebundles/cmd-test/runbook.robot for samples

@jon-funk
Copy link
Contributor Author

Example walkthrough of troubleshooting a postgres issue based on API logs:

  • using kubectl get logs in a previous line, a author could get OperationalError: FATAL: connection limit exceeded from their API logs
  • They then provide this to the Suggest keyword like so:
    ${next_steps}=    RW.NextSteps.Suggest    OperationalError: FATAL: connection limit exceeded for non-superusers

The produce next step will be organized to the following string:

Get Standard Postgres Resource Information, Describe Postgres Custom Resources, Get Postgres Pod Logs & Events, Get Postgres Pod Resource Utilization, Get Running Postgres Configuration, Get Patroni Output, Run DB Debug Queries


StatefulSet:$postgres_name
  • Either referencing codebundle configuration or pulling the object name with a CLI command, we can inject the database name into the above message like so:
    ${next_steps}=    RW.NextSteps.Format    ${next_steps}
    ...    postgres_name=${db_name.stdout}

The ${next_steps} string now adheres to the format requested by the platform team and is ready for submission in an issue.

This should:

  • reduce robot code bloat and repeated string format calls
  • leaves the next_steps formatting up to a keyword which we can update to match the platform's needs if needed in the future
  • allows platform troubleshooting experts to update mappings over time while other authors simply pull from the mapping with error messages
  • multiple mappings can be intertwined so troubleshooting next_steps can be generated for up the stack or down the stack relative to the current error being investigated

@jon-funk jon-funk marked this pull request as ready for review August 23, 2023 20:12
@jon-funk jon-funk requested a review from a team as a code owner August 23, 2023 20:12
Copy link
Contributor

@stewartshea stewartshea left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Love it, excited to try it and see how it fits into my current flow.

@stewartshea stewartshea merged commit ac7f852 into main Aug 24, 2023
1 check passed
@stewartshea stewartshea deleted the err-bingo branch August 24, 2023 13:20
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants