Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Track follow ups #89

Closed
12 of 16 tasks
Tracked by #64
, #57 ,  #109
...
hubertdeng123 opened this issue Jan 18, 2023 · 17 comments
Closed
12 of 16 tasks
Tracked by #64
, #57 ,  #109
...

Track follow ups #89

hubertdeng123 opened this issue Jan 18, 2023 · 17 comments
Labels
Type: Epic

Comments

@hubertdeng123
Copy link
Member

hubertdeng123 commented Jan 18, 2023 •
edited by chadwhitacre
Loading

Are customers happy with our response times?

poll-results Screen Shot 2023-01-18 at 12 15 16 PM

Well, yes and no. In Q4 2022, we've introduced slack notifications for routing/triage. We've seen how effective this can be, as our routing and triage numbers have gone up. These are our recent numbers
Screenshot 2023-04-24 at 9 01 05 AM
We've reached all time highs for quarterly numbers, with routing at 97.8% and triaging at 87.8%.

Screenshot 2023-04-19 at 3 44 56 PM Screenshot 2023-04-19 at 3 45 00 PM

However, here at the OSPO we've noticed two main issues:

  1. First off, triage numbers haven't risen as much as we've wanted them to. However, we hope that our changes this quarter in overhauling the routing process to use product owners instead of GitHub teams will boost these numbers up. This is currently ongoing, and we should observe the results of this project starting next quarter.
  2. Secondly, nothing is telling us how to respond to follow ups after an issue has been triaged. It's not uncommon for issues to go on forgotten after an initial response is received (example). So perhaps the system is not quite accurate in determining response times in the form of TTR and TTT.

How are we analyzing the problem?

Currently, we have two goals: drive up customer engagement numbers and provide a standard way to handle responses between GitHub users and Sentry product owners. Routing and triaging as it currently exists can be a bit confusing.

  • Only the support team performs routing, while triaging is done by the product owners.
  • Routing is only done in the repos sentry and sentry-docs, while triaging is done in every repo owned by Sentry. Issues that are cross repos are hard to track for teams.
  • The only Status labels that are super relevant to us are the Status: Untriaged and Status: Unrouted labels.
  • Nothing is telling us how to respond to follow ups after an issue has been triaged. It's not uncommon for issues to go on forgotten after an initial response is received (example).

How can we fix this?

In Q1 2023, our team has implemented a new process for maintaining EPD team changes and product mappings. Most importantly, each issue that comes in corresponds to a product area (Replays, Crons, Issues, etc). A product owner is then responsible for issues that fall under the product area that they own. The initial automations to replace team labels with product owners have been completed. We want to expand on this automation to allow for product owners to easily determine which issues need actionable items.

Right now, we have two Status labels that are primarily used for notifications Status: Unrouted and Status: Untriaged. Many of the other Status: * labels are unused, and are not used by automation for any purposes. What we really care about is whether or not someone at Sentry should respond to an issue, or if an issue is waiting for a response from the community. We can consolidate these Status: * labels into something simpler and more useful for automation purposes.

These are the new status labels:

Waiting for: Support = Support team at Sentry
Waiting for: Community = everyone else
Waiting for: Product Owner = Sentry members who own parts of the product

Waiting for: Support

This status is applied to all issues that are new or need to be rerouted. It will also be applied to issues that Support should be responsible for. Support serves as a overseer for which product areas an issue belongs to. The due date column will be occupied for a route by date. For automation purposes, there are two cases to handle

  1. New issue comes in
  2. Product Owner manually applies the Waiting for: Support status to an issue

Waiting for: Community

This status will be applied to issues that have been responded to by product owners. Product owners will manually add the Waiting for: Community label once they deem that they need more information from the creator of the issue. We will only apply stalebot to issues that are waiting for community.

Waiting for: Product Owner

This status will be applied to issues that have last been interacted with by a community member. If a community member comments to an issue, the status of the issue will become this.

We want to use these statuses instead of labels as a source of truth to send slack notifications. Furthermore, instead of tracking the success of two different processes (routing and triage), we can instead track the success of overall response time.

Rough Task List

Bonus

Won't do

  • For SDK repo, automatically apply a corresponding product area field (sentry-javascript -> SDKS - Web Frontend)
  • Community member comments, then deletes the comment. What do we do?
This was referenced Jan 18, 2023
Closed
Closed
@chadwhitacre
Copy link
Member

Something to consider here: in #86 we're realizing that GTM folks should be classed as "community" (basically) even though they're on staff.

@hubertdeng123 hubertdeng123 mentioned this issue Jan 18, 2023
Closed
36 tasks
@hubertdeng123 hubertdeng123 changed the title Provide a more simple and concise way to respond to GitHub Reach our final time to response targets consistently across teams Jan 18, 2023
@hubertdeng123 hubertdeng123 changed the title Reach our final time to response targets consistently across teams Reach our final time to respond targets consistently across teams Jan 18, 2023
@chadwhitacre chadwhitacre mentioned this issue Jan 25, 2023
Closed
14 tasks
@chadwhitacre chadwhitacre mentioned this issue Feb 15, 2023
Closed
@chadwhitacre chadwhitacre mentioned this issue Mar 1, 2023
Open
@chadwhitacre
Copy link
Member

From @HazAT:

Is it possible that we get some automation/label that gets added when the last response on an issue is not from a Sentry employee?
And in contrast, automatically have this label removed when we made the last reply?
The reason being in a high-traffic repo - it's sometimes challenging to keep track of a lot of issues in parallel, and while manually keeping track of it works most of the time, sometimes things slip through the cracks

@chadwhitacre chadwhitacre mentioned this issue Apr 13, 2023
Closed
21 tasks
@hubertdeng123 hubertdeng123 changed the title Reach our final time to respond targets consistently across teams Improve customer engagement by highlighting actionable issues for product owners Apr 20, 2023
@hubertdeng123 hubertdeng123 changed the title Improve customer engagement by highlighting actionable issues for product owners Track follow ups Apr 24, 2023
@chadwhitacre chadwhitacre mentioned this issue May 2, 2023
Merged
@hubertdeng123
Copy link
Member Author

hubertdeng123 commented May 8, 2023 •
edited
Loading

Weekly Update May 8

@hubertdeng123
Copy link
Member Author

hubertdeng123 commented May 15, 2023 •
edited
Loading

Weekly Update May 15

@hubertdeng123
Copy link
Member Author

hubertdeng123 commented May 22, 2023 •
edited
Loading

Weekly Update May 22

@hubertdeng123
Copy link
Member Author

hubertdeng123 commented May 30, 2023 •
edited
Loading

Weekly Update May 30

@hubertdeng123
Copy link
Member Author

hubertdeng123 commented Jun 6, 2023 •
edited
Loading

Weekly Update June 5

@hubertdeng123
Copy link
Member Author

hubertdeng123 commented Jun 13, 2023 •
edited
Loading

Weekly Update June 12

PR's up, not merged yet though

@hubertdeng123
Copy link
Member Author

hubertdeng123 commented Jun 20, 2023 •
edited
Loading

Weekly Update June 20

@hubertdeng123
Copy link
Member Author

hubertdeng123 commented Jun 26, 2023 •
edited
Loading

Weekly Update June 26

@hubertdeng123
Copy link
Member Author

hubertdeng123 commented Jul 5, 2023 •
edited
Loading

Weekly Update Jul 5

Short week!

@hubertdeng123 hubertdeng123 mentioned this issue Jul 7, 2023
Closed
1 task
@hubertdeng123
Copy link
Member Author

hubertdeng123 commented Jul 10, 2023 •
edited
Loading

Weekly Update Jul 10

@hubertdeng123
Copy link
Member Author

hubertdeng123 commented Jul 18, 2023 •
edited
Loading

Weekly Update Jul 17

@chadwhitacre
Copy link
Member

We're ready to close this out, yes?

@hubertdeng123
Copy link
Member Author

Tasks are done, just looking to write a shipped post before closing

@hubertdeng123
Copy link
Member Author

hubertdeng123 commented Jul 24, 2023 •
edited
Loading

Weekly Update Jul 24

@hubertdeng123
Copy link
Member Author

shipped post out! closing this, other items will be bonus

https://vanguard.getsentry.net/p/clkk9bcxb004ms60lyapf0vnm

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
Type: Epic
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@chadwhitacre @hubertdeng123