Skip to content

Update handbook content related to oncall #5789

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
zwass merged 8 commits into from
May 19, 2022
Merged

Update handbook content related to oncall #5789

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
8 commits
Select commit Hold shift + click to select a range
75cdb55
Update handbook content related to oncall
zwass May 17, 2022
2422068
Create community.md
Desmi-Dizney May 18, 2022
1234cd2
Update customers.md
Desmi-Dizney May 18, 2022
1abd70a
Update community.md
Desmi-Dizney May 18, 2022
347ca84
Update customers.md
Desmi-Dizney May 18, 2022
e86dd7e
Update engineering.md
Desmi-Dizney May 18, 2022
90e7de8
Update engineering.md
Desmi-Dizney May 18, 2022
af3bb8e
Desmi comments
zwass May 19, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
74 changes: 74 additions & 0 deletions handbook/community.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,80 @@ Fleet's users and broader audience are spread across many online platforms. Her
- [reddit.com/r/SysAdminBlogs](https://www.reddit.com/r/SysAdminBlogs/)
- [r/sysadmin Discord](https://discord.gg/sysadmin)

### Goals
Our primary quality objectives are *customer service* and *defect reduction*. We try to optimize the following:

- Customer response time
- The number of bugs resolved per release cycle
- Staying abreast of what our community wants and the problems they're having
- Making people feel heard and understood
- Celebrating contributions
- Triaging bugs, identifying community feature requests, community pull requests, and community questions

### How?

- Folks who post a new comment in Slack or issue on GitHub should receive a response **within one business day**. The response doesn't need to include an immediate answer.
- If you feel confused by a question or comment raised, [request more details](#requesting-more-details) to better your understanding of the next steps.
- If an appropriate response is outside of your scope, please post to `#help-engineering` (in the Fleet Slack)), tagging `@oncall`.
- If things get heated, remember to stay [positive and helpful](https://canny.io/blog/moderating-user-comments-diplomatically/). If you aren't sure how best to respond positively, or if you see behavior that violates the Fleet code of conduct, get help.

### Requesting more details

Typically, the *questions*, *bug reports*, and *feature requests* raised by community members will be missing helpful context, recreation steps, or motivations.

❓ For questions that you don't immediately know the answer to, it's helpful to ask follow-up questions to receive additional context.

- Let's say a community member asks the question "How do I do X in Fleet?" A follow-up question could be "What are you attempting to accomplish by doing X?"
- This way, you have additional details when you bring this to the Roundup meeting. In addition, the community member receives a response and feels heard.

🦟 For bug reports, it's helpful to ask for re-creation steps so you're later able to verify the bug exists.

- Let's say a community member submits a bug report. An example follow-up question could be "Can you please walk me through how you encountered this issue so that I can attempt to recreate it?"
- This way, you now have steps that verify whether the bug exists in Fleet or if the issue is specific to the community member's environment. If the latter, you now have additional information for further investigation and question-asking.

💡 For feature requests, it's helpful to ask follow-up questions in an attempt to understand the "Why?" or underlying motivation behind the request.

- Let's say a community member submits the feature request "I want the ability to do X in Fleet." A follow-up question could be "If you were able to do X in Fleet, what's the next action you would take?" or "Why do you want to do X in Fleet?."
- Both of these questions provide helpful context on the underlying motivation behind the feature request when brought to the Roundup meeting. In addition, the community member receives a response and feels heard.

### Closing issues

It is often a good idea to let the original poster (OP) close their issue themselves since they are usually well equipped to decide to mark the issue as resolved. In some cases, circling back with the OP can be impractical, and for the sake of speed, issues might get closed.

Keep in mind that this can feel jarring to the OP. The effect is worse if issues are closed automatically by a bot (See [balderashy/sails#3423](https://github.com/balderdashy/sails/issues/3423#issuecomment-169751072) and [balderdashy/sails#4057](https://github.com/balderdashy/sails/issues/4057) for examples of this).

### Version support

To provide the most accurate and efficient support, Fleet will only target fixes based on the latest released version. In the current version fixes, Fleet will not backport to older releases.

Community version supported for bug fixes: **Latest version only**

Community support for support/troubleshooting: **Current major version**

Premium version supported for bug fixes: **Latest version only**

Premium support for support/troubleshooting: **All versions**

### Tools

Ther script located in `scripts/oncall` for use during oncall rotation (only been tested on macOS and Linux).
Its use is completely optional but contains several useful commands for checking issues and PRs that may require attention.
You will need to install the following tools in order to use it:

- [GitHub CLI](https://cli.github.com/manual/installation)
- [JQ](https://stedolan.github.io/jq/download/)

### Resources

There are several locations in Fleet's public and internal documentation that can be helpful when answering questions raised by the community:

1. Find the frequently asked question (FAQ) documents in each section in the `/docs` folder. These documents are the [Using Fleet FAQ](../docs/Using-Fleet/FAQ.md), [Deploying FAQ](../docs/Deploying/FAQ.md), and [Contributing FAQ](../docs/Contributing/FAQ.md).

2. Use the [internal FAQ](https://docs.google.com/document/d/1I6pJ3vz0EE-qE13VmpE2G3gd5zA1m3bb_u8Q2G3Gmp0/edit#heading=h.ltavvjy511qv) document.

### Assistance from engineering

Community team members can reach the engineering oncall for assistance by writing a message with `@oncall` in the `#help-engineering` channel of the Fleet Slack.

## Fleet docs

Expand Down
3 changes: 3 additions & 0 deletions handbook/customers.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -69,6 +69,9 @@ When creating a new issue, ensure the following:
- Is there a key date or timeframe that the customer hopes to meet? If so, please post about that in #g-product with a link to the issue, so the team can discuss it before committing to a time frame.
- Have we provided a link to that issue for the customer to remind everyone of the plan and for the sake of visibility, so other folks who weren't directly involved are up to speed (e.g., "Hi everyone, here's a link to the issue we discussed on today's call: […link…](https://omfgdogs.com)")?

### Assistance from engineering

Customer team members can reach the engineering oncall for assistance by writing a message with `@oncall` in the `#help-engineering` channel of the Fleet Slack.
## Runbook

### Responding to a request to change a credit card number
Expand Down
121 changes: 14 additions & 107 deletions handbook/engineering.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -126,118 +126,27 @@ Non-release blocking bugs may include known issues that were not targeted for th
Documentation on completing the release process can be found
[here](../docs/Contributing/Releasing-Fleet.md).

## On-call rotation
## Oncall rotation

This section outlines the on-call rotation at Fleet.
The oncall engineer is a second-line responder to questions raised by customers and community members. The Community team is responsible for the first response to GitHub issues, pull requests, and Slack messages in the osquery and other public Slacks. The Customer team is responsible for the first response to messages in private customer Slack channels.

The on-call engineer is responsible for responding to technical Slack comments, Slack threads, and GitHub issues raised by customers and the community which cannot be handled by the [Customer Success team](./customers.md).

### Goals
Our primary quality objectives are *customer service* and *defect reduction*. We use the following Key Performance Indicators (KPIs) to measure our success with these goals:

- Customer response time.
- The number of bugs resolved per release cycle.
- Stay abreast of what our community wants and the problems they're having.
- Make people feel heard and understood.
- Celebrate contributions.
- Triage bugs, identify community feature requests, community pull requests, and community questions.

### How?

- Folks who post a new comment in Slack or issue on GitHub **must** receive a response from the on-call engineer **within 1 business day**. The response doesn't need to include an immediate answer.
- The on-call engineer can discuss any items that require assistance at the end of the daily standup. They are also requested to attend the "Customer experience standup" where they can bring questions and stay abreast of what's happening with our customers.
- If you do not understand a question or comment raised, [request more details](#requesting-more-details) to best understand the next steps.
- If an appropriate response is outside your scope, please post to `#help-oncall`, a confidential Slack channel in the Fleet Slack workspace.

- If things get heated, remember to stay [positive and helpful](https://canny.io/blog/moderating-user-comments-diplomatically/). If you aren't sure how best to respond in a positive way, or if you see behavior that violates the Fleet code of conduct, get help.

### Requesting more details

Typically, the *questions*, *bug reports*, and *feature requests* raised by members of the community will be missing helpful context, recreation steps, or motivations respectively.

❓ For questions that you don't immediately know the answer to, it's helpful to ask follow-up questions to receive additional context.

- Let's say a community member asks the question "How do I do X in Fleet?" A follow-up question could be "What are you attempting to accomplish by doing X?"
- This way, you have additional details when the primary question is brought to the Roundup meeting. In addition, the community member receives a response and feels heard.

🦟 For bug reports, it's helpful to ask for recreation steps so you're later able to verify the bug exists.

- Let's say a community member submits a bug report. An example follow-up question could be "Can you please walk me through how you encountered this issue so that I can attempt to recreate it?"
- This way, you now have steps that verify whether the bug exists in Fleet or if the issue is specific to the community member's environment. If the latter, you now have additional information for further investigation and question-asking.

💡 For feature requests, it's helpful to ask follow-up questions in an attempt to understand the "Why?" or underlying motivation behind the request.

- Let's say a community member submits the feature request "I want the ability to do X in Fleet." A follow-up question could be "If you were able to do X in Fleet, what's the next action you would take?" or "Why do you want to do X in Fleet?."
- Both of these questions provide helpful context on the underlying motivation behind the feature request when it is brought to the Roundup meeting. In addition, the community member receives a response and feels heard.

### Feature requests

If the feature is requested by a customer, the on-call engineer is requested to create a feature request issue and follow up with the customer by linking them to this issue. This way, the customer can add additional comments or feedback to the newly filed feature request issue.

If the feature is requested by anyone other than a customer (ex. user in #fleet Slack), the on-call engineer is requested to point the user to the [feature request GitHub issue template](https://github.com/fleetdm/fleet/issues/new?assignees=&labels=idea&template=feature-request.md&title=) and kindly ask the user to create a feature request.

### Closing issues

It is often a good idea to let the original poster (OP) close their issue themselves since they are usually well equipped to decide whether the issue is resolved. In some cases, circling back with the OP can be impractical, and for the sake of speed, issues might get closed.

Keep in mind that this can feel jarring to the OP. The effect is worse if issues are closed automatically by a bot (See [balderashy/sails#3423](https://github.com/balderdashy/sails/issues/3423#issuecomment-169751072) and [balderdashy/sails#4057](https://github.com/balderdashy/sails/issues/4057) for examples of this).

### Version support

In order to provide the most accurate and efficient support, Fleet will only target fixes based on the latest released version. Fixes in current versions will not be backported to older releases.

Community version supported for bug fixes: **Latest version only**

Community support for support/troubleshooting: **Current major version**

Premium version supported for bug fixes: **Latest version only**

Premium support for support/troubleshooting: **All versions**

### Sources

There are four sources that the on-call engineer should monitor for activity:

1. Customer Slack channels - Found under the "Connections" section in Slack. These channels are usually titled "at-insert-customer-name-here."

2. Community chatroom - https://osquery.slack.com, #fleet channel

3. Reported bugs - [GitHub issues with the "bug" and ":reproduce" label](https://github.com/fleetdm/fleet/issues?q=is%3Aissue+is%3Aopen+label%3Abug+label%3A%3Areproduce). Please remove the ":reproduce" label after you've followed up in the issue.

4. Pull requests opened by the community - [GitHub open pull requests](https://github.com/fleetdm/fleet/pulls?q=is%3Apr+is%3Aopen)

### Tools

There is a script located in `scripts/on-call` for use during on-call rotation (only been tested on macOS and Linux).
Its use is completely optional but contains several useful commands for checking issues and PRs that may require attention.
You will need to install the following tools in order to use it:

- [Github CLI](https://cli.github.com/manual/installation)
- [jq](https://stedolan.github.io/jq/download/)

### Resources

There are several locations in Fleet's public and internal documentation that can be helpful when answering questions raised by the community:

1. The frequently asked question (FAQ) documents in each section are found in the `/docs` folder. These documents are the [Using Fleet FAQ](../docs/Using-Fleet/FAQ.md), [Deploying FAQ](../docs/Deploying/FAQ.md), and [Contributing FAQ](../docs/Contributing/FAQ.md).

2. The [Internal FAQ](https://docs.google.com/document/d/1I6pJ3vz0EE-qE13VmpE2G3gd5zA1m3bb_u8Q2G3Gmp0/edit#heading=h.ltavvjy511qv) document.
Oncall engineers do not need to actively monitor Slack channels, except when called in by the Community or Customer teams. Members of those teams are instructed to `@oncall` in `#help-engineer` to get the attention of the oncall engineer to further discuss any issues that come up. In some cases, the Community or Customer representative will continue to communicate with the requestor, and in others, the oncall engineer will communicate directly.

### Handoff

Every week, the on-call engineer changes. Here are some tips for making this handoff go smoothly:
Every week, the oncall engineer changes. Here are some tips for making this handoff go smoothly:

1. The new on-call engineer should change the `@oncall` alias in Slack to point to them. In the
1. The new oncall engineer should change the `@oncall` alias in Slack to point to them. In the
search box, type "people" and select "People & user groups". Switch to the "User groups" tab.
Click `@oncall`. In the right sidebar, click "Edit Members". Remove the former on-call, and add
Click `@oncall`. In the right sidebar, click "Edit Members". Remove the former oncall, and add
yourself.

2. Handoff newer conversations. For newer threads, the former on-call can unsubscribe from the
thread, and the new on-call should subscribe. The former on-call should explicitly share each of
2. Hand off newer conversations. For newer threads, the former -call can unsubscribe from the
thread, and the new oncall should subscribe. The former oncall should explicitly share each of
these threads, and the new on-call can select "Get notified about new replies" in the "..." menu.
The former on-call can select "Turn off notifications for replies" in that same menu. It can be
helpful for the former on-call to remain available for any conversations they were deeply involved
in, so use your judgment on which threads to handoff.
The former oncall can select "Turn off notifications for replies" in that same menu. It can be
helpful for the former oncall to remain available for any conversations they were deeply involved
in, so use your judgment on which threads to hand off.

## Project boards

Expand All @@ -251,13 +160,11 @@ The following rituals are engaged in by the directly responsible individual (DR

| Ritual | Frequency | Description | DRI |
|:-----------------------------|:-----------------------------|:----------------------------------------------------|-------------------|
| Stand up | Daily | Discuss items being worked during the current iteration and any blockers. | Zach Wasserman |
| Pull request review | Daily | Engineers go through pull requests on which their review has been requested. | Zach Wasserman |
| Engineering group discussions | Weekly | Engineering groups meet to discuss issues in depth that are too big or complex to discuss adequately during a stand up. | Zach Wasserman |
| On-call handoff | Weekly | Handoff the on-call engineering responsibilities to the next on-call engineer. | Zach Wasserman |
| Pull request review | Daily | Engineers go through pull requests for which their review has been requested. | Zach Wasserman |
| Engineering group discussions | Weekly | See "Group Weeklies". | Zach Wasserman |
| On-call handoff | Weekly | Hand off the on-call engineering responsibilities to the next on-call engineer. | Zach Wasserman |
| Release ritual | Every three weeks | Go through the process of releasing the next iteration of Fleet. | Zach Wasserman |


## Slack channels

The following [Slack channels are maintained](https://fleetdm.com/handbook/company#group-slack-channels) by this group:
Expand Down