Establish 24/7 engineering on-call

[lukeheath]

Goal

User story
As a Fleet customer,
I want to know that Fleet provides 24/7 engineering on-call,
so that I know that Fleet can respond to incidents quickly outside of business hours.

How?

• Sign up for incident.io.
• Document 24/7 on-call SLA and rotation.

[lukeheath]

1. We're announcing after hours on-call at next Eng Together net Wednesday.
2. We're implementing it 5pm on Monday, February 23.
3. Will need one person with a 1 hour SLA on after hours on-call and one person with a less aggressive (4-6 hours) SLA as backup that can review code changes for a hotfix or patch if necessary.

[lukeheath]

Victor: Since we need infrastructure online anyway (and they're already 24 hours) we should give them the ability to be reviewers for any hotfix.

[lukeheath]

I've put together a handbook update formalizing this change: #35314

[JordanMontgomery]

Was discussing separate concerns with @iansltx and he asked me to drop this into this this ticket:

There's been some talk about customer "Dossiers" or just some centralized place where we have, or will have, information about customers around things like:
What MDM are they migrating from(multiple choice ideally since they may have one for Windows, one for Mac, one for Mobile, etc)? Are they using ABM migration or Fleet's migration(assuming they're mid migration)
What IDP are they using?
Self-hosted or Cloud? If self-hosted, cloud provider and other info
Customer environment configuration such as allowlisting, known firewall configs, other management products potentially in place
Customer teams configuration if we know anything about it
Update policy
Are customer's users distributed or mainly in-office somewhere(thinking about ratelimits)?
What sort of non-standard things are they doing we should know about?

We could probably bring this up at an Eng Together type meeting to get a better list of questions

[lukeheath]

@JordanMontgomery Great point and yes we do need what I've been calling a customer pokedex. That information is out there, but it's not centralized and easy to find. Some of it is in Salesforce, some in Grafana, some in spreadsheets. I think a task for #g-eng-ops next sprint is making that start to happen.