Renew SCEP certificate for hosts w/ old (non-Fleet) enrollment profile

[roperzh]

Goal

User story
As an organization that automatically migrated my workstations (#19387) from my old MDM solution to Fleet,
I want to renew the SCEP certificates on my hosts
so that I know MDM features (commands, configuration profiles, etc.) will work for these hosts.

Context

• Requestor(s): @roperzh
• Product designer: @noahtalerman

To renew SCEP certificates, we send an InstallProfile command with Fleet's enrollment profile to the devices.

Hosts that migrated using "Process for self-hosted macOS MDM migration to Fleet" (#19387), will have a different enrollment profile (one from the old MDM solution), so the InstallProfile command will fail and the SCEP certificate won't be renewed.

Changes

Product

• Fleet server changes: Add a job that checks for hosts' with SCEP certs that are due to expire in the next 30 days (same interval as existing job for sending a new Fleet enrollment profile). Delivers a new enrollment profile to these hosts. A new enrollment profile will tell the host to get a new SCEP cert
  • The enrollment profile (XML) will be a hidden (documented in config for contributors) environment variable.
• Research:
  • Confirm we can points hosts to Fleet's built-in certificate authority to issue the new SCEP certificates. Simplest/preferred approach is to update SCEP URL in the enrollment profile to point to Fleet
  • ✅ Get the customer's current enrollment profile
• Migration script: Get SCEP expiration date for each device
• REST API changes: No API changes
• Outdated documentation changes: Document the hidden config in the configuration for contributors docs

Engineering

• Database schema migrations: TODO
• Load testing: TODO

ℹ️  Please read this issue carefully and understand it. Pay special attention to UI wireframes, especially "dev notes".

QA

Risk assessment

• Requires load testing: TODO
• Risk level: Low / High TODO
• Risk description: TODO

Manual testing steps

1. Step 1
2. Step 2
3. Step 3

Testing notes

Confirmation

1. Engineer (@____): Added comment to user story confirming successful completion of QA.
2. QA (@____): Added comment to user story confirming successful completion of QA.

[dherder]

@roperzh do we know what the SCEP certificate lifespan is for the customer devices? I do know that some MDM systems will set this to a long lived value like 2099, so in those cases it would not be an issue. If the lifespan of the certificate is short lived, I would say that this would be a P2 blocker issue.

[roperzh]

@dherder good point! we should check with them, I know that micromdm/scep uses 1 year by default (-crtvalid flag) so unless they provided a custom value there, it's 1 year

[zayhanlon]

@lukeheath @noahtalerman per the process, letting you know that we have this as workflow/migration blocking and added the p2 label. let me know if anything else needs to be done to escalate

[lukeheath]

@zayhanlon P2 makes sense to me. Our response for P2 is:

Response: Issue is prioritized at the top of the next sprint. If opporunity cost of waiting for the next sprint is too high, it may be considered for current sprint.

We'll prioritize this for next sprint, which is scheduled to ship 7/15. Is that soon enough?

@noahtalerman @georgekarrv

[zayhanlon]

@lukeheath @georgekarrv @noahtalerman - there's a thread going in #g-customer-success https://fleetdm.slack.com/archives/C062D0THVV1/p1718733547340419?thread_ts=1718384351.332159&cid=C062D0THVV1

This new issue was surfaced by Roberto this week but is also migration blocking. I don't think 7/15 will work - any way to get it faster or patched sooner? 
 

@zwass @dherder FYI

[roperzh]

I made it a story so it gets product feedback is that I personally only see three ways to accomplish this:

1. We change how certificate renewals work to account for hosts with custom enrollment profiles
2. We build some product feature that allows them/us to build a flow to renew certificates (eg: a webhook, a config in the UI)
3. We build a script that issues cert renewals that lives outside Fleet

[noahtalerman]

Thanks @roperzh!

I threw some time on your calendar to dig into the options.

[roperzh]

we met with @noahtalerman and decided to do option 3 as a fist baby step:

We build a script that issues cert renewals that lives outside Fleet

I think this requires 3 action items:

1. Investigate if we can use Fleet's built-in CA to issue the new SCEP certificates (@roperzh)
2. Get required information from the customer:
   a. Expiration date for each device. This could be part of the export script (cc: @zwass)
   b. Current enrollment profile (cc: @zwass @dherder)
3. Define where this service will be hosted, could probably live alongside the proxy? (cc: @zwass @dherder)

cc: @zayhanlon

[noahtalerman]

Thanks @roperzh!

Define where this service will be hosted, could probably live alongside the proxy?

I think we decided to go with fleetdm.com instead of standing up a separate service. Why? So we can reduce surface area and understandability for Fleet contributors.

If this doesn't work please let me know.

I think this means that the enrollment profile (XML) will live as an environment variable in Heroku. We'll probably need @eashaw's help to add that variable.

I updated the issue description to reflect this.

[zwass]

I have several questions:

1. Who at Fleet will write the code? Who will maintain the code?
2. Where will the server be hosted? Who will be responsible for maintaining it? Alongside the proxy does not sound like a good option as we are currently doing that in the solutions consulting AWS and the server is intended to only live for a couple weeks while the migration is completed (see Process for self-hosted macOS MDM migration to Fleet #19387). This server seems like it needs to run indefinitely (until we make a more long-term feature for this?)
3. How will the scripts be triggered? Is this something that the server becomes responsible for?

[zwass]

I think we decided to go with fleetdm.com instead of standing up a separate service. Why? So we can reduce surface area and understandability for Fleet contributors.

Does this mean we would be putting customer SCEP cert/keys into fleetdm.com? That sounds pretty risky to me as I'm not aware that fleetdm.com has been designed/audited for storage of customer data (let alone important customer secrets).

Or maybe we are just talking about using fleetdm.com to trigger script execution for the hosts that are expiring? That seems potentially less risky but still something that would need to be well-understood. Would it require API keys for customer Fleet servers?

[noahtalerman]

Does this mean we would be putting customer SCEP cert/keys into fleetdm.com?

@zwass I don't think so. The enrollment profile would be an environment variable in Heroku. Once the enrollment profile is delivered the host will get the new SCEP cert from the Fleet server

Would it require API keys for customer Fleet servers?

I think so yes. We need the API key to deliver the enrollment profile via the Fleet API. This can be stored an as environment variable in Heroku.

@roperzh please correct me if I'm wrong.

[roperzh]

Who at Fleet will write the code? Who will maintain the code?
Where will the server be hosted? Who will be responsible for maintaining it?

who's the right person to answer this? don't want it to get lost in the convo

How will the scripts be triggered? Is this something that the server becomes responsible for?

some process needs to run at an interval and send commands, we were thinking this separate server (let's say fleetdm.com) do it

the challenge of building the functionality directly into Fleet is related to crafting the right enrollment profile, we thought that having a separate service gives us freedom to hardcode the profile to the customer's needs.

@noahtalerman maybe the profile could be provided to Fleet itself as a hidden config?

---

@zwass another option I just thought of: what if the proxy enqueues the command (using Fleet's API) to renew the SCEP certificate the first time it redirects a host to Fleet? this gives us 1 year to properly solve this problem.

[noahtalerman]

Who at Fleet will write the code? Who will maintain the code?

It's on the drafting board w/ the #g-mdm label. I think let's treat this as all other user stories at Fleet: bring it through estimation and into the next sprint.

Since this it sounds like the next release (2024-07-15) isn't fast enough I started a thread in #g-mdm in Slack here (internal) to chat about priority.

maybe the profile could be provided to Fleet itself as a hidden config?

@roperzh good idea. But is this because of a limitation of Heroku? If not, in order to move quickly, I think let's move forward with the current plan in the issue description.

If folks disagree, please bring jump in tomorrow's MDM design review to discuss.

Once we know what the enrollment profile will look like, we can get @eashaw's help to test. If we learn that using fleetdm.com won't work due to a Heroku limitation then I think we come back to other options.

[roperzh]

@roperzh good idea. But is this because of a limitation of Heroku? If not, in order to move quickly, I think let's move forward with the current plan in the issue description.

If folks disagree, please bring jump in tomorrow's MDM design review to discuss.

@noahtalerman sounds good! yeah, not a limitation with Heroku, but it might be simpler to run the cron in Fleet because:

1. It's a single server we have to worry about
2. It's very easy to add the expiration of the certs to the Fleet DB during the initial migration (vs having a separate db in fleetdm.com)

[zwass]

what if the proxy enqueues the command (using Fleet's API) to renew the SCEP certificate the first time it redirects a host to Fleet?

This seems possible. Currently there is no state maintained within the migration proxy, but state could be added.

[georgekarrv]

Hey team! Please add your planning poker estimate with Zenhub @dantecatalfamo @ghernandez345 @gillespi314 @roperzh

[georgekarrv]

Please add your planning poker estimate with Zenhub @jahzielv

[JoStableford]

Related to a Slack conversation

[roperzh]

As part of the research for this ticket I:

1. set up a MicroMDM server behind ngrok using https://roperzh-micromdm.ngrok.io
2. executed the database statements from the SQL generated by tweaks for SCEP renewals #20035
3. changed my ngrok config to point https://roperzh-micromdm.ngrok.io to a small proxy that redirects requests from /mdm/checkin and /mdm/connect to my Fleet server /mdm/apple/mdm
4. downloaded an enrollment profile from my Fleet server, and did the following changes:
   1. change ServerURL to be https://roperzh-micromdm.ngrok.io/mdm/connect (keep the old MicroMDM server URL)
   2. add a CheckInURL next to ServerURL with the value https://roperzh-micromdm.ngrok.io/mdm/checkin
   3. change the root PayloadIdentifier of the profile to be com.github.micromdm.micromdm.enroll
5. sent an InstallProfile command using the enrollment profile payload

I verified that:

1. The SCEP cert was renewed (🎉)
2. The new certificate was issued by Fleet's CA (so the customer doesn't need to keep their old CA around)
3. The enrollment profile in System Settings > Profiles now shows the host as enrolled by Fleet

---

Action items and stuff to coordinate on:

• @zwass I added you as a reviewer for the changes in the migration script because you have good context, but let me know and I can ask the team at large.
• @dherder @zayhanlon once add a workflow to renew SCEP certificates of migrated devices #20063 is released (estimated next week) we need to:
  • Build an enrollment profile using the instructions I left above for staging (and another for prod in the future)
  • Ask infra to set an environment variable FLEET_SILENT_MIGRATION_ENROLLMENT_PROFILE with the profile contents
  • Run the updated migration script to generate a new database dump and import it in staging.

[zwass]

@roperzh are you saying you got the enrollment profile replaced without user intervention? I'm not sure I understand how this experiment is connected with the touchless migration experience we are working on with customers.

[roperzh]

@zwass sorry for not being clear. This is to renew SCEP certificates for migrated devices (which is done by re-delivering the enrollment profile)

The enrollment profile was almost replaced, but three things need to be kept in our particular case:

1. change ServerURL to be https://roperzh-micromdm.ngrok.io/mdm/connect (keep the old MicroMDM server URL)
1. add a CheckInURL next to ServerURL with the value https://roperzh-micromdm.ngrok.io/mdm/checkin
1. change the root PayloadIdentifier of the profile to be com.github.micromdm.micromdm.enroll

[zwass]

Ah, so enrollment profiles can be redelivered without user intervention as long as the ServerURL and CheckInURL don't change?

[roperzh]

@zwass exactly! in my notes I have this as the full list of things that can't change:

• PushTopic
• ServerURL
• CheckInURL
• Enrollment type
• Access rights

I think the really important findings for us are:

1. We can switch to a diffrent CA for SCEP certs (in this case Fleet's built-in CA)
2. We can renew SCEP certificates for migrated devices seamlessly

[zayhanlon]

@roperzh how are we doing on target ETA to get this in a patch next week? thanks :D

[roperzh]

@zayhanlon thanks for checking, still on track! but please note that the issue w/profiles is probably a bigger blocker. This is majorly a blocker for the prod deploy, the profiles is limiting their testing in staging prior to any production changes.

[zayhanlon]

@roperzh yup! i'm on it - discussing with Noah today

[PezHub]

Paired w/ Roberto to test on his locally setup mircomdm server to ensure the workflow succeeded.

[noahtalerman]

Hey @zayhanlon, this customer-rosner request was shipped in Fleet 4.54.

No docs needed. Customer expect the MDM solution to handle SCEP renewal behind the scenes.

[fleet-release]

Old profiles renew,
Fleet's secure touch in the clouds,
Peace in tech ensues.