Add retries for software installs

[cdcme]

Fixes #34068

Summary

Adds automatic retries (up to 3 attempts) for failed software installs from host details, self-service, and setup experience across all installer types.

Details

Non-policy software installs previously had no retry mechanism. A single failure required manual re-triggering. This extends the existing policy-automation retry pattern to all install paths.

Key changes:

• New WithRetries flag on HostSoftwareInstallOptions enables attempt tracking at activation time
• On failure, the service layer re-queues the install up to MaxSoftwareInstallAttempts (3 times)
• Manual re-installs reset old attempt counters and cancel pending retries for a fresh sequence
• Setup experience status updates are deferred during intermediate retry attempts
• VPP app install retries are broadened from license-not-found errors only (VPP app install retry on 9610 #38008) to any MDM error (verification timeouts still mark the install as failed without retrying)

Expected Behavior

• Failed installs from host details, self-service, and setup experience retry up to 3 times automatically
• VPP app installs retry on any MDM command error (up to 3 times); verification timeouts don't trigger retries
• Manual re-triggers clear old retry state and start fresh
• Policy automation retries are unchanged
• Activity feed entries are created for each attempt

[codecov]

Codecov Report

❌ Patch coverage is 73.45133% with 30 lines in your changes missing coverage. Please review.
✅ Project coverage is 66.32%. Comparing base (6f52646) to head (40bfec5).
⚠️ Report is 281 commits behind head on main.

Files with missing lines	Patch %	Lines
server/service/orbit.go	66.66%	9 Missing and 5 partials ⚠️
server/datastore/mysql/software_installers.go	79.41%	4 Missing and 3 partials ⚠️
ee/server/service/software_installers.go	16.66%	4 Missing and 1 partial ⚠️
server/service/apple_mdm.go	60.00%	2 Missing and 2 partials ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main   #39827      +/-   ##
==========================================
+ Coverage   66.26%   66.32%   +0.06%     
==========================================
  Files        2438     2447       +9     
  Lines      195266   196546    +1280     
  Branches     8643     8643              
==========================================
+ Hits       129386   130366     +980     
- Misses      54167    54362     +195     
- Partials    11713    11818     +105     

Flag	Coverage Δ
backend	68.12% <73.45%> (+0.06%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[mna]

Arrrrrrghhhh I had a somewhat completed review and lost all comments when I switched back to conversation and back again to files changed...

[mna]

Oh no wait, I can go back to it, it just started a new review but the pending one is still available... New UX I think?

[mna]

I took a first look without reviewing the tests, left some comments. My main concern is with the unified queue, to ensure we don't introduce scenarios where the queue gets stuck.

I'll take a full look (including tests) on my next review.

[cdcme]

@mna I think I've addressed your feedback either with changes or by adding comments. Please lmk if anything else jumps out at you, and ty!

[mna]

@cdcme ty for the ping, I'll take another look this PM!

[mna]

LGTM, just a minor thing and the reuse of the cancel activity logic. Let me know if you'd prefer to address this in a subsequent PR, if so I'll approve this one.

[iansltx]

Looking at other usages here, it looks like we manage retries three different ways:

1. This (manually initiated installs)
2. Setup experience software retries (already existed prior)
3. Policy failure software install retries (already existed prior)

The latter two don't use the retry mechanism implemented here because they perform retries at a higher level of abstraction, right?

I'm pretty sure I saw the setup experience retries ticket come across a bit ago but don't recall when policy failure based retries happened. Can you link me the ticket for that one?

[cdcme]

Good question! Yes, broadly speaking, I think that's right. Are you thinking of PR #38018?

[iansltx]

Yep, that's what I'm thinking of for policies. Setup experience was done earlier.

[iansltx]

Deferring back to @mna for the last review pass since he's starting at 90% context here rather than 20%, so this'll get merged quicker (on Monday) that way.

[mna]

LGTM!