Fresh Intune Sync Design

Here it is, in project doc format so it pastes straight in.

Intune Collector: Design and MVP Scope

Purpose (final)

Build an Intune collector that retrieves corporate Windows managed device data from Microsoft Graph, writes a normalized JSON snapshot, and feeds fs_enrich for identity matching, field enrichment, adoption stamping, and agent gap reporting. Intune is an enricher, not a lifecycle owner. The Freshservice Discovery Agent keeps lifecycle for laptops and desktops. No direct Freshservice writes from the collector. No seeding in MVP.

Design principles

1. One collector, one API. Collector talks only to Graph. It knows nothing about Freshservice.
2. Update only. Discovery Agent owns create and retire for this class. Intune never writes fields the agent maintains (OS, hardware, serial, hostname).
3. Single writer per field. Intune owns the intune_* group exclusively. Nothing else writes those fields, Intune writes nothing else.
4. Corporate Windows only, filtered server side in the Graph query, not client side after pulling everything.
5. Fail stale, never null. Collector failure leaves the old snapshot in place. fs_enrich skips stale sources. No blanking fields on API errors.
6. All writes flow through fs_enrich guards: dry run default, per run cap, cascade halt on match rate collapse, ledger line per action.
7. Unmatched corporate devices are an agent coverage gap, not a seeding queue. Output is a report for the endpoint team.
8. Idempotent. Any run can be repeated with no side effects beyond the ledger.

Target architecture

Auth: Entra app registration, client credentials flow, application permission DeviceManagementManagedDevices.Read.All. Token fetched with plain requests against the tenant token endpoint. No new EE dependencies, the existing image already carries everything needed.

Vault: new path secret/infra/dev/intune with keys tenant_id, client_id, client_secret. Same AppRole pattern as vcenter, scoped read only policy, Path to Auth explicitly approle.

Collector: tools/intune_collector.py in the existing repo.

Graph query:

GET /v1.0/deviceManagement/managedDevices
  ?$filter=managedDeviceOwnerType eq 'company' and operatingSystem eq 'Windows'
  &$select=id,deviceName,serialNumber,azureADDeviceId,userPrincipalName,
           emailAddress,complianceState,lastSyncDateTime,enrolledDateTime,
           managementState,managedDeviceOwnerType,model,manufacturer

Paginate on nextLink until exhausted. Write to a temp file, atomic rename to /var/lib/awx/fs_staging/intune_latest.json.

AAP: job template FS Intune Collector, instance group cp01_only, daily schedule in a free UTC slot, credentials injected via the vault pattern.

Snapshot record shape

{
  "source": "intune",
  "collected_at": "2026-07-29T05:10:00Z",
  "record_count": 1834,
  "records": [
    {
      "identity": {
        "serial": "5CG4123ABC",
        "hostname": "lt-jsmith-01",
        "azure_ad_device_id": "a7f1...",
        "user_principal_name": "jsmith@company.com"
      },
      "payload": {
        "compliance_state": "compliant",
        "last_checkin": "2026-07-29T04:52:00Z",
        "enrollment_type": "windowsAzureADJoin",
        "management_state": "managed",
        "model": "Precision 5690"
      }
    }
  ]
}

Collect wide, write narrow. Payload holds more than we map on day one.

Fresh fields to request (Computer level)

Field	Type	Notes
intune_compliance	text or dropdown	compliant, noncompliant, unknown
intune_last_checkin	date	device to Intune checkin
intune_enrollment_type	text
intune_management_state	text
intune_primary_user	text	parked candidate for Used By promotion
intune_last_seen	date	our stamp, staleness proof

match_key, source, instance_uuid already exist per the screenshot. Adoption stamp reuses them, zero new fields.

Data flow

flowchart TD
    GRAPH[Microsoft Graph<br>corporate Windows filter] --> COLL[intune_collector.py<br>daily, cp01]
    COLL --> SNAP[(fs_staging/intune_latest.json)]
    FRESHSNAP[(Fresh snapshot<br>15 min)] --> ENGINE
    SNAP --> ENGINE[fs_enrich<br>identity match, guards]
    ENGINE -->|matched| PUT[PUT intune_* fields<br>+ adoption stamp]
    ENGINE -->|unmatched| GAP[(agent gap report)]
    PUT --> FS[Freshservice]
    PUT --> LEDGER[(audit_ledger.jsonl)]
    GAP --> TEAM[Endpoint team remediation]

Identity matching, MVP rules

1. Serial number, exact match against Fresh laptops and desktops, after blocklist filter (pending your item 5 verdict, blocklist ships with obvious defaults like "System Serial Number", "To be filled by O.E.M.", "Default string").
2. No serial match, fall back to hostname exact match, unique hits only.
3. Multiple matches on either key, straight to pending, never guess.
4. On successful match, store azure_ad_device_id as an alias in the identity map and stamp match_key and source on the asset. Future sources match instantly.

Agent gap report

Corporate Intune device with no Fresh match. Per run output intune_agent_gap_TIMESTAMP.csv in fs_reports, deduplicated per device, entries age out after 30 days unseen. Delivered to the endpoint team as agent remediation work. Stretch goal, not MVP: the reverse report, agent managed Fresh laptop absent from Intune, which is an enrollment gap. Cheap to add once both datasets sit in staging.

Guards config for this source

FS_ENRICH_SOURCE=intune
FS_REPORT_FRESHNESS_SECONDS=93600        26h, daily cadence plus margin
FS_ENRICH_MATCH_RATE_HALT=80             halt if under 80 pct match
FS_ENRICH_MAX_PER_RUN=500                initial cap, raise after trust
FS_APPLY=false                           dry run default, as always

MVP scope

In: collector, staging snapshot, identity matching with blocklist, intune_* enrichment, adoption stamping, agent gap report, dry run soak, capped live enable, ledger, daily schedule.

Out: seeding, Used By promotion, BYOD, macOS and Jamf, device actions, reverse enrollment report, dedicated Grafana panels.

Acceptance criteria

1. Collector completes full corporate fleet paginated inside 10 minutes, atomic snapshot written.
2. Dry run week one: match rate reported, you spot check 10 matched devices and 5 gap entries, gap entries confirmed as genuinely missing agents.
3. Live: PUTs under cap, every write has a complete ledger line, rerun produces zero deltas.
4. Gap report handed to endpoint team and at least one entry independently confirmed.

Open items before build

1. Entra app registration approval, fires day one.
2. Fresh dev creates the six fields above.
3. Serial quality verdict from you, blocklist ships regardless.
4. Used By activity log check on three or four laptops, decides promotion later, blocks nothing now.

Build effort: 3 to 4 days, same shape as the SCCM collector estimate. Slots into the sprint plan wherever you want it, cleanest fit is alongside or right after SCCM in Sprint 2 since both exercise the same engine path, or its original Sprint 5 slot if the MVP queue stays fixed.

Anything you want changed before this goes into project files?

Fresh sync error:

No, this is NOT an API key expiration. An expired or invalid API key returns 401 Unauthorized or 403 Forbidden.

The error HTTP Error 429: Too Many Requests means you have exceeded the Freshservice API rate limit quota.

---

Root Cause Analysis

1. API Rate Limiting (HTTP 429): Freshservice explicitly returned retry_after: '38' and x_fw_ratelimiting_managed: 'true'. Freshservice is telling the automation to wait 38 seconds before making another API call.

2. Retry Delay Mismatch in Ansible: In main.yml:6-7, the default retry settings are:

   • fs_max_retries: 5
   • fs_retry_delay_base: 2 (waits only 2 seconds between retries)

   In upsert.yml:39-43, when a 429 occurs, Ansible retries every 2 seconds. All 5 retries burn out within ~10 seconds while Freshservice's 38-second rate-limit window is still active.

3. Why this happens without recent project code changes:

   • Freshservice Policy: Freshservice may have tightened API rate limits on your account or gateway level (x_fw_ratelimiting_managed: 'true').
   • Concurrent Jobs: Another process (e.g., snapshot generation fs_inventory_snapshot.py or another AAP job) might be using the same API key bucket at the same time.
   • Batch Concurrency: serial: 10 in sync_all_vms.yml:5 sends requests too fast for the API rate limit bucket.

---

Recommended Solutions

1. Reduce Batch Concurrency: Lower serial in sync_all_vms.yml:5 from 10 to 2 or 3 to space out PUT requests.

2. Increase Retry Delay & Retries: Update main.yml:6-7:

   fs_max_retries: 8
   fs_retry_delay_base: 10

   This gives Ansible enough time to wait out the 38-second retry_after window.

3. Stagger Schedules: Ensure fs_snapshot_refresh.yml and bulk sync runs are staggered so they don't consume API quota simultaneously.

Even though the code and calculated rate limits haven't changed in 3 months, encountering HTTP 429 (Too Many Requests) usually happens due to subtle shifts in the operational environment or source data rather than code bugs.

Here is why a sync configured for rate limits can suddenly start failing with 429s:

---

1. Mass Delta Cascade (The Primary Suspect)

• How the rate limit calculation works: The system was designed around eventual consistency & deltas. During a typical run, 95%+ of VMs are unchanged and skipped (No changes for <VM>, skipping update), so only a few dozen PUT requests hit Freshservice per run.
• What changed in the environment: If an external system or script recently updated a custom attribute across all vSphere VMs (e.g. defender_health, wsus_patching_configured, powerstatesince, or vCenter tags), every single VM now has a delta (fs_needs_update = true).
• Instead of sending 20–50 PUT requests, AAP is suddenly trying to send thousands of PUT requests continuously with serial: 10, immediately blowing past Freshservice's API rate limit bucket (e.g. 500 requests/minute limit).

---

2. Snapshot or sync_last_seen Stale Refresh

• In upsert.yml, if sync_last_seen_55000745586 is missing, unparseable, or older than fs_refresh_threshold_hours (6 hours), the playbook marks _fs_refresh_due: true (refresh_stale).
• If the snapshot file /var/lib/awx/fs_snapshot/fs_vmware_assets.json wasn't generated properly or Freshservice returned empty type_fields for last_seen, the playbook forces an update on 100% of VMs.

---

3. Ansible Retry Timing vs. Freshservice retry_after

Look closely at the error response Freshservice returned:

retry_after: '38'
msg: 'HTTP Error 429: Too Many Requests'

• Freshservice edge proxy explicitly tells the client to wait 38 seconds before retrying.
• However, Ansible's defaults in defaults/main.yml are:

  fs_max_retries: 5
  fs_retry_delay_base: 2

• Ansible retries after 2s, 4s, 6s, 8s, 10s — exhausting all 5 retries in under ~20 seconds, well before Freshservice's 38-second window expires!

---

4. Shared API Key / Tenant Usage

• If another team member, script, or automated job (like TVM reconciliation, reporting, or fs_inventory_snapshot.py) is making API calls using the same Freshservice API key or tenant concurrently, they consume the same rate-limit quota pool.

---

Quick Fixes to Resume Your Work

1. Verify why hosts are updating: Run a dry run (sync_dry_run: true) on a subset of hosts to check the debug log message:

   "DRY RUN would update VM, reason <delta|refresh_stale>, changed fields [...]" This will tell you instantly if a specific attribute changed across all VMs.

2. Adjust retry timing in roles/freshservice_vm_sync/defaults/main.yml: Change fs_retry_delay_base to 10 and fs_max_retries to 6 so Ansible waits out the 38-second retry_after window instead of burning through retries instantly.

3. Temporarily reduce concurrency: In playbooks/sync_all_vms.yml, lower serial: 10 to serial: 2 or serial: 3 until the mass updates settle.

Created 4 todos