Inventory Freshness Developer Guide

🛠️ Inventory freshness - Developer Guide

How inventory freshness (#200, part B) is built, and how the agent came to share one inventory implementation with the REST API (the #194 follow-up). For what it does, see Inventory freshness. For why it is shaped this way, see the design.

In one paragraph: every source that reports what it saw on a device - the agent, or a REST tool - goes through AssetInventoryService. It reconciles the source's rows in the inventory tables, and for each section records two things in asset_inventory_sections: the last good data (observed_at, fingerprint) and the last attempt (last_outcome, last_error, last_attempt_at). The asset page reads that back through get_asset_inventory_status.php and says, per source and section, how fresh each list is.

---

1. 📁 The files

File	What it does
includes/services/asset_inventory.php	AssetInventoryService: submit() (REST), agentReport() + agentState() (agent), provenance(), staleDays(), hasOutcomeColumns(); internally applySection(), replaceAgentOnly(), recordOutcome(), agentItems(), bindAgent()
api/external/system-info/submit/index.php	The agent's main report. Matches the asset as before, then hands volumes, physical drives, adapters and software to agentReport()
api/external/device-manager/submit/index.php	The agent's Device Manager report, through agentReport()
api/external/software-inventory/submit/index.php	The older software-only agent, through agentReport()
scripts/Invoke-AssetInventory.ps1	The agent. Get-Section, $collectionFailures, report_version = 2
api/assets/get_asset_inventory_status.php	What the asset page reads: provenance() + rows per source + the stale setting
asset-management/index.php	loadInventoryStatus(), renderInventoryLines(), renderInventoryPanel(), invEmptyHtml(), invNeverCollected(); the inv-* styles
asset-management/settings/index.php, manifest.php	Inventory counts as stale after (days) on Discovery & reconciliation (asset_inventory_stale_days)
api/v1/lib/openapi_schemas.php, api/v1/spec.json	AssetInventoryProvenance gains the outcome fields; GET /assets/{id}/inventory describes them
includes/db_verify_schema.php, includes/db_verify_inventory.php, database/freeitsm.sql	The three new columns, and four relaxed to NULL
tests/inventory-freshness.php	The agent side (33 checks)
tests/asset-api-inventory.php	The REST side, now also checking that a failed section is recorded
tests/agent-empty-report-guard.php	Unchanged and still green: an older agent's empty list never wipes anything

---

2. 🔑 One implementation, two front doors

Before this, the same rows were written by four copies of the same logic: the REST service, and three agent endpoints that each did their own delete-and-reinsert. The review of #197 showed what that costs: narrowing one copy's delete quietly broke another source that relied on it.

Now there is one:

REST POST /assets/{id}/inventory ─▶ AssetInventoryService::submit()      strict
agent system-info / device-manager /                                    lenient
      software-inventory          ─▶ AssetInventoryService::agentReport()
                                          │
                     ┌────────────────────┼────────────────────┐
              applySection()       replaceAgentOnly()     recordOutcome()
       software, volumes, adapters  physical drives,      asset_inventory_sections
       row by row, by identity      Device Manager

The two front doors differ on purpose:

	submit() (REST)	agentReport() (agent)
Bad input	Refused - 422 invalid_field, the section fails	Trimmed to fit - a long registry string must never stop a PC reporting
Two items resolving to one application	The section fails	The first is kept, as the agent always did
Observation time	The tool's observed_at	Now
Ownership, device references, replay and stale checks	Yes	No - the agent has already matched the asset with AssetsService::reconcileAsset()
New catalogue application	source = 'api'	The column default ('agent')

Both reconcile through applySection(): rows matched by identity are updated in place (their ids survive), new ones are inserted, and for a complete section the rest are removed. The agent gives volumes and adapters an identity it never used to send, in agentItems():

Section	external_id the agent gets
Volumes	The drive letter, upper case (C:)
Network adapters	The MAC address, upper case; the name if there is none
Software	Not needed - the identity is the catalogue app_id

Two items with the same identity (Hyper-V gives virtual adapters the host's MAC) are told apart with a suffix (AA:BB:...#2) rather than one being lost.

Rows from before the upgrade have external_id NULL. applySection() collects them separately as $unnamed and a complete report removes them, so the first report after upgrading replaces them with identified rows rather than leaving them beside the new ones.

Physical drives and Device Manager live in tables only the agent writes, with no source or identity column, so replaceAgentOnly() replaces the asset's rows outright, as before - but inside the same savepoint and bookkeeping as everything else.

TRAP: the agent replaces Intune's stand-in volume

// TRAP: 'intune' too. Intune's volume is only a stand-in until the agent
// reports (intuneSyncDisks() skips an asset with agent rows) ...
$conn->prepare("DELETE FROM asset_disks WHERE asset_id = ? AND source = 'intune'")

This is the regression the #197 review caught, now inside agentReport(). tests/agent-empty-report-guard.php fails if it goes.

---

3. 🗄️ The last good data, and the last attempt

asset_inventory_sections            one row per (source link, section)
  observed_at      DATETIME NULL    when the last GOOD data was observed - NULL until it has once succeeded
  fingerprint      CHAR(64) NULL    sha256 of that data - the REST replay watermark
  imported_at      DATETIME NULL    when it was received
  imported_by_id   INT NULL         the analyst behind the REST key; NULL for the agent
  last_outcome     VARCHAR(16)      succeeded | empty | skipped | failed
  last_error       VARCHAR(500)     the failure, in the source's words
  last_attempt_at  DATETIME         when the last attempt happened, whatever it came to

The first four were NOT NULL in 3.12.0. dbVerifyInventoryUpgrade() relaxes them, probe-then-MODIFY, because a section can now be on record before it has ever succeeded, and the agent is no analyst.

recordOutcome() writes one of four things:

Outcome	Good data	Attempt	When
succeeded	moves on	now	A section applied with items (or a partial REST section)
empty	moves on	now	A complete section applied with no items
failed	untouched	now, with the error	The source said it could not collect, or applying it threw
skipped	untouched	now	The agent left it out - only updates a row that exists, so a section never sent stays "never collected"

TRAP: a failure never moves observed_at or the fingerprint

They are the REST replay watermark (REST-API-Assets). Moving them on a failure would make old data look new, and would make a retry of the failed report look like a replay.

Why REST never records skipped

The REST contract says an omitted section is untouched, and tools legitimately send sections on different schedules: software daily, disks weekly. Marking every omitted section skipped would paint them grey for no reason. The agent always sends everything in one report, so for the agent an omission means something.

---

4. 🖥️ The agent's side - report_version 2

An agent that predates this release sends [] both when a section is empty and when it could not find out. That is how a WMI blip once wiped 226 Device Manager rows (Inventory agent). So the server cannot believe an empty list unless the agent says it can:

{
  "report_version": 2,
  "collection_failures": { "devices": "Could not read Device Manager: ..." },
  "hostname": "LAPTOP-01",
  "software": [],
  "disks": { "logical": [ ... ], "physical": [ ... ] }
}

AssetInventoryService::agentState() is the one place that draws the line:

	Section is a list	Section is in collection_failures	Section is left out
report_version 2	apply it - empty included	failed	skipped
older agent	apply it if not empty; an empty list is skipped	(ignored)	skipped

TRAP: the endpoint decides what an empty list means

agentReport() takes the state it is given. Call it with 'apply' for an older agent's empty list and it will wipe the machine's inventory. Always go through agentState().

In the script, Get-Section wraps each collection, adds -ErrorAction Stop so a WMI failure throws, and records it in $collectionFailures. Volumes, physical drives, adapters and Device Manager pass -EmptyIsFailure: Windows answering nothing for something every reporting PC has is a failure to look, so it is reported as failed and the last list stays. Software is the one list trusted when empty.

Device Manager still goes to its own endpoint, now on every run (it used to be skipped when empty), so a failed or empty collection is recorded rather than looking like no report at all.

Compatibility, both ways:

• New agent, older server: report_version and collection_failures are ignored, and an empty list is still "left unchanged". Safe.
• Older agent, new server: agentState() treats it as version 1. Nothing changes for it, except that it now records outcomes too.

---

5. 🔗 Who reports this asset

asset_inventory_sources links a source to an asset. A REST tool's link is made by its first report, with its own device reference (REST-API-Assets). The agent's is made by bindAgent() on its first report after upgrading, with source = 'agent' and a device reference of asset:<id>:

• The agent has already matched the machine to the asset (serial first, then hostname), so the asset is its device reference. A hostname would collide when a name is reused; the asset id never can.
• 'agent' is in RESERVED, so no REST tool can claim it, and unlink() refuses it.
• provenance() lists the agent first and hides its reference (source_device_ref: null).

The link carries the company (tenant_id), which is what keeps one device to one asset per company. So when an asset moves company, AssetsService::moveToCompany() moves its links in the same transaction, and refuses the move if the new company already links the same device to a different asset - see Moving an asset between companies - developer guide. Before 3.15.0 the links stayed on the old company.

---

6. 🖼️ The asset page

loadInventoryStatus() runs beside the list loaders. The status and the lists arrive in either order, so both sides check:

• "Never collected" needs no record AND no rows (invNeverCollected()). An asset the agent reported before 3.15.0 has rows and no record, and must not say "never reported" - so it says nothing until its next report.
• The empty message under a list comes from invEmptyHtml(), which says never reported once the status is known. If the status arrives after the list drew its ordinary empty message, renderInventoryLines() swaps it.
• The tab badge shows – for never collected, so 0 means a real empty list. The layout's badge observer reads parseInt('–') as NaN and ignores it, so a hidden tab with nothing ever collected is not offered as "hidden details".
• Ages are elapsed, never calendar ("3 hr ago"), for the reason in seenHtml(). Amber after staleDays(). The exact time is in the title.
• The Inventory sources panel is deliberately not a data-layout-section. It is how a hidden list's freshness stays reachable, so no layout may hide it.

Nothing here reads the type's layout. Freshness comes from what each source did (design, Where we landed, point 1).

---

7. ⏳ Before Database Verification

An install that has pulled this code but not yet run Database Verification must keep taking agent reports, or every agent in the estate stops at once. hasOutcomeColumns() asks once per request:

• Agent: without the columns, agentReport() skips the source link and the bookkeeping entirely and only reconciles the rows.
• REST: recordOutcome() falls back to the 3.12.0 shape - success only, no failures recorded.
• Reads: provenance() reports a 3.12.0 row as last_outcome: succeeded, which is all it could ever have meant.

---

8. 🧪 Tests

php tests/inventory-freshness.php          # the agent: outcomes, identity, old vs new agent, status endpoint, stale setting
FREEITSM_TEST_URL=http://localhost/freeitsm-app php tests/asset-api-inventory.php   # REST, incl. failed sections recorded
php tests/agent-empty-report-guard.php     # an older agent's empty list never wipes anything

The checks to keep:

• a failed section keeps its rows and its observed_at
• an older agent's empty software list is skipped, a version 2 agent's is believed
• a volume row from before the upgrade (no identity) is replaced, not left beside the new one
• two adapters sharing a MAC are both kept

---

9. ⬜ Not done yet

• Intune still writes its own rows its own way (includes/intune.php) and records no outcomes. It is the next candidate for agentReport()-style treatment.
• No alerting on a failed or stale section. Watchtower counts machines that stopped reporting entirely.
• Parts C and D followed in the same release: Connected assets and Asset review.