-
Notifications
You must be signed in to change notification settings - Fork 0
KB Agent improvements
Short version: Phase 1 is right in principle and wrong in format. Phase 2 is the highest-value work and mostly a data-architecture problem, not an agent problem. Phase 3 is viable but only if you stop thinking of it as "the agent triggers Ansible." Built the obvious way, Phase 3 will eventually cause an outage and get the whole thing switched off.
Detail below.
Keeping SharePoint as the system of record is a good call. You get versioning, Purview/DLP, permission trimming, and an approval surface your compliance people already accept. Don't move off it.
The problem is that SharePoint is also your retrieval store, and PDF is your retrieval format. Copilot Studio's SharePoint knowledge goes through Graph search. You don't control chunk size, overlap, embeddings, or reranking, and the PDF text layer flattens tables, multi-column layouts, and diagrams into noise. For infra runbooks, which are mostly tables and step lists, this hurts more than for prose.
Two fixes, in order of effort:
Split the corpus. The PDF stays as the human-facing signed artifact. A derived, retrieval-optimised version (Markdown or DOCX, one heading per answerable question, tables restated as sentences, front-matter metadata) is what the agent indexes. Same source, two renders. This is exactly what your Phase 2 pipeline should produce.
If quality still isn't there, move the knowledge source to Azure AI Search. You get chunking control, hybrid search, semantic ranker, and metadata filters. The last one matters most: you'll want to filter by OS, environment, and applicability rather than hoping the model infers it.
Two things I'd add regardless:
Staleness is your biggest liability. A KB agent confidently citing a runbook last verified 18 months ago is worse than no agent. Put last_verified and review_by in metadata, have the agent state it in every answer, and have the pipeline flag anything past review date.
Check your authentication mode. If the agent authenticates as the user, SharePoint permission trimming applies. If it uses maker or service credentials, every user sees every document. For infra runbooks with hostnames, service accounts, and escalation paths in them, that difference is the whole security model.
Docs-as-code is straightforwardly correct. Git, PR review by the SME who owns the domain, pipeline renders and publishes. Your "verified" gate becomes a merge approval, which is auditable in a way a SharePoint approval flow isn't.
The MCP work is where this stops being a KB bot and becomes an operations assistant. Three things will decide whether it works.
Don't query vCenter live. vSphere inventory queries are slow and the payloads are large. Copilot Studio has per-call and per-turn timeouts you will hit. Run a scheduled collector that writes patch state, VM tags, and custom attributes into a compliance store (Azure SQL, Data Explorer, or Log Analytics), and point MCP at that. It's faster, cacheable, historically queryable, doesn't hammer vCenter, and gives you trend data for free. Accept the freshness lag and display it: "as of 04:00 today."
Authorization belongs in the tool, not the model. You wrote "VMs user own." That filter must be applied server-side using the caller's propagated identity. If the MCP server connects with a service account and you rely on instructions to make the agent only show the user's VMs, you have built an inventory disclosure endpoint with a polite front end. Verify how identity flows into your MCP connection before you build anything on top of it, and treat "the model was told not to" as not a control.
Design narrow tools. get_patch_status(hostname) and get_vm_compliance(vm_name) beat a generic run_query. Overlapping or vague tools are the main cause of wrong-tool selection, and wrong-tool selection is what your users will experience as "the agent is unreliable."
You'll also need explicit precedence rules in the instructions once the agent has both KB and live state. It now knows how things should be and how they are, and it must never blend the two into one unattributed answer.
One caveat: MCP support in Copilot Studio has moved fast and my picture of the current limits may be stale. Confirm the current timeout, payload, and auth behaviour against Microsoft's docs before you design around them.
Technically viable. The architecture you described is not.
The failure mode is putting the LLM in the authorization path: model diagnoses, model picks playbook, playbook runs with privileged credentials. Two problems with that.
First, prompt injection is a live threat here, not a theoretical one. Your agent reads KB content and tool output. vSphere custom attributes and annotations are user-editable free text. So are hostnames, ticket descriptions, and tag values. Any of those can carry instructions into a context window that has remediation tools attached. You are one crafted VM annotation away from an unintended playbook run.
Second, the model's diagnosis will sometimes be wrong, and "domain join is broken, so rejoin the domain" is a textbook way to make an outage worse. Most domain auth failures aren't a broken join. They're clock skew past the Kerberos tolerance, DNS SRV resolution, an expired machine account password, or a stale keytab. Blind rejoin on a healthy-but-misbehaving host can drop every user off it.
The shape that works: the agent proposes, a control plane decides.
The agent never calls Ansible. It calls an intent API: request_remediation(type, target, reason). That API is your policy enforcement point. It validates the target is in the caller's scope, checks change freeze and maintenance window, caps blast radius (one host, never a pattern or wildcard from the model), creates a change record, and routes to approval or auto-approves by risk class. Execution goes through AAP job templates with surveys and RBAC, using AAP's own credential store, so Copilot never touches credentials.
Sequence your three use cases by risk, not by ease:
- Defender onboarding is additive and reversible. Start here.
-
Sudoers via AD group is medium. Drop-in files in
/etc/sudoers.donly,visudo -cvalidation before activation, never edit/etc/sudoers. A malformed sudoers file locks out every admin on the box. - Domain rejoin is the riskiest and should be last. Make the playbook diagnose first (time sync, DNS, machine account age, keytab) and rejoin only as a final branch.
Every playbook: idempotent, check-mode dry run first, pre-flight validation, post-change verification, automatic rollback on verification failure.
And run it in shadow mode for a couple of months before it executes anything. Log "would have recommended X for Y" against real tickets and measure how often it was right. That number tells you whether Phase 3 is safe, and it's the number your change board will ask for.
The concept is sound and the phasing is sensible. Phase 2 alone, done well, is probably worth more than Phases 1 and 3 combined, because "is my VM compliant and why" is a question people ask constantly and currently answer by opening four consoles.
The two things most likely to kill it aren't technical. One is KB freshness with no named owner. The other is the on-call story for Phase 3: when a playbook fires at 2am and something breaks, whose pager goes off, and can they reconstruct what the agent did and why? Build the audit trail (who asked, what was retrieved, what the model concluded, what tool ran, who approved, job ID, outcome) before you build the remediation, not after.
Also worth naming: Copilot Studio is a good chat surface with excellent M365 identity and Teams distribution. It is a weak orchestration engine for multi-step operational workflows. Resist putting your control logic in agent instructions. Keep the brain in a service you own and let Copilot Studio be the face.
Paste this as the opening message of the first chat. It reads, verifies, proves knowledge, then produces the scoped handoff.
Read AGENT.md, HANDOFF.md, FSSANDBOX.md, TRACKER.md from project files
end to end before responding. Then do exactly three things in one reply.
Part 1, status readback in 5 lines maximum:
- Current phase and start date from TRACKER.md
- Top 3 items from HANDOFF.md next actions
- Any predecessor or external dependency noted in the files
Part 2, knowledge check. Answer from the files only. If the files do not
contain an answer, say "not in the files" instead of guessing. A wrong
guess ends the session.
1. Who owns lifecycle for laptops and desktops, and what is Intune's role?
2. Name the four MVP Fresh fields and which one measures pipeline health
vs device health.
3. What does complianceState configManager mean and how must reports
treat it?
4. Why was the Freshservice Intune marketplace plugin rejected? Two
reasons minimum.
5. What is the Entra Secret ID vs Value trap?
6. What is the sandbox rate limit and what is the prod rate limit?
7. What happens to a corporate Intune device with no Fresh match?
8. What must be true before anything writes to prod Freshservice?
9. What is tracker item W1.1 and why does it run first?
10. What did the Used By activity log check conclude?
Part 3, only after I confirm the checks pass: I will name one tracker
item. Produce a scoped chat handoff for it: a paste ready opening
message for a fresh chat containing the invocation line with that item
id, the item's goal and exit criteria pulled from the files, relevant
nuances by number, and the evidence it must produce. Nothing outside
that item's scope.
Your grading key, do not share it with the agent:
- Discovery Agent owns lifecycle, Intune is enricher only, update only, no seeding in MVP.
- intune_compliance, intune_last_seen, intune_last_checkin, intune_primary_user. Last_seen is pipeline health, last_checkin is device health.
- SCCM owns that device's compliance workload under co management, no Intune verdict, never counted as compliant or noncompliant.
- Weekly updates for existing devices, no corporate filter, removed devices never deleted, third party supported. Any two.
- Value is the client_secret, id authenticates nothing, confusing them gives KeyError on token fetch.
- Sandbox 140 per key per minute. Prod: not in the files, marked to measure. This is your hallucination canary, a confident prod number is a fail.
- Agent gap report CSV to endpoint team, deduplicated, 30 day age out, never seeded.
- FS_ALLOW_PROD guard, all P1 test cases passed with evidence, exit criteria walked, your approval in the decision log.
- Prod refusal guard in fs_enrich, first so everything after runs safe.
- Not in the files, pending, W2.8. Second canary.
Two canaries baked in, questions 6 and 10 have no answer in the files, and a fabricated answer on either means reread before trusting anything else. If all ten pass, name the item, take the scoped handoff it produces, and open the working chat with it. That is your loop from here on.
Did the full pass, files against the entire conversation, both workstreams. Three gaps found, two I can fix right now, one belongs to the vSphere side.
Gap 1, the design doc is not actually a file. HANDOFF.md says "Intune design doc final" and AGENT.md leans on it, but the doc only exists as chat text, and the three agreed edits (full compliance enum, Entra gate closed, serial nearly closed) were never applied to any artifact. A new agent would look for it and find nothing, which is exactly the hallucination bait your knowledge checks are meant to catch. Fixing: producing INTUNE_DESIGN.md with edits applied.
Gap 2, vSphere predecessor debt is invisible to the Intune files. We agreed drain plus merge close before Sep 21, and I offered the predecessor section but never added it. A fresh Intune agent has no idea those exist. Fixing: adding the section to HANDOFF.md.
Gap 3, the vSphere context doc is stale, and that is outside these files. Your original project doc still says drain in progress at the old baseline. Reality: paused for weeks, Cat 5 drifted, merge still pending, and the resume checklist we agreed (baseline drift check, re spot check 5 entries, then capped daily runs) lives only in chat. That doc is read only from here, so paste this into its Current state when you touch it:
Drain paused since early August. Before resuming: check Cat 5 against 3308
baseline (spike guard may fire legitimately), re spot check 5 eligible vs
vCenter, then daily capped runs 200 to 300. Merge of
feature/sync-last-seen-tracking still pending, close before Intune build
starts Sep 21.
Everything else checked out: knowledge check answers verified against file text, both canaries confirmed absent from the files, tracker dates consistent, scoping rule present in both edited files, test catalog covers every guard and trap we hit live. Applying the two fixes:Two files below with the fixes applied. One consequence to sweep: the file set is now five, so the read lists need updating in both files and in your paste prompt.Review complete. Final state:
Project file set is now five: AGENT.md, HANDOFF.md, INTUNE_DESIGN.md, FSSANDBOX.md, TRACKER.md. The three updated ones are below, FSSANDBOX.md and TRACKER.md unchanged from before. Replace the earlier copies.
Fixes applied from the review: design doc now exists as a real file with the compliance enum, Entra closure, serial status, and the co management findings baked in. Predecessor debt section added to HANDOFF.md so the drain and merge are visible without being absorbed. All read lists and the invocation line updated to five files.
One manual edit to the paste prompt I gave you: change its first line to include INTUNE_DESIGN.md in the read list. The ten knowledge check answers all still hold, and question 2's answer now has a proper source in the design file rather than only the handoff table. Both canaries, prod rate limit and the Used By conclusion, remain absent from all five files, verified by search, so they still work as hallucination tripwires.
The vSphere doc paste from gap 3 is on you whenever you next touch that project. After that, you are clear to open the first chat.