Directory Sync

Directory sync — importing people from Active Directory

Status: built and shipped, apart from scheduling. This page records the whole design conversation — the questions, the answers, the reasoning, the three places the first answer was wrong, and what changed once it met a real directory. It is the thinking behind the feature, not the manual for it: for that, see Importing people from Active Directory or LDAP.

Sections marked Built say how the idea actually turned out, which is not always how it was planned — §3.9 in particular records a design note here that was only half right.

The request, verbatim, by email:

"And another thing: how do I import my Active Directory users to assign them the assets?"

Related: LDAP and Active Directory (sign-in, which already exists) · LDAP — Developer Guide · Setting up LDAP with Docker · Multi-tenancy: users and self-service

---

1. Why the answer was "you can't"

FreeITSM has had LDAP/AD sign-in since #47. It binds with a service account, gates on a group, and creates the person's record when they first sign in — just-in-time provisioning.

That is a complete answer to "let my staff log in" and no answer at all to the question asked, for one reason:

To assign a laptop to Jane, Jane must already exist. JIT means Jane does not exist until she signs in — and the people you most want to assign assets to are exactly the people who will never sign in.

The no-mailbox warehouse and shop-floor staff the self-service portal was built for are the same population that holds equipment and never logs into anything. So the asset register cannot be populated ahead of the humans arriving, which is the normal direction of work: kit is issued to a new starter before they turn up.

A search of the codebase for any bulk import or sync returned nothing. ldapFindUser() substitutes one login into one filter — it is built to answer "is this person real", not "list everyone".

---

2. What already exists, and is better than expected

Thing	Where	Why it matters here
Per-company providers	auth_providers.tenant_id	"sync per company" needs no schema change
Multiple providers per company	multiple rows	"one or more sources per company" already possible
A stable identity anchor	user_sso_identities.subject + auth_providers.ldap_attr_guid	This is objectGUID. Tracking a person across an OU move or a rename is already architecturally solved — the hard part of question 4 was done before we started
Attribute mapping, partially	ldap_attr_username, _email, _name, _guid	The pattern to extend, rather than invent
A two-flavour test directory	docker/ldap-test/	Samba AD and OpenLDAP, so code cannot silently grow one directory's assumptions

And one thing that does not exist and blocks everything:

🔴 users has no is_active column. There is no way to record that somebody has left. The only options today are delete — which destroys asset history, tickets and handover records — or nothing.

---

3. The questions, the answers, and the reasoning

These were asked in one go after the survey above. They are answered in the order asked.

3.1 Users, analysts, or both?

Users first, and they are separate problems.

Analysts get permissions, so bulk-creating them is a security event, not a convenience — and analysts already have a reasonable answer in JIT plus a group gate. An analyst who has never signed in does not need to exist.

Asset holders are the exact opposite: the entire point is that they exist before they ever log in.

3.2 Sync per company

Already supported. auth_providers.tenant_id and users.tenant_id exist. A sync inherits its provider's company.

3.3 One or more sources per company

Already possible. The interesting part is not the plumbing, it is what happens when the same human appears in two directories — see §4, where the first answer to this was wrong.

3.4 Somebody moves out of a synced OU, or cannot be found

This is the question that decides whether the feature is safe, and it deserves more caution than the others combined.

Never delete: assets, tickets and handover documents all hang off user_id.

Three rules:

1. A sanity brake. If a run finds dramatically fewer people than the last one — more than about 20% missing — it stops and reports rather than deactivating anybody. A typo in a base DN, a service account quietly losing read rights, or a directory being slow would otherwise deactivate an entire company in one pass. That failure would end trust in the feature permanently, and it would be our bug presenting as their disaster.
2. N consecutive misses before acting. Missing once is noise. Missing three runs is a fact.
3. Record last_seen_in_source, so "when did this person stop appearing" is answerable after the event.

3.5 Accounts disabled in the directory

Deactivate, never delete — which needs the is_active column from §2.

There is a bonus here worth building for deliberately: a deactivated person still holding equipment is an offboarding alert. "Three people who left this month are holding nine items between them" is a report every IT manager wants and almost none have.

⚠️ "Disabled" and "moved out of scope" are different states and a sync must not conflate them. The test fixture has one of each on purpose (q.deleon disabled but still in Sales; y.tanaka disabled and moved to Leavers).

3.6 What fields do we have, and do we need more?

Today users is an authentication record, not a person: id, email, display_name, password_hash, preferred_name, TOTP and MFA columns, auth_provider_id, theme_preference, tenant_id, username.

Proposed additions:

Field	Why
is_active	§3.4/§3.5 — nothing else will do
department, job_title	"who in Finance has a laptop"
office	🔑 only 1 of 566 assets on the development install has a location set. AD's physicalDeliveryOfficeName populates the very field the ticket asset-picker needs and nobody fills in by hand
phone, mobile	contacting whoever is holding a thing
employee_id	matching against HR and payroll exports
manager_id	📌 already wanted and absent — catalogue-request approvals (#928 slice 2) has been blocked on this. AD gives you the org chart for free
directory_username	§4.3
last_seen_in_source, sync_source, is_managed	bookkeeping for §3.4 and §3.13

3.7 Custom attributes

Yes, but not as columns. A key/value side table for whatever a particular customer's directory happens to carry, with the fields above kept as real columns because you will want to search and filter on them. Columns for what FreeITSM understands; key/value for what it merely stores.

3.8 An attribute-mapping interface

Essential, not optional — and the test directory proves why: Samba AD uses sAMAccountName, OpenLDAP uses uid, and nothing can guess between them. The pattern already exists in ldap_attr_*; this extends it.

Built, as a tab of its own. The eleven attribute boxes had ended up split across two tabs — four on Signing in, seven on Importing people — which read as two unrelated settings rather than the single map they are. They are now one table: FreeITSM's field on the left, the directory's attribute on the right, in the order the sentence is actually written.

The Test button is the point of it. An attribute name is abstract and has to be taken on trust; employeeID → NW-1011 does not. A mistyped attribute is the mistake nobody catches, because nothing fails — a column is simply empty, and it goes unnoticed for weeks.

An empty result is ambiguous on its own, so the test also lists every attribute the sample person carries. That is what separates "the attribute name is wrong" from "this person has not filled it in":

Sample	Attributes carried	Result
Nia Hughes	38	every mapped field found a value
Wendy Warehouse	33	six mapped fields empty — and the list shows she genuinely has none of them

It tests the values on the form, not the saved ones: testing saved values would only ever confirm the last thing saved, which is not the question anybody is asking. And if neither the sign-in name nor the unique id resolves, it says so outright and in red, because an import would skip that person entirely — the table and the attribute list still render, since those are exactly what show you why.

3.9 An OU browser with tick boxes

The best idea on the list. Typing a base DN is where people fail; browsing a tree is where they succeed.

Built. Two things about it were decided during the build rather than here.

A ticked branch means the whole branch, including anything added to it later. The alternative — ticking exactly the OUs you want, and nothing else — was rejected because a new top-level OU then never imports and nothing says so: the new starters are simply absent. So what is stored is the exceptions, not the members: which branches are in, and which have been carved back out of them. Storing every member OU would freeze the selection at today's shape of the directory, which is the same silent-staleness problem wearing a different hat.

That makes "everybody in Staff except Contractors" expressible while still picking up Staff/Legal when HR creates it next year. Unticking something inside a ticked branch writes a carve-out, and the tree draws it struck through.

⚠️ The comma in the suffix test is load-bearing. "Is this DN under that branch?" is a string comparison, and without requiring , before the ancestor, OU=Sales matches OU=WholesaleSales — so a carve-out silently swallows an unrelated department. A wrong answer that looks exactly like a working feature; asserted in both directions in tests/directory-sync-scopes.php.

Head counts are what make a tick checkable. Each branch shows how many people sit in it directly and how many are in the whole branch, using the same filter the import uses — a different one would promise 24 and deliver 19, which is worse than showing no number at all. They come from one paged search rolled up in PHP, not a search per OU.

The DN-versus-GUID question, answered honestly

The concern above was real, and the GUID self-healing was not built. DNs are stored on their own. What was built instead is detection — because the failure has two halves and only one of them had been noticed:

What changed in AD	What the next run does	Why
A selected branch is renamed or moved	Refuses, and changes nothing, naming the branch	Importing the branches that do still exist is precisely how the damage happens: everybody in the missing branch looks like they have left, and after sync_deactivate_after runs they are marked as such
A carved-out branch is renamed or moved	Proceeds, and says so loudly	The people you excluded start arriving. Unwanted, but additive and undoable — refusing would stop everybody's import over the smaller problem

The second row is the half the original note missed, and it is the more dangerous one to leave silent: the sanity brake only guards against a sudden drop, so people being added is invisible to it. Nothing in the system would ever have mentioned it.

The check costs one base-scope existence test per stored branch, and there are usually one or two. It is skipped entirely for installs still using a typed starting point, which have always behaved this way — warning on every run of a working setup only teaches people to ignore warnings.

GUID self-healing is still the better answer and still worth doing: it would make a rename a non-event rather than a message. Detection came first because the thing worth killing is the silent wrong answer, and a message you can act on kills it.

3.10 A sync log

Yes, following the pattern of the email send log: a per-run summary plus per-record detail, recording what changed. "47 updated" is useless. "47 updated, 3 deactivated, and here they are" is what somebody would actually read.

3.11 A preview / test run

Yes, and mandatory for the first run of a new source. A dry run that shows exactly what would be created, updated and deactivated, writing nothing.

This is the natural partner to the sanity brake: together they turn "I hope this is right" into "I can see what it will do".

Built in slice 2 — and then found to be only half-built, which is worth recording because the half that was missing was not the hard one.

A preview produced four numbers: "31 found, 1 added, 1 updated, 0 marked as left". That asks to be trusted, which is the opposite of what a preview is for. The per-person detail had been written to directory_sync_entries since the engine was first built, and get_directory_sync_log.php had served it by run_id the whole time. Nothing ever called it. The information existed, was correct, and was invisible.

Previewing now opens the list of everyone the import would touch, with what would happen to each and — for a change — which field, from what, to what:

Person	What happens	Detail
Priya Newstarter	Will be added	Email: p.newstarter@northwind.test; Job title: Sales Executive; Department: Sales
Nia Hughes	Will be updated	Job title: 1st Line Analyst → 2nd Line Analyst; Phone: 020 7946 0011 → 020 7946 0099

The tense follows the mode. A preview says "Will be added"; a completed import says "Added". Labelling a preview row "Added" states as fact something nobody has agreed to yet, and not-having-happened is the entire value of the feature.

Every row of the import history opens the same view, so a run from three weeks ago answers "updated how, and who?" exactly as readily as the one just performed. Two smaller things surfaced while building it: the detail was printing the database's column names (job_title:) rather than yours, and a person about to be created said only "Would be created." — true, and impossible to check. A new arrival now lists what their record will hold, or says plainly "No details beyond a name", which is itself worth seeing before an import.

3.12 The users screen

asset-management/users.php is read-only today — search, select, view assets, print a handover.

The dilemma raised was: it belongs under Assets, but users also relate to tickets. That dilemma dissolves once you separate two decisions:

• Sync configuration is not in question. LDAP providers are configured at System → SSO, and the sync settings belong beside the directory they describe. Putting them under Asset Management would mean configuring the same AD twice, and the next module that wants directory data makes it three.
• The people screen is the real question, and the industry pattern is consistent: the person is a core record owned by the platform, and modules provide filtered views of it. The module that happened to need people first should not own them.

Decision: one screen, two doors. The screen becomes the canonical directory under System → Users; Asset Management keeps its entry point, linking to the same screen filtered to people holding equipment. One place a person is managed, and nobody has to learn that people live under Assets for historical reasons.

⚠️ Whatever moves, the old URL keeps working, and the redirect goes in the code, not in .htaccess — see issue #68 for what that costs when it is done the other way round.

3.13 The directory as source of truth

Agreed in principle, but "source of truth" has to be per-field or it will bite.

If AD owns display_name and an analyst corrects a typo in FreeITSM, the next sync silently reverts it, and they conclude FreeITSM lost their edit.

Decision: directory-owned fields are shown read-only in the UI for directory-managed people, with a note saying where the value comes from. If it cannot be edited, nothing is silently overwritten. Fields FreeITSM owns stay editable.

---

4. Where the first answer was wrong, and what the build changed

The most useful part of this page. §4.1 to §4.3 were caught by a question before anything was written; §4.4 to §4.6 were caught by using the thing, which no amount of design would have found. Every one of the six came from a question rather than from testing, which is worth noting in itself.

4.1 "Two records, unless you match on email" — impossible

The original answer to §3.3 offered a setting: when the same human appears in two sources, either match them by email or create two separate records.

The second option does not exist. users.email carries a UNIQUE index (uq_users_email). A second row with the same address is rejected by the database. The setting as described could never have been implemented, and the schema comment had already anticipated the whole question:

-- NULL because a directory (LDAP) user may genuinely have no mailbox —
-- warehouse and shop-floor staff are never given one (GitHub #47). The
-- UNIQUE index below still applies: MySQL permits many NULLs in a unique
-- index, so any number of mailbox-less people coexist while real addresses
-- stay unique.
--
-- ⚠️ An absent address MUST be stored as NULL, never ''.

🔑 The lesson: check the constraints before designing a setting that depends on them. The database had already made this decision and the design was arguing with it.

4.2 The setting that survived, in better shape

Reframed, the setting is still worth having — it is just a different question:

When sync meets somebody who is already here: ○ Adopt them — attach the directory identity to the existing record (default) ○ Leave them alone and flag it for review

The consequence that makes this a genuine decision: adopting sets auth_provider_id, and from that moment their portal password stops working. Sign-in takes the pinned-provider branch and refuses the local password. That is either exactly right — the directory becomes the source of truth for authentication too — or an unexpected lockout for somebody who never asked for it.

Two constraints regardless of the setting:

• 🔴 Matching is within one company only. Two tenants can legitimately share admin@, or a contractor's address. Cross-company matching would merge two customers' people: a data leak wearing a convenience costume.
• ⚠️ Email matching means a second directory can take over a record created by the first. Fine when both are yours; less fine for an MSP. An argument for the setting existing, not against the default.

4.3 Usernames: a real problem, but not the one it looked like

The concern raised: "if my AD username is smithj, it would be annoying to get a mangled username because another company also has a smithj."

Correct, and worth understanding precisely. username is globally unique (uq_users_username), but sAMAccountName is only unique within one directory.

Why global uniqueness is there, and is not simply a mistake: the multi-company portal routes sign-in by email domain (resolveTenantIdForAddress). Somebody with no email has no domain to route on, so when they type smithj there is nothing to say which company they belong to. Global uniqueness is what makes username sign-in work at all in the absence of company context.

So the blast radius is narrow:

Install	Affected
Single company	Never
Multi-company, everyone has an email	Never — email is legitimately globally unique
Multi-company, mailbox-less staff at two companies sharing a username	Yes

Decision: split the two jobs that one column was doing.

Column	Means	Unique
directory_username (new)	what the directory calls them — smithj, always, faithfully	per provider
username	what they type into the portal	globally, as now

Sync stores smithj in directory_username and never mangles it, so matching and re-linking are unaffected by any collision. username is populated only for people who actually sign in — which most mailbox-less asset holders never do — and on the rare genuine collision FreeITSM reports it rather than silently inventing smithj2.

Explicitly not done now: making username unique per tenant. The proper fix is per-company portal URLs so the tenant is known before sign-in, and that is a multi-tenancy feature deserving its own discussion — not something to smuggle in through an AD import.

---

4.4 A modal was the wrong container, and nobody noticed until it was full

The provider settings had been bolted onto the dialog on the Authentication list page, following the pattern OIDC already used. That was reasonable when a provider was an issuer, a client id and a secret.

By the end of slice 2 an LDAP provider carried a connection, a sign-in scope, group gating, an import scope, attribute mapping, safety thresholds and a run history. The verdict on seeing it:

"a modal is not the right place to configure the connection as there is an enormous amount of information" — and, tellingly, "I couldn't see the preview function".

That second sentence is the real finding. The Preview button was there and had been for the whole slice; it had simply fallen below the fold of a scrolling dialog. A feature that exists and cannot be found has not shipped. Nothing in the design or the tests could have detected that, because both were checking whether it worked.

Decision: LDAP providers get a full-width page of their own, using the shared renderSettingsTabBar() and the same .tabs markup as every module settings screen — so it behaves the way the rest of the application has already taught people to expect. OIDC keeps the modal, because an issuer, a client id and a secret genuinely is a dialog's worth of information. The tab is carried in the URL, so a reload, a bookmark or the back button all land where you were; the modal could not do that at all.

⚠️ Two layout traps followed, both found by measuring rather than reading: a max-width content cap that made a "full width" page not full width, and calc(100vh - 48px) on a page whose header is 58px — which hung the Save bar ten pixels below the fold with no way to scroll to it. The fix is a flex shell (flex: 1; min-height: 0; overflow-y: auto) so nothing has to know how tall the header is.

4.5 "Everyone" must mean everyone

The people screen shipped with a filter whose All option quietly applied is_active = 1. Somebody marked as having left vanished from a list labelled All.

"I marked Alice Johnson as left but she only appears when I filter the list to left/inactive. Should she appear on Everyone?"

A filter that says All and hides people is a contradiction, and the kind that teaches people not to trust the screen. The four scopes were named in that same message and adopted verbatim:

Scope	Includes leavers?	Why it exists
Current people	No	The everyday list
Leavers	Only leavers	Offboarding
Everyone	Yes	Because that is what the word means
Holding equipment	Yes	🔑 Somebody who has left and still has a laptop is the single most useful row on the screen. Excluding leavers here would hide exactly the case worth looking for

The last row is the one worth arguing about, and it is why Holding equipment is a separate scope rather than a checkbox on Current people.

4.6 A reporting line is stored once, pointing upwards

manager_id gives you "who does Bob report to" for free. It does not give you "who reports to Bob" — that direction simply did not exist in the interface, and was noticed within minutes of the field being usable:

"I set Bob Traveller as the manager of Bob Taylor. That setting stuck, but how do I see who someone manages?"

The answer is not another column. assetsForUser() now returns the person's reports alongside their manager, and both are links, so you can walk up and down the org chart from anybody. People who have left appear in that list rather than being hidden, flagged as such — a leaver who still has three direct reports is a fact somebody needs to act on, not one to filter away.

🔑 The general lesson: importing a hierarchy is only half of adopting it. The import populated manager_id correctly and the feature was still missing the half people actually use.

---

5. The slices

Sliced so that stopping early still leaves something useful.

Slice	Contents	Standalone value
1 ✅	Schema (is_active, the person fields, manager_id, directory_username) + the expanded users screen	Useful with no directory at all — and unblocks catalogue approvals, which has been waiting on manager_id
2 ✅	Sync engine: manual run, preview, log, sanity brake. Base DN typed by hand	The actual request answered
3 ◐	OU browser ✅, attribute-mapping interface ✅, preview detail ✅, scheduling still to do	Makes it pleasant instead of merely possible
4	Analysts, if still wanted once users are done	Probably unnecessary — JIT may be the right answer for analysts

Scheduling is the only part of slice 3 outstanding. Everything else in this page is built and tested against the Samba fixture.

What the tests cover

File	Needs the fixture?	Covers
tests/directory-sync-scopes.php	No	The scope arithmetic: what counts as "under" a branch, overlapping ticks, carve-outs, and the upgrade fallback. 19 assertions, runs anywhere
tests/directory-sync.sh	Yes — freeitsm-samba-ad	End to end against the real directory: the brake, deactivation, reporting lines, non-ASCII names, OU selection and the missing-branch warnings. 26 assertions, skips cleanly when the container is not running

⚠️ An end-to-end check that a carve-out worked cannot be a count: excluding the wrong two people satisfies it exactly as well as excluding the right two. The assertions name who must be present and who must not. The first attempt at that control asserted on contractor names that had been invented rather than looked up, passed, and proved nothing.

---

6. The test fixture

docker/ldap-test/seed-ad-people.sh grows the Northwind test directory to ~30 people across 8 OUs, awkward on purpose, because a tidy fixture would let us build a sync that only works on tidy directories:

• OU=Contractors that must not be synced — proves OU selection selects
• two different people with the same display name — matching must not fall back to names
• somebody whose email is another person's username — a near miss
• three accounts with no mail attribute at all
• one person disabled but still in Sales; another disabled and moved to Leavers
• a manager chain three deep, and IT reporting to Finance — a sync assuming managers share a department gets it wrong
• non-ASCII names, an apostrophe, a double-barrelled surname, a deeply nested OU

---

7. Open questions

1. Should adoption clear the local password_hash? Leaving it is harmless but untidy; clearing it is irreversible. Undecided.
2. What happens to a person's assets when they are deactivated? The offboarding report in §3.5 is the obvious answer, but whether deactivation should prompt for return is not settled.
3. Nested groups as a sync filter — the fixture has them (NW-All-Staff contains groups, not people). Sign-in already handles the AD chain matching rule; whether sync should filter by group as well as OU is not decided.
4. Scheduling — sync belongs on the existing scheduled-task machinery, but nothing has been designed about frequency, overlap or what happens when a run takes longer than its interval. The only part of slice 3 not built.
5. OU identity across renames — see §3.9. A renamed branch is now detected and reported rather than silently wrong, but storing each OU's GUID and self-healing would make a rename a non-event. Worth doing; not urgent now that it cannot fail quietly.

The three below came out of auditing this page's own question list against the shipped code. All three are gaps against decisions recorded here, not newly invented ideas — which is the point of keeping the list.

6. 🔴 Precedence between two directories that contain the same person. §3.3 said multiple sources per company were "already possible" and treated the plumbing as the easy part. The plumbing is fine; the policy was never decided. Every run sets auth_provider_id to the provider that is running, so if a human appears in two directories, each run claims them and overwrites the other's values — a job title can flap between runs depending on which ran last. Nobody is duplicated and nothing breaks, but there is no notion of a winner. Options: first-writer-wins, an explicit priority order on the provider, or per-field precedence. Found by reading the code, not by hitting it — a single-directory install will never see this.

7. Custom attributes (§3.7) were not built. The key/value side table for whatever a particular customer's directory happens to carry does not exist; only the fixed columns do. Clean gap, nothing half-built. Worth revisiting when somebody actually asks — the fixed columns cover every field asked for so far.

8. The users screen has one door, not two (§3.12). The decision was that the canonical screen would live under System → Users, with Asset Management keeping an entry point into the same screen. Only the Assets door was built; there is no system/users/. Everything works and nothing is missing functionally — people are simply still managed from under Assets, which §3.12 argued against on the grounds that the module which happened to need people first should not own them. Also note §3.5's offboarding alert ("three people who left this month are holding nine items") is reachable as a filter on that screen but was never built as a proactive report.

---

Related pages

• LDAP and Active Directory — directory sign-in, which exists today
• LDAP — Developer Guide — how the sign-in code works
• Setting up LDAP with Docker — the test directories
• Multi-tenancy: users and self-service — why the portal routes by email domain
• Email send log — the logging pattern §3.10 follows