CardDAV Import Internals

CardDAV import internals

← Back to CardDAV contact sync — developer guide, which covers the transport and the vCard mapping.

This page is the half that writes to the database: how scope is resolved, how a run is bookkept, what the policy layer does for free, and the two bugs that only showed up when the documented scheduled-task command was actually run.

---

1. 🔴 Group scope needs two passes

A KIND:group card lists its members by UID. So you cannot decide whether an ordinary card is in scope until you have read the group card — and filtering in a single pass silently imports nobody when the group card happens to sort after its members, which alphabetically it usually does.

cdsyncResolveScope() therefore resolves the whole scope up front and returns a set, before anything is imported:

/**
 * @return array ['uids' => set of in-scope UIDs, 'emails' => set of in-scope emails]
 *               or null when the scope is 'all' and everything is in scope.
 */
function cdsyncResolveScope(array $cards, string $scope, array $wanted): ?array

Three modes, and the two filtered ones work differently:

carddav_scope	Passes	How
all	—	returns null; cdsyncInScope() short-circuits to true
category	one	a tag lives on each card, so one pass is enough
group	two	find the chosen group cards, then collect their MEMBERs

⚠️ MEMBER values are URIs, and clients disagree about which kind

All three of these are legal, and different clients write different ones:

MEMBER:urn:uuid:alice-1234
MEMBER:mailto:alice@example.com
MEMBER:alice-1234

So the scheme is stripped and both the UID and the email are accepted as ways of naming a member — which is why the resolved scope is two sets rather than one:

if (stripos($member, 'mailto:') === 0) {
    $emails[mb_strtolower(substr($member, 7))] = true;
    continue;
}
$pos = strrpos($member, ':');                  // urn:uuid:xxx, or a bare UID
$uidPart = $pos === false ? $member : substr($member, $pos + 1);

Groups match on UID or name

$isWanted = in_array(mb_strtolower($uid), $wantedLower, true)
         || in_array(mb_strtolower($name), $wantedLower, true);

The picker stores the UID, which is the stable identifier — but a group deleted and recreated on the server gets a new UID and keeps its name. Accepting either means that case survives instead of silently importing nobody.

Category comparison is case-insensitive for the same class of reason: clients do not agree on case, and ITSM and itsm are the same tag to every one of them.

---

2. The two things the settings screen needs from the server

cardDavScanBook() reads every card once and reports what the book can be filtered by, so the UI offers a list of tick boxes instead of a text box the operator has to guess into:

$out = [
    'contacts'   => 0,    // ordinary cards — what "bring in everything" would import
    'groups'     => [],   // KIND:group cards, each with its MEMBER count
    'categories' => [],   // every distinct CATEGORIES value, with a count
];

🔑 Both are always reported, never one or the other. A server can use either or both, the two conventions come from different client families — KIND:group is how Apple Contacts stores a group, CATEGORIES is how Thunderbird and many Android clients do it — and the operator is the only one who knows which of their groups is the real one.

Categories come back most-used first (arsort), because the tag somebody wants is far more likely to be the one on eighty cards than the one on a stray two.

This is also why the original request — "scope it to one group called itsm" — was answered with an enumerated picker rather than a text field. Asking the server what it has is both more flexible and less to explain than asking the operator to type a name correctly.

---

3. The run, and what comes for free

cardDavSyncRun($conn, $provider, $mode, $analystId) mirrors directorySyncRun() and shares its bookkeeping table, its counts and every one of its safety rules. $mode is 'live' or 'preview'; preview runs the identical code path with the writes suppressed.

INSERT directory_sync_runs (status 'running')
        ↓
  address book chosen?                     → throw
  cardDavFetchCards()                      → throw on transport failure
  cap at CDSYNC_MAX_CONTACTS
        ↓
  cdsyncResolveScope()
  scope resolved to NOTHING?               → throw  (see below)
        ↓
  map every card, drop groups and no-UID cards, filter by scope
  counts['seen'] = count($people)
        ↓
  syncBrakeTripped()                       → 'stopped', change nothing
        ↓
  ── the policy layer, reused verbatim ──
  foreach person:
      dsyncFindExisting()   → [$existing, $how]
      conflict + flag mode? → count, log, skip
      existing?             → dsyncApplyToExisting()   (adopted | updated | unchanged)
      otherwise             → dsyncCreate()
        ↓
  dsyncHandleMissing()                     → rules 1 and 3
  UPDATE auth_providers SET sync_last_count (live only — this is the brake's baseline)
        ↓
  dsyncFinishRun(); return dsyncGetRun()

⚠️ An empty scope must stop the run

if ($resolved !== null && !$resolved['uids'] && !$resolved['emails']) {
    throw new RuntimeException('Stopped without changing anything: nothing in that
        address book matches the chosen group. That is far more often a group or tag
        renamed on the server than everybody leaving it, so no contact has been touched.');
}

Without this the run imports nobody, and then dsyncHandleMissing() correctly observes that every managed contact was absent and starts counting them towards deactivation. A renamed group would begin quietly retiring your customer list. Same reasoning as the sanity brake, and it has to be a separate check because the brake only compares counts against a previous run.

The safety rules, and their real defaults

Rule	Where	Default
Nobody is ever deleted	dsyncHandleMissing() deactivates, never deletes	—
A run that looks wrong changes nothing	syncBrakeTripped()	sync_brake_percent 20; 0 disables; a first run (no baseline) is never braked
Missing once is noise	sync_missed_count increments	sync_deactivate_after 3; 0 means never deactivate automatically

---

4. 🔴 The counts bug: right data, zero counted

// ⚠️ `dsyncApplyToExisting()` returns a DESCRIPTION OF THE CHANGES, not an
// action keyword. Written as `if ($action === 'updated')` this loop compiled,
// ran, imported everybody correctly — and counted nothing, because no branch
// ever matched.

The import was completely correct. Every job title, department and office was written. And the run summary and the History tab both said "0 updated", because the return value was being compared against words it never contains.

🔑 Found by changing a card on the server and reading the counts — not by reading the code. A function that returns "job title, office" and one that returns "updated" are indistinguishable until you look at a number that should not be zero. It also put the changes string into the log rows' action column, which is where it first became visible.

Whether somebody was adopted rather than updated comes from $how, which says which rung of dsyncFindExisting() matched:

if ($how === 'email') {
    $counts['adopted']++;
    // ⚠️ Their contact details are now maintained in the address book, so they
    // can no longer be edited in FreeITSM — and their portal password stops
    // working. That deserves its own word in the log, not "updated".
}

---

5. 🔴 The return shape, and why the exit code depended on it

cardDavSyncRun() originally returned a hand-built ['status' => …, 'counts' => […]]. directorySyncRun() returns the run's database row via dsyncGetRun(). It also used the words 'error' and 'refused' where the LDAP path uses 'failed' and 'stopped'.

Two things broke silently, and neither was visible from the screen:

• scripts/directory_sync.php prints the flat column names — $run['seen_count'] and friends — so every number the CLI reported was 0 while the message printed beside it said "Contacts read: 2. Created: 2." A scheduled task's own output contradicting itself.
• Worse: the script sets its exit code from those two words. With 'error' and 'refused' it matched neither, so a CardDAV import that failed exited 0 — and anything monitoring that task would have reported a healthy nightly import of nobody.

// 🔴 Return the run ROW, exactly as directorySyncRun() does, and use its
// status words — 'ok' | 'stopped' | 'failed'.
return dsyncGetRun($conn, $runId);

One shape and one vocabulary means the CLI, the web endpoint, the page and the History tab treat both kinds of source identically for free.

Verified by running it: exit 1 on failure, 2 on a stopped run, 0 when healthy.

⚠️ And the first attempt to verify that was itself wrong. php … | tail reports tail's exit code, not PHP's, so everything looked like 0 and it nearly sent me chasing a bug that was already fixed.

---

6. Routing: two protocols that import, three that exist

auth_providers.protocol is 'oidc' | 'ldap' | 'carddav'. Only two of the three import people, and every place that assumed a binary had to be found:

// scripts/directory_sync.php --all
AND protocol IN ('ldap','carddav')

🔴 --all previously selected on enabled and sync_enabled with no protocol filter. That was a complete question while exactly one protocol could import, and wrong the moment a second could — it would have handed an address book to the LDAP engine. An OIDC id asked for by name is now refused with a sentence rather than a connection error.

🔴 Three protocols broke four isLdap ? … : … ternaries in api/system/save_sso_provider.php. A two-way conditional reads as exhaustive and compiles fine; it would have stored every CardDAV provider as OIDC. Anywhere a protocol is branched on, grep for the binary form.

---

7. Scheduling

There is no internal scheduler. The operator sets up a task:

php scripts/directory_sync.php --all

It picks up every enabled source with importing switched on, address books and directories alike, and routes each by protocol. Documented on Scheduled tasks — where, until 1.8.0, the LDAP job had never been listed either, so no operator could learn from the documentation that any of this could be automated.

---

8. 🐳 The test rig

docker/ldap-test/ runs Samba AD and OpenLDAP side by side precisely so a new LDAP flavour cannot be added blind. CalDAV and CardDAV §5 predicted a CardDAV source would need its own equivalent "or it ships untested against a real server". It does, and it got one.

docker/carddav-test/ — Baïkal 0.12.1, php:8.3-apache, port 8092:

docker compose up -d        # then complete the web installer once
./seed.sh                   # two address books, five contacts, a group and tags

Notes worth keeping:

• The image installs the official release zip, pinned by sha256 (0449abb72b151d39d9c08c63cb83a05d9e9adb065b1165ef6786b0b6a13d203c). ⚠️ sabre/dav itself is a library, not a server — you cannot composer require your way to something that answers PROPFIND. That is why the fixture is Baïkal.
• Leave dav_auth_type at its default of Digest. It is tempting to switch it to Basic to make the fixture simpler, and doing so would have hidden the single most likely real-world failure.
• The seeded data deliberately includes a KIND:group card, CATEGORIES tags, a folded CATEGORIES line, and a card with no UID — one per trap on this page and the transport page.

Testing against real data

🔴 The dev database is Ed's real data. Capture the row ids first, scope every write to them, and restore afterwards — verified back to 71 users, 0 runs, 0 identities after each round.

And the lesson that cost two rounds: assert that the response parses, not that it reads correctly.

A PHP warning printed before the JSON breaks every caller's JSON.parse while the body still reads perfectly to a human. It happened twice in one day — once from in_array($data['x'] ?? 'auto', […]) ? $data['x'] : 'auto', which reads the key twice, and once from the carddav branch of save_sso_provider.php never setting $secretInput. A test that greps the body for the right words passes both times.

🔴 And a third time, shipped (#133, #1722). The read-it-twice fix was applied to carddav_auth, and the identical line for carddav_scope, directly below it, was left alone. It stayed harmless for exactly as long as the Add dialog sent a scope, which was twelve minutes; after the picker moved to carddav.php, nobody could add an address book from 1.8.0 to 2.0.0. When you fix a pattern, grep for its twins.

The same report exposed #1723: three screens save a provider and none sends every setting, so the endpoint now follows the house rule from Contact details - on an update, absent means don't touch - for every setting the protocol reads from the request.

---

See also

• CardDAV contact sync — developer guide — transport, auth and vCard parsing
• CardDAV write-back internals — the other direction. ⚠️ The import records source_ref and source_etag on every sighting for the write-back; if you change what dsyncLinkIdentity() stores, read that page first.
• CardDAV contact sync — the administrator's view
• Directory sync developer guide — the policy layer in full
• CalDAV and CardDAV — §5 priced this work before it was done, and §10 records what was expected to move the write-back
• Scheduled tasks — automating the import
• Contact details — the seven fields and who owns each one