Ticket importers: per-source conversion modules, run inside time-boxed migration windows

[lennonflima]

Follow-up to #7181, as agreed there: backfilling ticket history is its own feature, built on top of the Partner API ticket write surface (waves 3 and 4) rather than beside it. In practice that means the same ticketService functions and the same service-principal machinery, with no parallel write path.

Problem

An MSP moving its desk to Breeze ticketing wants Breeze to be the single source of truth. That means bringing the history along, not just mirroring what happens from now on. Keeping the old system around for its history means two systems and two places to search, indefinitely.

The everyday write routes are the wrong tool for that, and on purpose: they model live work. A backfill through them:

• emails the requester again for every historic public comment (isPublic: true notifies);
• cannot assign a ticket to an agent who has since left (400 ASSIGNEE_NOT_ELIGIBLE, because the assignee must be an active user);
• dates every ticket and comment "now", since no write accepts a historic timestamp. SLA targets are computed at import time too, so imported open tickets start their clocks then;
• drops attachments, since the Partner API has no attachment route;
• credits everything to the integration's service principal, with no record of who originally wrote each comment.

Those behaviours are right for live work. A backfill needs them paused, for a bounded time and on purpose.

The real case: our Zammad

Numbers from our self-hosted Zammad, as posted on #7181:

Tickets	17,460 over five years (17,116 closed, 326 merged)
Articles	69,663, of which 17,112 are Zammad system messages
Attachments	71,814 files, 3.4 GB
Organizations	17
Final-state backfill	98,439 writes, about 3.4 days at 1200/h

Two lessons from it:

1. A backfill is not a one-time event. On 23–24 Sep we connected our WhatsApp channel, which runs on a separate platform, and about 2,100 tickets arrived within a few hours. That was more than a year after our main migration. Every new source a desk connects is another backfill, so whatever we build has to be repeatable.
2. An importer should convert into the target's model, not the other way round. We moved to Zammad with Zammad's own Freshdesk migrator. Zammad ships migrators for Freshdesk, Zendesk and OTRS, and what makes them work is that each one converts into Zammad's model; Zammad never bends toward the source. That is the shape proposed here.

Proposal

Two pieces, deliberately separable.

1. Migration windows (a platform primitive)

A migration window is an explicit, time-boxed mode for one service principal: the one an import runs as (see below).

• A partner admin opens it with a start, an end, and the organizations it may write to. It is visible in the UI, recorded in the audit log, and closes on its own when it ends.
• Inside it, that principal's ticket writes run in import mode:
  • no notifications: no requester email, no technician alerts or push;
  • historic values are accepted: created, updated, resolved and closed timestamps on tickets, created timestamps on comments, and the final status set directly;
  • historic people are accepted: an agent who has since been deactivated can be the assignee or the author of record, and an author with no Breeze account is kept as a name and an email;
  • no SLA clock for a ticket that arrives already closed;
  • its own throughput limit (the import ceiling discussed on Partner API tickets surface: expose ticketing to PSA/ITSM integrations via partner service principals (tickets:read / tickets:write) #7181), separate from the Partner API's everyday 1200/6000.
• Everything written in a window is marked as imported, with the window's id, so it can be shown, filtered and audited, and, if a run goes wrong, found as a batch.
• Outside a window, every route behaves exactly as it does today. A window is a scoped exception, never a state the platform is left in.

Import mode lives in the service layer, so the public write routes do not change. Whether an external tool may also open a window for its own principal can be decided later, without reworking anything.

2. Import modules (conversion interfaces), inside Breeze

One module per source: Zammad first, then Freshdesk, Zendesk, OTRS, Autotask, ConnectWise, HaloPSA, or a generic CSV/JSON format. A partner admin picks a source in the UI, connects it, reviews a dry run, and starts the import, which runs on a worker. Each module:

• reads the source and maps it onto Breeze's native ticket contract: states, priorities, organizations, requesters, agents and tags. The admin reviews and adjusts that mapping before anything is written, and nothing source-shaped goes past the module;
• runs as its own service principal, created for that import. The machinery built for the Partner API then applies unchanged: attribution to the principal, the audit log, and ticket_external_refs as the record of where each ticket came from, which is the one place foreign identity is allowed to touch the model;
• writes through the same ticketService functions the write routes use, inside a window;
• is resumable: each source id is recorded against what it created (ticket_external_refs for tickets; comments and attachments need the same), so a rerun skips what is already imported;
• reports a dry run first, with counts, unmapped states or agents, and what it will skip.

The contract never learns that Zammad exists. A new source is a new module, never a new branch in ticketService.

Why inside Breeze. It gives the MSP one place to connect a source, review the mapping and watch progress, with nothing extra to install or host. That is the experience Zammad's migrators give. Writes go straight through the service layer, so import mode never becomes public API contract. Storing the source's credentials is not new territory either: Breeze already keeps third-party credentials as a matter of course (PSA, accounting, Pax8, EDR, backup providers, SSO), encrypted through encryptedColumnRegistry and classified for tenant export. A source token is one more credential on an existing path.

The costs are real, and manageable:

• each source's client ships with Breeze and follows its release cycle, so a module needs tests against recorded source responses;
• a long import needs a per-tenant concurrency limit, so one migration cannot starve other tenants' workers.

An external CLI or SDK stays possible later for in-house sources, because the window is a platform primitive rather than something tied to the wizard.

Attachments

The import path should require the object-storage backend. Breeze keeps attachment bytes in S3 when a bucket is configured, but without one it falls back to inline bytea (blobStorage.selectBackend), and a 3.4 GB backfill should not land in Postgres. Our Zammad keeps every file inside its own database, and that is a habit worth not inheriting.

Out of scope

• Continuous two-way sync. That is the mirror built on the wave 3 write routes and wave 4 webhooks, with the origin-principal loop guard.
• Exporting from Breeze.

Open questions

1. Source credentials. Keep them only for the length of the import and its window, or keep them for a later re-import, such as the next new channel?
2. Who can open a window. Partner admins only, or also a dedicated permission for the person running the migration?
3. Historic authors. A display-only "originally by" field, or contacts and users created for those authors?
4. Merged tickets. Breeze has no ticket merge. Do Zammad's 326 merged tickets arrive as closed tickets linked to the surviving one, or are their comments folded into it?
5. System messages. Skip auto-replies and notifications by default (17,112 here), with an option to keep them as internal notes?
6. Ticket numbers. Allocate Breeze numbers and keep the source number only in the external ref, or preserve the source number?
7. Automations and webhooks during a window. Suppress them, or deliver webhooks marked as imported so an existing mirror can decide for itself?

Suggested order

1. Windows and import mode in the service layer: historic timestamps, paused notifications, relaxed live-work checks, the imported marker and the throughput limit.
2. Attachments in import mode, S3 only.
3. The module framework and wizard (connect, dry run, mapping review, run, report), with Zammad as the reference module and our instance as its first real run.
4. More sources, one module each.

If the direction looks right, I'll write the design for step 1 as a spec PR under docs/superpowers/specs/, the way we did for #7181.

[ToddHebebrand]

Direction is right — go ahead with the step-1 spec. Migration windows as a scoped, time-boxed service-layer mode for one service principal, with the public write routes unchanged and every imported row marked with the window id, is exactly the shape I'd want. The two lessons from Zammad (a backfill is repeatable, and a module converts into the target's model) should go into the spec's opening as the design constraints.

My answers to the open questions, as V1 defaults:

1. Credentials: keep them for re-import, stored on the existing encrypted path with a per-source revoke; a later channel is the common case, so throwing them away buys nothing.
2. Who opens a window: partner admins only in V1. A dedicated permission is cheap to add later; a leaked one is not cheap to take back.
3. Historic authors: display-only "originally by" name + email on the comment. No shadow users or contacts; auto-creating contacts from five years of history is how a desk ends up with 4,000 dead contacts.
4. Merged tickets: closed tickets linked to the survivor via the Ticketing: ticket type (incident / request / problem / change) and ticket-to-ticket links #7900 ticket-to-ticket links; never fold comments across tickets.
5. System messages: skip by default, option to keep as internal notes. Matches what the desk actually reads.
6. Ticket numbers: allocate Breeze numbers; source number lives in ticket_external_refs and must be searchable from the ticket search box.
7. Automations and webhooks in a window: suppress both. Webhooks in particular — a mirror receiving 98k imported events it has to classify is the failure mode we built the origin-principal loop guard to avoid. Publish one migration.window_closed summary event with counts when the window ends.

Two constraints for the spec: the window and import marker are partner-axis data (one principal, one partner, many orgs) — follow the partner-axis RLS shape and the cascade registration in CLAUDE.md; and the import worker needs a per-tenant concurrency cap from day one, as you note, so one migration can't starve other tenants.

Status: direction approved; step-1 spec PR under docs/superpowers/specs/ next.

[ToddHebebrand]

Sorry these sat. #8341 merged before I answered §7. All six are confirmed as written:

1. No per-write audit rows in import mode. The opened/closed rows with counts, plus the window id on every imported row, are the trail.
2. No SLA targets on imported open tickets. A clock started at import would breach on day one for anything old.
3. migrationWindowId on Partner API reads and the feed, one-time revision churn accepted. Hiding imported rows from the feed would be worse.
4. One migration.window_closed per org.
5. Keyless import principal while the window is open. Unreachable by construction is the right guarantee.
6. Limits: one open window per partner, 14 days max, 6000 writes/h default (600–20000), all but the first as env settings.

The three changes you made against the plan (no new ticket_source, the keyless principal, and merged tickets imported closed with a note until #7900's links land) are all improvements. Go ahead with the implementation waves.

Status: §7 confirmed; implementation unblocked.

[lennonflima]

Thanks, that settles §7. I'm starting with wave 1a: the two tables with RLS, the window routes with the step-up and the key-route refusal, the sweep, the close with its migration.window_closed event, and the cascade, merge and export registrations. Then 1b, import mode in the service layer. Each wave will be its own PR against main, referencing #8313 and the spec.

[lennonflima]

@ToddHebebrand One addition to wave 1b, now that our own migration plan is settled.

We'll move off Zammad with a cutover done only through the importer, with no live sync:

1. a bulk import a few days ahead;
2. on the day, freeze Zammad and run a delta import on the same principal;
3. switch.

The spec already covers most of the delta. New tickets are created, and a resumed ticket gets its missing comments and its final status. What it misses is a header field that changed between the two passes, such as the assignee, priority, subject, category or tags. §4.6 resumes only comments and status, so those changes would be lost.

Proposal: on resume, the writer also applies header changes. When a resumed ticket's converted header differs from the stored one, the writer applies the difference through updateTicketFields and assignTicket in import mode, at the source's updatedAt:

• It uses the same mode and the same functions. There are no new side effects, and the writes are charged to the window's bucket like any other.
• A field a Breeze user changed after the import keeps the Breeze value. The writer skips any field whose field_provenance is user.
• The assignee follows the same relaxed import eligibility as on create.

This fits the first design constraint: a backfill is repeatable, so a re-import should converge on the source, not only append to it. I'm including it in the 1b PR, flagged as a deviation from §4.6. If you'd rather keep 1b to the spec, I'll split it into a follow-up.

[ToddHebebrand]

Header convergence on resume: agreed, and it's fine in 1b. The design satisfies the repeatable-backfill constraint, uses the same functions and import mode, and the field_provenance = user skip is the right tiebreak. I've approved and enqueued #8782 with it included.

Gaps that remain for the bulk-then-delta cutover, based on what's in #8782 now:

• Status on resume still overwrites a Breeze user's change with the source's final status. Please apply the same Breeze-wins rule, keyed on the latest status_change row (Q1 on the PR).
• Comments edited or deleted in Zammad between the two passes are not updated, because dedupe is (origin_principal_id, import_source_id) and the writer only appends.
• A requester change between passes is not re-applied.
• Tickets deleted in the source aren't handled. That belongs in the step-3 Zammad module, if you care about it.
• Re-runs only line up if Zammad comment ids are unique across the whole source, not just per ticket. Worth a sentence in the module docs.

With Zammad frozen on the day, none of these block your cutover. The status rule is the only one I'd fix before you rely on a delta pass.

[lennonflima]

@ToddHebebrand The spec for steps 2 and 3 is ready: docs/superpowers/specs/api-platform/2026-10-19-ticket-import-modules-design.md (#9295). It covers attachments in import mode, the module framework with its worker and wizard, and Zammad 7.2.2 as the first module, with a runbook for our import-only cutover: bulk import a few days ahead, freeze Zammad, run a delta pass on the same principal, then switch.

First, a request: revisit answer 3 for imports. Your answer 3 was: "Historic authors: display-only "originally by" name + email on the comment. No shadow users or contacts; auto-creating contacts from five years of history is how a desk ends up with 4,000 dead contacts."

For an import-only cutover, Breeze becomes the single source of truth, and we'd like the history attached to real people:

• per-agent history and reporting on the tickets they owned;
• the requester's history on their contact, and portal access for them later.

So the spec adds two source options. Both are off by default, so answer 3 stays the product default. Comment authors stay display-only either way.

• Decision 24: "Create missing owners as disabled users".
  • An unmatched ticket owner with an email becomes a users row that is disabled, has no password and no partner_users membership or role, and gets no mail. That is the same shape a removed member already leaves.
  • On main it cannot sign in, reset a password, accept an invite or get an SSO session. It is not in GET /users or any assignee picker, and isEligibleTicketRecipient keeps it out of every notification. Import eligibility accepts it.
  • An admin activates a person later with the normal invite of the same email, which keeps the id, so their tickets stay assigned.
  • An email held by another tenant is never used; the ticket keeps the owner note.
  • A ledger lists "users created by this import". Deleting the source leaves them as ordinary disabled users.
• Decision 25: "Create missing requester contacts". To address the "4,000 dead contacts" concern:
  • Only requesters of tickets actually imported get a contact, never every Zammad customer, and never a customer without an email. An optional "updated since" date narrows it further.
  • Deduplication. Contacts are deduped by (org, email) under the inbound pipeline's own advisory lock, and across passes through contact_external_links. Live inbound mail already find-or-creates a contact per unknown sender, so this matches live behaviour.
  • Visible before any run. The dry run shows the count per org.
  • Removable as a set. Removal goes through the normal contact delete. Tickets fall back to display-only, because createTicket already keeps the name and email snapshot and the FK nulls only the contact id. I chose that over refusing removal, since every created contact is referenced.
  • No side effects: no portal access, no mail, no events.
  • No per-contact audit row: the link row records each creation, and the run's audit row carries the count. A row per contact would be the per-write audit step 1 declined.

The rest of the spec

• Attachments.
  • Object storage only.
  • Any file type, but anything outside JPEG/PNG/WebP/PDF is stored as application/octet-stream, so the existing routes serve it as a download with nosniff.
  • The 10 MiB cap stays.
  • The row is reserved before the bytes are put, so a crash leaves a pending row the existing reaper cleans up.
• Framework.
  • A module contract with a pure, fixture-tested conversion.
  • Seven partner-axis tables.
  • A worker in 5-minute slices under a renewable per-partner lease with a global cap.
  • A wizard at /tickets/import.
• Your 2026-10-14 gaps.
  • Comment edits and deletes and requester changes between passes are detected and reported, not applied, in v1.
  • Source deletions are reported (404 deleted, 403 hidden); nothing is deleted in Breeze.
  • Comment-id uniqueness is a module contract.
• Your fix(api): ticket import — status on resume, chunk byte bound, window lookup (#8782 follow-ups, #8313) #8884 findings. All eight are folded into the first wave.
• Merged tickets are linked duplicate_of their final survivor in import mode, with the note as the fallback.
• Three corrections to step 1's §8 (one-line note added there):
  • the cap is a renewable lease, not reviewConcurrency.ts;
  • the wizard, not the module, opens windows;
  • merged tickets get the link.

Decisions to confirm (spec §9 has the alternatives):

• 24 and 25 above: revisiting answer 3 for imports, opt-in.
• 1. No malware scanning on imported files. Download-only serving stops the browser, not a technician who opens an .exe. Your call.
• 2–5. Attachments: reserve before storing; buffered 10 MiB downloads; inline images as attachments with [image: name] placeholders; one parametrized HTML-to-text converter.
• 6–9. Mapping: groups to categories (default none); an org map plus one fallback; owners by email; requesters by an exact single contact match.
• 10. Mapping tables: pending close maps to resolved.
• 11. Merges: closed plus duplicate_of, and a note always.
• 12–14. Delta: the list plus a ledger, using by_ticket not all=true; deletions and between-pass edits are report-only.
• 15–16. Not imported as data: time accounting becomes one note; custom attributes are only counted.
• 17. Concurrency: a renewable per-partner lease plus a global cap.
• 18. Credentials: revoke clears the credential but keeps the source, because a new principal would duplicate every ticket.
• 19. SSRF: strict mode when hosted, and no URL followed from a response.
• 20. Principal: one keyless principal per source, owned by the connecting admin.
• 21. Throughput: one window unit per file plus a byte bucket.
• 22. Page: the wizard at /tickets/import.
• 23. Event: attachmentsImported on migration.window_closed.

One risk outside this spec: only a principal's owner can open its windows, and there is no ownership transfer. If the admin who connected a source leaves, the import can't continue on that principal. That seems like a #8332 follow-up.