Skip to content

Zammad Repository Connector

Piergiorgio Lucidi edited this page Nov 20, 2026 · 3 revisions

Zammad Helpdesk & Knowledge Base Repository Connector

Zammad Java Spring Boot OIS Compliance Claim Check E2E Verified

Module: oc-zammad-repository-connector
Connector Class: org.opencrawling.zammad.ZammadRepositoryConnector
Client Class: org.opencrawling.zammad.ZammadClient
Document Builder: org.opencrawling.zammad.ZammadDocumentBuilder
Security Mapper: org.opencrawling.zammad.ZammadSecurityMapper
Properties Class: org.opencrawling.zammad.ZammadProperties
Crawl State: org.opencrawling.zammad.ZammadCrawlState
Discovery SPI: DiscoverableRepositoryConnector, DocumentFetchingRepositoryConnector
Spring Bean: @Component (activated when spring.opencrawling.connector.type=zammad or SPRING_OPENCRAWLING_CONNECTOR_TYPE=zammad)
Issues / PRs: #342 (PR #372); #411, #412, #413, #414, #415 (PR #417)


📌 Overview

The Zammad Repository Connector (oc-zammad-repository-connector) crawls Zammad Helpdesk & ITSM instances (self-hosted and Zammad Cloud) via the official Zammad REST API v1 (/api/v1/...), streaming support tickets, public customer conversations, confidential internal agent notes, full role-annotated ticket conversations, binary ticket attachments, ticket tags, and localized Knowledge Base answers into OpenCrawling as Open Ingestion Standard (OIS) documents.

The connector supports two complementary helpdesk indexing models with fail-closed Zero-Trust enforcement:

  • Public vs. Internal Isolation (SPLIT_PUBLIC_AND_INTERNAL, default): emits customer-visible ticket articles (internal: false) and internal agent notes (internal: true) as separate OIS documents with independent SecurityConfig ACLs, preventing confidential agent triage notes and internal attachments from ever leaking into customer-facing or cross-organization RAG answers.
  • Full Ticket Conversations (TICKET): emits a single chronologically ordered, role-annotated document per ticket (zammad://<instance>/tickets/<id>) combining customer replies and internal notes for internal support/agent RAG, locked strictly to the ticket's owning group (group:zammad-group-<group>).

Built-in article filtering (excluding system triggers and automatic replies by default with content-hash deduplication), ticket selection filters (dates/periods, tags, title regexes, priorities, organizations, and custom base-query), privacy controls (privacy-profile: strict, identity rendering, ID-based customer ACLs, participant metadata for governance hooks), and periodic full scans inside incremental jobs make the connector production-ready for large enterprise helpdesks.

Note

Real-Stack Verification: End-to-end verified in CI (Zammad Decoupled Pipeline Test Suite) against the official Zammad 7.2.3-0008 + Elasticsearch 9.5.5 container stack (PostgreSQL, Redis, Memcached, Zammad Rails/Scheduler/WebSocket) wired into the full OpenCrawling decoupled pipeline (Kafka, Tika 4 ForkParser, Ollama mxbai-embed-large, pgvector vector_store_1024, and the Secure MCP Server), plus a fast standalone suite against the built-in Python REST API v1 emulator (zammad-mock).


🏗️ Architecture & Flow

graph TD
    subgraph Zammad["Zammad Helpdesk & ITSM (REST API v1)"]
        API["REST API v1 (/api/v1/...)"]
        Tickets["Tickets & Articles (public vs internal)"]
        Atts["Ticket Binary Attachments"]
        Tags["Ticket Tags & Lookups (Groups, States, Priorities, Roles, Users, Orgs)"]
        KB["Knowledge Base (POST /knowledge_bases/init & /answers/{id}?include_contents=...)"]
        ES["Elasticsearch Search Index (base-query + updated_at:[watermark TO *])"]

        Tickets & Atts & Tags & KB --> API
        ES --> API
    end

    subgraph OpenCrawling["OpenCrawling Crawler (Java 25)"]
        Client["ZammadClient (Token / OAuth2 / Basic / X-On-Behalf-Of, Jittered Retries, Page Caps)"]
        Connector["ZammadRepositoryConnector (SEARCH_DELTA / FULL / Periodic Full Scan, Filters, Explore, DryRun, FetchDocument)"]
        CrawlState["ZammadCrawlState (Scope-v2 Key, Watermark, Render Signatures, Content Hashes, lastFullScanAt)"]
        VThreads["Virtual Threads (StructuredTaskScope + ObservabilityTask, bounded parallelism)"]
        DocBld["ZammadDocumentBuilder (Ticket Conversations, Split Public/Internal, Per-Article, Roles, Attachments, KB)"]
        SecMap["ZammadSecurityMapper (Group, Customer Email/ID, Shared Org, Privacy Profiles, KB Roles, Fail-Closed)"]
        TempStream["TempFileInputStream (Size-Capped Attachment Stream, Auto-Deleted)"]

        API <-->|"HTTPS / JSON + Binary"| Client
        Client --> Connector
        Connector <--> CrawlState
        Connector --> VThreads
        VThreads --> DocBld
        VThreads --> SecMap
        VThreads --> TempStream

        OIS["OIS Document (UPSERT / DELETE Tombstone)"]
        DocBld --> OIS
        SecMap --> OIS
        TempStream --> OIS
    end

    OIS --> ClaimCheck["Claim-Check Store (Local / Apache Ozone)"]
    ClaimCheck --> Kafka["Kafka (opencrawling-documents -> Tika + Governance Gate + drop-metadata-after-gate + chunking -> opencrawling-chunks -> embedding -> opencrawling-embedded)"]
    Kafka --> Sinks[("Vector Stores: PGVector / Milvus / Qdrant / Solr / Vespa / OpenSearch / Elasticsearch / Luxir / OpenArch / Iceberg")]
Loading

Incremental SEARCH_DELTA, Periodic Full Scan & Reconciliation Flow

sequenceDiagram
    participant C as ZammadRepositoryConnector
    participant Z as Zammad REST API v1
    participant S as ZammadCrawlState (JSON)
    participant P as OpenCrawling Pipeline (Kafka)

    C->>Z: GET /api/v1/users/me (captures HTTP Date server clock)
    C->>Z: Paginated GET /groups, /ticket_states, /ticket_priorities, /roles
    C->>S: Load scope-v2 state (auto-migrating legacy file), watermark, lastFullScanAt, known tickets & KB
    alt SEARCH_DELTA with watermark and periodic full scan NOT due
        loop page = 1..N (per_page <= 200)
            C->>Z: GET /api/v1/tickets/search?query=([base-query AND ]updated_at:[<watermark> TO *])&sort_by=updated_at&order_by=asc
        end
    else FULL, initial crawl, or periodic full scan due (interval + window)
        alt base-query configured
            loop page = 1..N (per_page <= 200)
                C->>Z: GET /api/v1/tickets/search?query=<base-query>&sort_by=updated_at&order_by=asc
            end
        else no base-query
            loop page = 1..N (per_page <= 100)
                C->>Z: GET /api/v1/tickets?page=...&per_page=...&expand=true
            end
        end
    end
    par bounded parallelism (Virtual Threads)
        alt ticket fails selection filters (group, state, created-after, priority, org, title, tags)
            C->>P: Emit DELETE tombstones if previously indexed (otherwise skip)
        else updated_at, ACL fingerprint, and renderSignature unchanged
            C->>C: Skip unchanged ticket
        else new or changed
            C->>Z: GET /api/v1/ticket_articles/by_ticket/{id}, /tags, /users/{id}, /organizations/{id}
            C->>C: Filter system/excluded senders, excluded types, auto-replies, then render roles & identities
            C->>P: Emit OIS UPSERTs only when SHA-256 content hash changed and reuse unchanged attachments
            C->>P: Emit OIS DELETE tombstones for removed docs / previous granularity IDs / attachments
        end
    end
    opt Full enumeration (FULL, initial, or periodic full scan)
        alt seenTicketIds is non-empty
            C->>P: Emit OIS DELETE tombstones for vanished / hard-deleted tickets
        else 0 tickets returned while tickets are indexed
            C->>C: Log warning and emit NO tombstones (empty-enumeration guard)
        end
    end
    opt include-knowledge-base = true
        C->>Z: POST /api/v1/knowledge_bases/init (GET fallback on 404/405)
        C->>Z: GET /api/v1/knowledge_bases/{kbId}/answers/{answerId}?include_contents=<ids>
        C->>P: Emit OIS UPSERTs (active answers) and DELETE tombstones (archived/deleted)
    end
    C->>S: Atomically save state, advance watermark (serverDate - 60s) & lastFullScanAt only if 0 failures & not cancelled
Loading

✅ Prerequisites & Authentication

  1. A running Zammad instance (Zammad 6.x / 7.x self-hosted or Zammad Cloud) reachable over HTTP(S) from OpenCrawling.
  2. Elasticsearch enabled on Zammad for crawl-mode: search_delta or when base-query is configured (standard in Zammad deployments; crawl-mode: full without base-query works without search).
  3. API credentials for a Zammad agent/service account with read access to the target groups, tickets, articles, attachments, organizations, users, and Knowledge Base:
    • token (recommended): Personal HTTP Access Token (Authorization: Token token=<accessToken>), created in Zammad under Profile → Token Access.
    • oauth2: OAuth2 Bearer Token (Authorization: Bearer <accessToken>).
    • basic: HTTP Basic authentication (username + password).
    • Optional on-behalf-of: Sends the X-On-Behalf-Of header to delegate requests on behalf of another Zammad user login or email.

🚀 Key Features

  • Flexible Ticket Granularity & Machine-Readable Roles (article-granularity, #342, #411):
    • SPLIT_PUBLIC_AND_INTERNAL (default): Emits separate OIS HTML documents for customer-visible articles (zammad://<instance>/tickets/<id>/public) and internal agent notes (zammad://<instance>/tickets/<id>/internal). Tickets containing only internal notes emit only the /internal document (never an empty /public document).
    • TICKET (alias CONVERSATION): Emits one OIS document per ticket (zammad://<instance>/tickets/<id>) containing every included article in chronological order—both public replies and internal agent notes—with metadata zammad.entityType=ticket_conversation, zammad.articleCount, and zammad.hasInternalNotes. Because internal notes are included, ticket-level conversations and their attachments are always readable by the ticket's group only (group:zammad-group-<group>, fail-closed).
    • PER_ARTICLE: Emits one OIS document per article (zammad://<instance>/tickets/<id>/articles/<articleId>), carrying zammad.articleRole and inheriting either public or agent-only ACLs depending on article.internal.
    • PUBLIC_ONLY: Crawls only customer-visible articles (internal: false) and their attachments.
    • Semantic Role Markup & Localizable Labels: Every rendered <section> carries data-article-id, data-internal, data-role (customer, agent, internal, system), data-visibility (public / internal), data-sender, data-type, data-author, and data-created. Article headings use [<roleLabel>] (default Customer, Agent, Internal note, System), customizable per deployment with article-role-labels (e.g. customer=Kunde,agent=Mitarbeiter,internal=Interne Notiz,system=System).
  • Article Filtering & Content-Hash Deduplication (#412):
    • Leaves out noisy system articles (trigger notifications, SLA escalations, reminders) and automatic replies by default (excluded-article-senders: System, skip-auto-replies: true—matching Zammad's preferences.is-auto-response flag as well as mailer-daemon@ and postmaster@ delivery bounces), plus optional excluded-article-types (e.g. sms, twitter status).
    • Stores a SHA-256 content hash per rendered document (covering content bytes, non-volatile metadata excluding zammad.updatedAt, and ACL) in ZammadCrawlState. When a ticket is updated only by a filtered system article or auto-reply—or when only its internal thread changed—unchanged rendered documents are not re-indexed.
    • Immutable binary attachments already indexed under the same ACL fingerprint and render signature are retained without re-downloading from Zammad.
  • Ticket Selection Filters & Empty-Enumeration Guard (#413):
    • Narrow crawled tickets with created-after (ISO date 2024-01-01, instant 2024-01-01T00:00:00Z, or relative period/duration such as P2Y, P90D, PT36H), included-tags, excluded-tags (e.g. spam, test, duplicate), excluded-title-patterns (regular expressions), priorities, organizations, and base-query (Zammad search syntax combined with the delta watermark as (<base-query>) AND updated_at:[<watermark> TO *]).
    • Fully integrated with incremental state and tombstones: a ticket that stops matching (for example, tagged spam after indexing) emits DocumentAction.DELETE tombstones on the next scan. Discovery (explore), dryRun, and fetchDocument apply the exact same filters.
    • Empty-enumeration safety guard: If a full ticket enumeration returns 0 tickets while tickets are already indexed in state, the connector logs a warning and emits no vanished-ticket tombstones, protecting against accidental mass purges from revoked credentials or a broken Zammad search index.
  • Participant Metadata & Privacy Controls (#414):
    • identity-rendering (email default, name, id, none) controls how people appear in rendered ticket headers, article headings (data-author), and metadata (zammad.customer, zammad.owner, zammad.from). When name is chosen, it never falls back to an e-mail address (falling back to zammad-user-<id> instead).
    • customer-identity (email default, or id → zammad-user-<id>) keeps customer e-mails out of OIS permission rules while still enabling per-customer ACL enforcement.
    • participant-metadata (names default, names-and-ids, none) populates zammad.participants (and zammad.participantIds) from the customer, owner, and article authors so downstream governance/anonymization hooks can redact personal names; pair with opencrawling.governance.drop-metadata-after-gate=zammad.participants,zammad.participantIds in oc-runtime so those keys are never persisted in vector stores (see Configuration Guide — Pre-Embedding Governance).
    • privacy-profile: strict enforces zero e-mail addresses across rendered text, metadata, and permission rules (upgrades email rendering to name, forces customer-identity: id, and disables customer and shared-organization principals on public threads).
    • Privacy-by-default for new Admin UI definitions: Built-in Zammad templates created via the Admin UI start privacy-preserving (identityRendering: name, customerIdentity: id, includeCustomerInPublicAcl: false, includeSharedOrganizationInPublicAcl: false), while saved definitions missing these keys retain backward-compatible connector defaults.
  • Periodic Full Scans Inside Incremental Jobs (#415):
    • Configure full-scan-interval (ISO-8601 duration, e.g. P1D)—plus optional full-scan-window (01:00-05:00, may wrap midnight such as 22:00-04:00) and full-scan-timezone (default UTC)—on a crawl-mode: search_delta job so one run per interval automatically executes a full enumeration to detect and tombstone hard-deleted tickets (and tickets that left base-query), without scheduling a separate full job. ZammadCrawlState records lastFullScanAt and the scan summary log states whether enumeration was incremental, full, or a periodic full scan.
  • Scope-Keyed Crawl State & Seamless Granularity Switching:
    • ZammadCrawlState files are keyed by the crawl scope (base-url, group-names, ticket-states, include-tickets, include-knowledge-base) rather than rendering options. Switching article-granularity, article filters, identity rendering, or role labels updates renderSignature under the same state file, re-rendering tickets once and tombstoning obsolete document IDs (e.g., /public and /internal when moving to TICKET). Legacy state files written by earlier releases are automatically read and migrated on the first run.
  • Binary Ticket Attachments & Synthetic Body Deduplication:
    • Streams article attachments via GET /api/v1/ticket_attachment/{ticketId}/{articleId}/{attachmentId} into size-capped TempFileInputStream files (max-attachment-size-bytes, default 50 MiB), automatically deleting temporary files on close or cancellation.
    • Skips Zammad's synthetic content-alternative: true HTML email body attachments so article content is never indexed twice.
    • Attachment ACLs strictly inherit the parent document's visibility (internal article attachments and TICKET-granularity attachments are strictly group-only).
  • Zammad 7.x Knowledge Base Ingestion & Scheduled Publishing:
    • Loads KB structure via POST /api/v1/knowledge_bases/init (with automatic GET fallback on HTTP 404/405) and fetches localized translation contents via GET /api/v1/knowledge_bases/{kbId}/answers/{answerId}?include_contents=<comma-joined-content-ids>.
    • Resolves KnowledgeBaseLocale.system_locale_id against Locale assets and reads Zammad's computed permissions_effective (falling back to KB-level permissions_effective or inline permissions).
    • Evaluates published_at, internal_at, and archived_at against Zammad's server clock (CanBePublished::StateMachine), so drafts scheduled to publish in the future are never prematurely exposed and answers scheduled to archive in the future are not prematurely tombstoned.
  • Hardened Pagination, Rate-Limit Resilience & Cooperative Cancellation:
    • Enforces Zammad's real server pagination caps (max 100 on /api/v1/tickets, max 200 on /api/v1/tickets/search) and paginates metadata lookups (/groups, /roles, /ticket_states, /ticket_priorities), stopping ticket pagination only on empty or duplicate pages to prevent silent truncation and false tombstones.
    • Retries HTTP 429, 502, 503, 504 and transient IOExceptions honoring Retry-After (both integer seconds and RFC 1123 HTTP-date) with jittered exponential backoff.
    • Closes owned HttpClient pools on disconnect(), clears per-scan user/organization lookup caches on connect(), and honors downstream Flux cancellation/interruption without advancing incremental state.
  • Pre-Crawl Discovery & On-Demand Fetching:
    • Implements DiscoverableRepositoryConnector (explore, dryRun, crawlStart) and DocumentFetchingRepositoryConnector (fetchDocument) with page-by-page VisibilityFilter and selection-filter enforcement (supporting group:<id> and /Groups/<GroupName> references, <ticketId>:public|internal, zammad://.../tickets/<id>/public|internal, and under TICKET granularity ticket:<id> / zammad://.../tickets/<id>).

⚙️ Configuration Properties

Spring properties bind under spring.opencrawling.connector.zammad.* (kebab-case); Admin UI and connectors.json configurations accept camelCase (or kebab-case) keys.

Spring Property Config Key Default Description
base-url baseUrl http://localhost:8080 Base URL of the Zammad instance (trailing / and /api/v1 stripped automatically)
auth-type authType token Authentication mode: token (Token token=...), oauth2 (Bearer ...), or basic
access-token accessToken — Zammad personal HTTP access token or OAuth2 bearer token (masked in Platform API)
username username — Username/email for auth-type: basic
password password — Password for auth-type: basic (masked in Platform API)
on-behalf-of onBehalfOf — Optional X-On-Behalf-Of user login/email for API delegation
group-names groupNames (all) Comma-separated Zammad group names to include (case-insensitive; empty = all groups)
ticket-states ticketStates (all) Comma-separated ticket states to include (e.g. open,closed; empty = all states)
include-tickets includeTickets true Enable crawling support tickets and articles
include-internal-articles includeInternalArticles true Include internal agent notes (internal: true) with strictly agent-only ACLs (forced false when article-granularity=PUBLIC_ONLY)
article-granularity articleGranularity SPLIT_PUBLIC_AND_INTERNAL SPLIT_PUBLIC_AND_INTERNAL, TICKET (alias CONVERSATION), PER_ARTICLE, or PUBLIC_ONLY
include-attachments includeAttachments true Stream and index binary ticket article attachments
include-knowledge-base includeKnowledgeBase true Crawl localized Zammad Knowledge Base answers
include-tags includeTags true Fetch ticket tags via GET /api/v1/tags?object=Ticket&o_id={id} (also fetched automatically if tag filters are set)
crawl-mode crawlMode search_delta search_delta (updated_at:[<watermark> TO *], with optional periodic full scan) or full (full scan + hard-delete reconciliation)
state-file stateFile data/zammad-state Path to a JSON state file or directory storing watermarks, lastFullScanAt, render signatures, content hashes, ACL fingerprints, and emitted document IDs. Jobs crawling the same instance with different selection filters must use distinct state-file values.
max-attachment-size-bytes maxAttachmentSizeBytes 52428800 Maximum attachment size in bytes (50 MiB; 0 = unlimited). Oversized attachments are skipped
include-acls includeAcls true Enforce Zero-Trust ACL mapping (false marks all documents public)
include-customer-in-public-acl includeCustomerInPublicAcl true (Admin UI new template: false) Grant customer principal (user:<email> or user:zammad-user-<id> per customer-identity) read access on public ticket threads/articles/attachments. Forced false when privacy-profile=strict.
include-shared-organization-in-public-acl includeSharedOrganizationInPublicAcl true (Admin UI new template: false) Grant group:zammad-org-<org> read access on public ticket views when organization.shared == true. Forced false when privacy-profile=strict.
allow-public-kb allowPublicKb false When true, published (non-internal) KB answers are marked public; when false, restricted to KB category roles
excluded-article-senders excludedArticleSenders System Comma-separated article sender types to exclude (case-insensitive; e.g. System filters trigger notifications and escalations). Set to "" to keep all senders.
excluded-article-types excludedArticleTypes (none) Comma-separated Zammad article types to exclude (case-insensitive; e.g. sms, twitter status)
skip-auto-replies skipAutoReplies true Exclude articles flagged with Zammad's preferences.is-auto-response: true and mailer-daemon@ / postmaster@ delivery bounce notices
article-role-labels articleRoleLabels (default English labels) Optional comma-separated role=label overrides for article headings (customer, agent, internal, system), e.g. customer=Kunde,agent=Mitarbeiter,internal=Interne Notiz,system=System (defaults: Customer, Agent, Internal note, System)
created-after createdAfter (none) Only crawl tickets with created_at >= this threshold. Accepts ISO date (2024-01-01), ISO-8601 instant (2024-01-01T00:00:00Z), or relative ISO-8601 period/duration (P2Y, P90D, PT36H)
included-tags includedTags (all) Comma-separated tags (case-insensitive); ticket must match at least one tag
excluded-tags excludedTags (none) Comma-separated tags (case-insensitive, e.g. spam, test, duplicate); takes precedence over included-tags
excluded-title-patterns excludedTitlePatterns (none) Java regular expressions (one per line, or comma-separated when single-line) matched via find() against ticket titles (e.g. ^\[ALERT\])
priorities priorities (all) Comma-separated Zammad priority names to include (case-insensitive, e.g. 2 normal, 3 high)
organizations organizations (all) Comma-separated Zammad organization names to include (case-insensitive)
base-query baseQuery (none) Optional Zammad search query (e.g. group.name:"Support" AND NOT tags:spam). Combined with watermark as (<base-query>) AND updated_at:[<watermark> TO *] in incremental runs; used on /api/v1/tickets/search in full enumerations
privacy-profile privacyProfile standard standard (settings as configured) or strict (keeps e-mail addresses out of rendered text, metadata, and permission fields: upgrades email rendering to name, forces customer-identity: id, and disables customer/shared-org public ACLs)
identity-rendering identityRendering email (Admin UI new template: name) How customer, owner, and article authors are rendered in headers, headings, and metadata: email, name, id (zammad-user-<id>), or none
customer-identity customerIdentity email (Admin UI new template: id) Customer principal format in public ticket ACLs: email (e-mail/login) or id (zammad-user-<id>)
participant-metadata participantMetadata names Writes ticket participant metadata for downstream governance/anonymization hooks: names (zammad.participants), names-and-ids (zammad.participants + zammad.participantIds), or none
full-scan-interval fullScanInterval (disabled) Optional ISO-8601 duration (e.g. P1D, PT12H, P7D) triggering a periodic full ticket enumeration inside a search_delta job to tombstone hard-deleted tickets
full-scan-window fullScanWindow (anytime) Optional 24-hour time window (HH:mm-HH:mm, e.g. 01:00-05:00 or wrapping midnight 22:00-04:00) during which a due periodic full scan may run
full-scan-timezone fullScanTimezone UTC IANA timezone ID for evaluating full-scan-window (e.g. Europe/Berlin)
page-size pageSize 50 Requested page size (1..200; automatically capped at 100 for /api/v1/tickets and 200 for /api/v1/tickets/search)
parallelism parallelism 4 Max tickets processed concurrently on Java 25 Virtual Threads
max-retries maxRetries 5 Max retries for HTTP 429/502/503/504 and transient I/O errors
initial-backoff-ms initialBackoffMs 1000 Base delay (ms) for jittered exponential backoff
timeout-seconds timeoutSeconds 30 HTTP connect and request timeout in seconds

Important

Upgrade & Compatibility Notes (PR #417):

  • Default article filtering: System articles (excluded-article-senders: System) and automatic replies (skip-auto-replies: true) are excluded by default. Set excluded-article-senders: "" and skip-auto-replies: false to retain previous output.
  • Internal note headings: Internal-note headings now display the role label ([Internal note] by default) instead of the raw sender type ([Agent]).
  • First run after upgrade: Legacy state files are automatically read and migrated to the new scope-v2 filename. Existing entries without a renderSignature re-render once on the first run; subsequent runs skip unchanged rendered documents via SHA-256 content hashes.
  • Distinct state-file per selection filter: Because the default state filename hashes the core scope (base-url, group-names, ticket-states, include-tickets, include-knowledge-base) and not selection filters, jobs crawling the same Zammad instance with different selection filters (created-after, tags, priorities, organizations, base-query, etc.) must configure distinct state-file paths.
  • Saved vs. new Admin UI definitions: Existing saved definitions keep backward-compatible connector defaults (identity-rendering: email, customer-identity: email, include-customer-in-public-acl: true, include-shared-organization-in-public-acl: true). New built-in definitions created in the Admin UI start privacy-preserving (identityRendering: name, customerIdentity: id, includeCustomerInPublicAcl: false, includeSharedOrganizationInPublicAcl: false).

Spring Boot (application.yml) Example

spring:
  opencrawling:
    connector:
      type: zammad
      zammad:
        base-url: "https://support.acme.com"
        auth-type: "token"
        access-token: "${ZAMMAD_ACCESS_TOKEN}"
        group-names: "Tier-2-Identity-Security,Billing-Operations"
        ticket-states: ""
        article-granularity: "SPLIT_PUBLIC_AND_INTERNAL"  # SPLIT_PUBLIC_AND_INTERNAL | TICKET | PER_ARTICLE | PUBLIC_ONLY
        include-internal-articles: true
        include-attachments: true
        include-knowledge-base: true
        include-tags: true
        crawl-mode: "search_delta"
        state-file: "/data/zammad-state"
        max-attachment-size-bytes: 52428800
        include-acls: true
        include-customer-in-public-acl: true
        include-shared-organization-in-public-acl: true
        allow-public-kb: false
        # Article filters (#412)
        excluded-article-senders: "System"
        excluded-article-types: ""
        skip-auto-replies: true
        article-role-labels: ""       # e.g. customer=Kunde,agent=Mitarbeiter,internal=Interne Notiz,system=System
        # Ticket selection (#413)
        created-after: ""             # 2024-01-01, 2024-01-01T00:00:00Z, or P2Y
        included-tags: ""
        excluded-tags: ""             # e.g. spam, test, duplicate
        excluded-title-patterns: ""   # one regular expression per line
        priorities: ""
        organizations: ""
        base-query: ""                # Zammad search syntax, combined with watermark
        # Privacy (#414)
        privacy-profile: "standard"   # standard | strict
        identity-rendering: "email"   # email | name | id | none
        customer-identity: "email"    # email | id
        participant-metadata: "names" # names | names-and-ids | none
        # Periodic full scan (#415)
        full-scan-interval: ""        # e.g. P1D
        full-scan-window: ""          # e.g. 01:00-05:00
        full-scan-timezone: "UTC"
        page-size: 50
        parallelism: 4

Scoped Crawl Path Overrides

When starting a job via Admin UI, CLI, or Platform API, the crawl path (basePath) supports:

  • default or / — full configured scope (tickets + Knowledge Base)
  • /Groups/<GroupName> or group:<id> — tickets in a single Zammad group
  • kb-category:<id> or /KnowledgeBase — Knowledge Base scope only
  • Any valid Zammad search expression (e.g., state.name:open AND priority.name:"3 high")

📦 OIS Document Model & Metadata Schema

Canonical OIS Document IDs

Entity Canonical OIS Document ID zammad.entityType
Full ticket conversation (TICKET) zammad://<instance>/tickets/<ticketId> ticket_conversation
Public ticket thread (SPLIT_PUBLIC_AND_INTERNAL / PUBLIC_ONLY) zammad://<instance>/tickets/<ticketId>/public ticket
Internal agent notes thread (SPLIT_PUBLIC_AND_INTERNAL) zammad://<instance>/tickets/<ticketId>/internal ticket_internal
Single article (PER_ARTICLE) zammad://<instance>/tickets/<ticketId>/articles/<articleId> ticket_article
Binary article attachment zammad://<instance>/tickets/<ticketId>/articles/<articleId>/attachments/<attachmentId> ticket_attachment
Knowledge Base localized answer zammad://<instance>/kb/<kbId>/answers/<answerId>/<locale> kb_answer

Rendered HTML Article Structure

Each ticket article rendered in HTML (TICKET, SPLIT_PUBLIC_AND_INTERNAL, PER_ARTICLE, and PUBLIC_ONLY) is wrapped in a machine-readable <section> element:

<section data-article-id="1003" data-internal="true" data-role="internal" data-visibility="internal" data-sender="Agent" data-type="note" data-author="Mark Agent" data-created="2026-10-18T10:00:00Z">
  <h2>[Internal note] Mark Agent (2026-10-18T10:00:00Z)</h2>
  <p>Root cause: DHCP lease of 60 minutes on the gateway.</p>
</section>

Metadata Fields (getSchema)

Metadata Key Type Description
zammad.entityType STRING ticket, ticket_internal, ticket_conversation, ticket_article, ticket_attachment, or kb_answer
zammad.ticketId LONG Zammad numeric ticket ID
zammad.ticketNumber STRING Human-readable ticket number
zammad.title STRING Ticket or Knowledge Base answer title
zammad.group / zammad.groupId STRING / LONG Owning Zammad group name and numeric ID
zammad.state / zammad.priority STRING Ticket state (new, open, closed, …) and priority (1 low, 2 normal, 3 high)
zammad.customer / zammad.organization STRING Customer (rendered per identity-rendering) and organization name (omitted on /internal agent note documents)
zammad.owner STRING Assigned agent, rendered per identity-rendering
zammad.participants STRING (multi-valued) Unique display names of the customer, owner, and article authors (controlled by participant-metadata; droppable after governance via opencrawling.governance.drop-metadata-after-gate)
zammad.participantIds STRING (multi-valued) Unique Zammad user IDs (zammad-user-<id>) of participants when participant-metadata=names-and-ids
zammad.hasInternalNotes BOOLEAN Whether a ticket_conversation document (TICKET granularity) contains internal notes
zammad.articleRole STRING Article role on PER_ARTICLE documents: customer, agent, internal, or system
zammad.from STRING Article author on PER_ARTICLE documents, rendered per identity-rendering
zammad.tags STRING Comma-separated ticket tags
zammad.internal BOOLEAN true for internal agent notes, internal attachments, internal KB answers, and ticket_conversation documents that include internal notes
zammad.articleCount / zammad.articleId INTEGER / LONG Article count (thread/conversation view) or article ID (per-article & attachment documents)
zammad.attachmentId, zammad.fileName, zammad.fileSize, zammad.parentId LONG / STRING Attachment identifiers, size, and parent OIS document ID
zammad.kbId, zammad.answerId, zammad.categoryId, zammad.categoryTitle, zammad.locale, zammad.visibility LONG / STRING Knowledge Base hierarchy, locale code, and visibility (published / internal)
zammad.createdAt, zammad.updatedAt, zammad.closeAt TIMESTAMP ISO-8601 timestamps
zammad.webUrl STRING Direct Zammad Web UI deep link (/#ticket/zoom/<id> or /#knowledge_base/<kbId>/locale/<locale>/answer/<id>)
zammad_identity_users / zammad_identity_groups STRING Flat comma-separated resolved OIS user and group principals

🔒 Zero-Trust Security & ACL Mapping

ZammadSecurityMapper enforces fail-closed Zero-Trust access boundaries across every entity type:

Zammad Entity Effective OIS SecurityConfig Rules Notes
Ticket conversation (article-granularity: TICKET) & its attachments group:zammad-group-<group> only Always fail-closed to the owning group because internal notes are included in the conversation document.
Public ticket thread / public article / public attachment group:zammad-group-<group> + user:<customerEmail> or user:zammad-user-<id> (if enabled, per customer-identity) + group:zammad-org-<org> (only if organization.shared == true and enabled) Non-shared organizations (shared: false) never grant access to other organization members. Under privacy-profile: strict, customer and shared-organization principals are disabled (group-only).
Internal agent notes / internal article / internal attachment group:zammad-group-<group> only Customer and organization principals are strictly excluded; zammad.customer and zammad.organization metadata are also omitted.
Internal-only ticket (0 public articles, split modes) Only /tickets/<id>/internal is emitted No /public document is emitted, preventing ticket title/tags/group metadata from leaking to customers.
Orphan / unresolvable group ticket Fail-closed (permissions: [], acl: "") Never defaults to public.
Published Knowledge Base answer public if allow-public-kb=true; otherwise group:zammad-role-<role> for each category reader/editor role Falls back to active Zammad agent groups only when no granular permissions exist.
Internal Knowledge Base answer group:zammad-role-<role> for each category reader/editor role (ignores allow-public-kb) Categories with explicit granular permissions denying all roles (access: "none") fail closed (permissions: []).

All group, organization, and role names are normalized into stable identity tokens via ZammadSecurityMapper.sanitizeToken(...) (e.g. "ClientCorp Europe" → zammad-org-ClientCorp-Europe, "Tier-2 Identity Security" → zammad-group-Tier-2-Identity-Security).

Privacy & Pre-Embedding Governance Integration (#414)

To eliminate personal e-mail addresses and redact customer/agent names before embedding:

  1. Enable privacy-profile: strict (or configure identity-rendering: name / id / none and customer-identity: id explicitly).
  2. Keep participant-metadata: names (or names-and-ids) so the connector passes zammad.participants to the Pre-Embedding Governance Gate (opencrawling.governance.enabled=true).
  3. Configure OPENCRAWLING_GOVERNANCE_DROP_METADATA_AFTER_GATE=zammad.participants,zammad.participantIds (opencrawling.governance.drop-metadata-after-gate) in oc-runtime so governance/anonymization hooks can read participant names while ensuring they are stripped before chunking and never stored in the vector index. Protected keys (acl, security, ois:*, etc.) cannot be dropped.

🧪 Testing & Verification

1. Fast Standalone Connector Suite (scripts/test-zammad-connector.sh)

Runs REST API v1 contract smoke checks, all 43 unit/regression/over-HTTP tests in oc-zammad-repository-connector (including ZammadHelpdeskFeaturesTest covering ticket conversations, role labels, granularity-switch tombstones, article/ticket filters, strict privacy, ID-based customer identities, periodic full scans, legacy state migration, and the empty-enumeration guard; ZammadPropertiesTest; and tests against the local Python 3 stdlib emulator oc-zammad-repository-connector/zammad-mock/zammad_mock.py on port 18390 or an optional live Zammad instance via ZAMMAD_URL + ZAMMAD_ACCESS_TOKEN), and oc-runtime wiring & governance tests (ConnectorKindTest, ConnectorFactoryTest, ConnectorCheckerServiceTest, IngestionConsumerGovernanceTest):

# Default: starts local zammad_mock.py automatically and runs full contract + unit + over-HTTP + runtime tests
./scripts/test-zammad-connector.sh

# Optional: also probe a live Zammad instance before running the suite
ZAMMAD_URL=https://support.acme.com ZAMMAD_ACCESS_TOKEN=... ./scripts/test-zammad-connector.sh

2. Full Decoupled Pipeline E2E Suite Against Real Zammad 7.2 + Elasticsearch 9.5 (scripts/test-zammad-decoupled.sh)

Boots a real multi-container Zammad 7.2.3-0008 + Elasticsearch 9.5.5 stack alongside the OpenCrawling decoupled pipeline (oc-zammad-repository-connector/docker/docker-compose-decoupled-with-zammad.yml):

# Full production Docker build + 10-step real Zammad E2E test
./scripts/test-zammad-decoupled.sh

# Fast local iteration using BuildKit ~/.m2 cache or existing images
./scripts/test-zammad-decoupled.sh --local
./scripts/test-zammad-decoupled.sh --skip-build --keep

The 10-step suite verifies:

  1. Real Zammad bootstrap (PostgreSQL, Elasticsearch 9.5.5, Redis, Memcached, zammad-init, zammad-railsserver, zammad-scheduler, zammad-websocket).
  2. Seeding realistic Zammad groups (Tier-2-Identity-Security, Billing-Operations), shared vs. non-shared organizations, multi-article tickets (public + internal + real binary attachments), tags, and Knowledge Base answers (published + internal + archived) via Zammad Rails & REST API v1.
  3. Initial decoupled crawl through Kafka → Tika 4 ForkParser → Ollama mxbai-embed-large → PostgreSQL vector_store_1024.
  4. MCP Zero-Trust ACL Enforcement across 5 personas:
    • Customer (claire.dupont@clientcorp.eu): sees her public ticket & public attachment, blocked from internal notes, internal attachments, other customers, and internal KB.
    • Shared-org colleague (zammad-org-ClientCorp-Europe): sees shared org public ticket, blocked from non-shared PrivateBank AG and internal notes.
    • Non-shared org customer (lukas.meyer@privatebank.ch): sees only his own public ticket.
    • Tier-2 agent (zammad-group-Tier-2-Identity-Security): sees both public and internal notes/attachments for Tier-2, blocked from Billing-Operations.
    • Security Auditor (zammad-role-Security-Auditor): sees internal Security Advisories KB answer.
  5. Live REST mutations + Incremental SEARCH_DELTA: creates a new ticket, updates a ticket removing its internal note and attachments, archives a KB answer, and verifies upserts and DELETE tombstones propagate all the way to pgvector.
  6. Hard-delete + FULL reconciliation: deletes a ticket via REST API and verifies crawlMode=full emits the DELETE tombstone and purges its vectors from pgvector.
  7. Step 9 — Ticket-level conversations, filters, strict privacy & periodic full scans: switches the same job and state file to article-granularity=TICKET, privacy-profile=strict, excluded-tags=spam, base-query='group.name:"Tier-2-Identity-Security"', and full-scan-interval (PT1S, then P1D), verifying against real Zammad 7.2 + Elasticsearch + MCP that:
    • Previous /public and /internal split documents are tombstoned and replaced by /tickets/<id> conversations;
    • System articles, automatic replies (is-auto-response), and spam-tagged tickets are excluded;
    • Conversations carry zammad.entityType=ticket_conversation, zammad.articleCount, zammad.hasInternalNotes, [Internal note] labels, display names with zero e-mail addresses, zammad.participants, and group-only ACLs (verified via MCP secureVectorSearch and getDocumentContent);
    • Periodic full scans inside search_delta tombstone both a ticket subsequently tagged spam and a hard-deleted ticket;
    • Setting full-scan-interval=P1D switches the next run back to incremental enumeration combining (<base-query>) AND updated_at:[... TO *] on Zammad's Elasticsearch.
  8. Step 10 — Summary & teardown.

Both suites run automatically on pull requests and pushes to main via .github/workflows/integration-tests.yml (Zammad Connector Test Suite and Zammad Decoupled Pipeline Test Suite).


⚠️ Known Limitations

  • Knowledge Base binary attachments (KnowledgeBase::Answer::Translation::Content attachments) are not yet crawled; currently binary attachment streaming covers ticket articles only.
  • Discovery (explore / dryRun / fetchDocument) currently focuses on Zammad Groups and ticket documents (tickets/<id> under TICKET, or tickets/<id>/public|internal under split granularities); Knowledge Base category discovery and on-demand KB/attachment fetching are planned for a follow-up iteration.
  • Hard-deleted tickets in SEARCH_DELTA mode: Zammad does not expose a deletion-event feed for hard-deleted tickets; configure full-scan-interval (with optional full-scan-window and full-scan-timezone) on the incremental job—or schedule a separate crawl-mode: full reconciliation run sharing the same state-file—to purge hard-deleted tickets.
  • Local crawl state file & selection filters: ZammadCrawlState persists JSON atomically on local disk (state-file). Horizontally scaled crawler replicas should mount a persistent volume per job scope or coordinate single-writer execution, and jobs crawling the same instance with different selection filters must configure distinct state-file paths.

🗺️ Documentation


🎓 Tutorials


🧪 Testing


🤝 Community

Clone this wiki locally