Repository navigation
Zammad Repository Connector
Module: oc-zammad-repository-connector
Connector Class: org.opencrawling.zammad.ZammadRepositoryConnector
Client Class: org.opencrawling.zammad.ZammadClient
Document Builder: org.opencrawling.zammad.ZammadDocumentBuilder
Security Mapper: org.opencrawling.zammad.ZammadSecurityMapper
Properties Class: org.opencrawling.zammad.ZammadProperties
Crawl State: org.opencrawling.zammad.ZammadCrawlState
Discovery SPI: DiscoverableRepositoryConnector, DocumentFetchingRepositoryConnector
Spring Bean: @Component (activated when spring.opencrawling.connector.type=zammad or SPRING_OPENCRAWLING_CONNECTOR_TYPE=zammad)
Issues / PRs: #342 (PR #372); #411, #412, #413, #414, #415 (PR #417)
The Zammad Repository Connector (oc-zammad-repository-connector) crawls Zammad Helpdesk & ITSM instances (self-hosted and Zammad Cloud) via the official Zammad REST API v1 (/api/v1/...), streaming support tickets, public customer conversations, confidential internal agent notes, full role-annotated ticket conversations, binary ticket attachments, ticket tags, and localized Knowledge Base answers into OpenCrawling as Open Ingestion Standard (OIS) documents.
The connector supports two complementary helpdesk indexing models with fail-closed Zero-Trust enforcement:
-
Public vs. Internal Isolation (
SPLIT_PUBLIC_AND_INTERNAL, default): emits customer-visible ticket articles (internal: false) and internal agent notes (internal: true) as separate OIS documents with independentSecurityConfigACLs, preventing confidential agent triage notes and internal attachments from ever leaking into customer-facing or cross-organization RAG answers. -
Full Ticket Conversations (
TICKET): emits a single chronologically ordered, role-annotated document per ticket (zammad://<instance>/tickets/<id>) combining customer replies and internal notes for internal support/agent RAG, locked strictly to the ticket's owning group (group:zammad-group-<group>).
Built-in article filtering (excluding system triggers and automatic replies by default with content-hash deduplication), ticket selection filters (dates/periods, tags, title regexes, priorities, organizations, and custom base-query), privacy controls (privacy-profile: strict, identity rendering, ID-based customer ACLs, participant metadata for governance hooks), and periodic full scans inside incremental jobs make the connector production-ready for large enterprise helpdesks.
Note
Real-Stack Verification: End-to-end verified in CI (Zammad Decoupled Pipeline Test Suite) against the official Zammad 7.2.3-0008 + Elasticsearch 9.5.5 container stack (PostgreSQL, Redis, Memcached, Zammad Rails/Scheduler/WebSocket) wired into the full OpenCrawling decoupled pipeline (Kafka, Tika 4 ForkParser, Ollama mxbai-embed-large, pgvector vector_store_1024, and the Secure MCP Server), plus a fast standalone suite against the built-in Python REST API v1 emulator (zammad-mock).
graph TD
subgraph Zammad["Zammad Helpdesk & ITSM (REST API v1)"]
API["REST API v1 (/api/v1/...)"]
Tickets["Tickets & Articles (public vs internal)"]
Atts["Ticket Binary Attachments"]
Tags["Ticket Tags & Lookups (Groups, States, Priorities, Roles, Users, Orgs)"]
KB["Knowledge Base (POST /knowledge_bases/init & /answers/{id}?include_contents=...)"]
ES["Elasticsearch Search Index (base-query + updated_at:[watermark TO *])"]
Tickets & Atts & Tags & KB --> API
ES --> API
end
subgraph OpenCrawling["OpenCrawling Crawler (Java 25)"]
Client["ZammadClient (Token / OAuth2 / Basic / X-On-Behalf-Of, Jittered Retries, Page Caps)"]
Connector["ZammadRepositoryConnector (SEARCH_DELTA / FULL / Periodic Full Scan, Filters, Explore, DryRun, FetchDocument)"]
CrawlState["ZammadCrawlState (Scope-v2 Key, Watermark, Render Signatures, Content Hashes, lastFullScanAt)"]
VThreads["Virtual Threads (StructuredTaskScope + ObservabilityTask, bounded parallelism)"]
DocBld["ZammadDocumentBuilder (Ticket Conversations, Split Public/Internal, Per-Article, Roles, Attachments, KB)"]
SecMap["ZammadSecurityMapper (Group, Customer Email/ID, Shared Org, Privacy Profiles, KB Roles, Fail-Closed)"]
TempStream["TempFileInputStream (Size-Capped Attachment Stream, Auto-Deleted)"]
API <-->|"HTTPS / JSON + Binary"| Client
Client --> Connector
Connector <--> CrawlState
Connector --> VThreads
VThreads --> DocBld
VThreads --> SecMap
VThreads --> TempStream
OIS["OIS Document (UPSERT / DELETE Tombstone)"]
DocBld --> OIS
SecMap --> OIS
TempStream --> OIS
end
OIS --> ClaimCheck["Claim-Check Store (Local / Apache Ozone)"]
ClaimCheck --> Kafka["Kafka (opencrawling-documents -> Tika + Governance Gate + drop-metadata-after-gate + chunking -> opencrawling-chunks -> embedding -> opencrawling-embedded)"]
Kafka --> Sinks[("Vector Stores: PGVector / Milvus / Qdrant / Solr / Vespa / OpenSearch / Elasticsearch / Luxir / OpenArch / Iceberg")]
sequenceDiagram
participant C as ZammadRepositoryConnector
participant Z as Zammad REST API v1
participant S as ZammadCrawlState (JSON)
participant P as OpenCrawling Pipeline (Kafka)
C->>Z: GET /api/v1/users/me (captures HTTP Date server clock)
C->>Z: Paginated GET /groups, /ticket_states, /ticket_priorities, /roles
C->>S: Load scope-v2 state (auto-migrating legacy file), watermark, lastFullScanAt, known tickets & KB
alt SEARCH_DELTA with watermark and periodic full scan NOT due
loop page = 1..N (per_page <= 200)
C->>Z: GET /api/v1/tickets/search?query=([base-query AND ]updated_at:[<watermark> TO *])&sort_by=updated_at&order_by=asc
end
else FULL, initial crawl, or periodic full scan due (interval + window)
alt base-query configured
loop page = 1..N (per_page <= 200)
C->>Z: GET /api/v1/tickets/search?query=<base-query>&sort_by=updated_at&order_by=asc
end
else no base-query
loop page = 1..N (per_page <= 100)
C->>Z: GET /api/v1/tickets?page=...&per_page=...&expand=true
end
end
end
par bounded parallelism (Virtual Threads)
alt ticket fails selection filters (group, state, created-after, priority, org, title, tags)
C->>P: Emit DELETE tombstones if previously indexed (otherwise skip)
else updated_at, ACL fingerprint, and renderSignature unchanged
C->>C: Skip unchanged ticket
else new or changed
C->>Z: GET /api/v1/ticket_articles/by_ticket/{id}, /tags, /users/{id}, /organizations/{id}
C->>C: Filter system/excluded senders, excluded types, auto-replies, then render roles & identities
C->>P: Emit OIS UPSERTs only when SHA-256 content hash changed and reuse unchanged attachments
C->>P: Emit OIS DELETE tombstones for removed docs / previous granularity IDs / attachments
end
end
opt Full enumeration (FULL, initial, or periodic full scan)
alt seenTicketIds is non-empty
C->>P: Emit OIS DELETE tombstones for vanished / hard-deleted tickets
else 0 tickets returned while tickets are indexed
C->>C: Log warning and emit NO tombstones (empty-enumeration guard)
end
end
opt include-knowledge-base = true
C->>Z: POST /api/v1/knowledge_bases/init (GET fallback on 404/405)
C->>Z: GET /api/v1/knowledge_bases/{kbId}/answers/{answerId}?include_contents=<ids>
C->>P: Emit OIS UPSERTs (active answers) and DELETE tombstones (archived/deleted)
end
C->>S: Atomically save state, advance watermark (serverDate - 60s) & lastFullScanAt only if 0 failures & not cancelled
- A running Zammad instance (Zammad 6.x / 7.x self-hosted or Zammad Cloud) reachable over HTTP(S) from OpenCrawling.
-
Elasticsearch enabled on Zammad for
crawl-mode: search_deltaor whenbase-queryis configured (standard in Zammad deployments;crawl-mode: fullwithoutbase-queryworks without search). -
API credentials for a Zammad agent/service account with read access to the target groups, tickets, articles, attachments, organizations, users, and Knowledge Base:
-
token(recommended): Personal HTTP Access Token (Authorization: Token token=<accessToken>), created in Zammad under Profile → Token Access. -
oauth2: OAuth2 Bearer Token (Authorization: Bearer <accessToken>). -
basic: HTTP Basic authentication (username+password). -
Optional
on-behalf-of: Sends theX-On-Behalf-Ofheader to delegate requests on behalf of another Zammad user login or email.
-
-
Flexible Ticket Granularity & Machine-Readable Roles (
article-granularity,#342,#411):-
SPLIT_PUBLIC_AND_INTERNAL(default): Emits separate OIS HTML documents for customer-visible articles (zammad://<instance>/tickets/<id>/public) and internal agent notes (zammad://<instance>/tickets/<id>/internal). Tickets containing only internal notes emit only the/internaldocument (never an empty/publicdocument). -
TICKET(aliasCONVERSATION): Emits one OIS document per ticket (zammad://<instance>/tickets/<id>) containing every included article in chronological order—both public replies and internal agent notes—with metadatazammad.entityType=ticket_conversation,zammad.articleCount, andzammad.hasInternalNotes. Because internal notes are included, ticket-level conversations and their attachments are always readable by the ticket's group only (group:zammad-group-<group>, fail-closed). -
PER_ARTICLE: Emits one OIS document per article (zammad://<instance>/tickets/<id>/articles/<articleId>), carryingzammad.articleRoleand inheriting either public or agent-only ACLs depending onarticle.internal. -
PUBLIC_ONLY: Crawls only customer-visible articles (internal: false) and their attachments. -
Semantic Role Markup & Localizable Labels: Every rendered
<section>carriesdata-article-id,data-internal,data-role(customer,agent,internal,system),data-visibility(public/internal),data-sender,data-type,data-author, anddata-created. Article headings use[<roleLabel>](defaultCustomer,Agent,Internal note,System), customizable per deployment witharticle-role-labels(e.g.customer=Kunde,agent=Mitarbeiter,internal=Interne Notiz,system=System).
-
-
Article Filtering & Content-Hash Deduplication (
#412):- Leaves out noisy system articles (trigger notifications, SLA escalations, reminders) and automatic replies by default (
excluded-article-senders: System,skip-auto-replies: true—matching Zammad'spreferences.is-auto-responseflag as well asmailer-daemon@andpostmaster@delivery bounces), plus optionalexcluded-article-types(e.g.sms, twitter status). - Stores a SHA-256 content hash per rendered document (covering content bytes, non-volatile metadata excluding
zammad.updatedAt, and ACL) inZammadCrawlState. When a ticket is updated only by a filtered system article or auto-reply—or when only its internal thread changed—unchanged rendered documents are not re-indexed. - Immutable binary attachments already indexed under the same ACL fingerprint and render signature are retained without re-downloading from Zammad.
- Leaves out noisy system articles (trigger notifications, SLA escalations, reminders) and automatic replies by default (
-
Ticket Selection Filters & Empty-Enumeration Guard (
#413):- Narrow crawled tickets with
created-after(ISO date2024-01-01, instant2024-01-01T00:00:00Z, or relative period/duration such asP2Y,P90D,PT36H),included-tags,excluded-tags(e.g.spam, test, duplicate),excluded-title-patterns(regular expressions),priorities,organizations, andbase-query(Zammad search syntax combined with the delta watermark as(<base-query>) AND updated_at:[<watermark> TO *]). - Fully integrated with incremental state and tombstones: a ticket that stops matching (for example, tagged
spamafter indexing) emitsDocumentAction.DELETEtombstones on the next scan. Discovery (explore),dryRun, andfetchDocumentapply the exact same filters. - Empty-enumeration safety guard: If a full ticket enumeration returns 0 tickets while tickets are already indexed in state, the connector logs a warning and emits no vanished-ticket tombstones, protecting against accidental mass purges from revoked credentials or a broken Zammad search index.
- Narrow crawled tickets with
-
Participant Metadata & Privacy Controls (
#414):-
identity-rendering(emaildefault,name,id,none) controls how people appear in rendered ticket headers, article headings (data-author), and metadata (zammad.customer,zammad.owner,zammad.from). Whennameis chosen, it never falls back to an e-mail address (falling back tozammad-user-<id>instead). -
customer-identity(emaildefault, orid→zammad-user-<id>) keeps customer e-mails out of OIS permission rules while still enabling per-customer ACL enforcement. -
participant-metadata(namesdefault,names-and-ids,none) populateszammad.participants(andzammad.participantIds) from the customer, owner, and article authors so downstream governance/anonymization hooks can redact personal names; pair withopencrawling.governance.drop-metadata-after-gate=zammad.participants,zammad.participantIdsinoc-runtimeso those keys are never persisted in vector stores (see Configuration Guide — Pre-Embedding Governance). -
privacy-profile: strictenforces zero e-mail addresses across rendered text, metadata, and permission rules (upgradesemailrendering toname, forcescustomer-identity: id, and disables customer and shared-organization principals on public threads). -
Privacy-by-default for new Admin UI definitions: Built-in Zammad templates created via the Admin UI start privacy-preserving (
identityRendering: name,customerIdentity: id,includeCustomerInPublicAcl: false,includeSharedOrganizationInPublicAcl: false), while saved definitions missing these keys retain backward-compatible connector defaults.
-
-
Periodic Full Scans Inside Incremental Jobs (
#415):- Configure
full-scan-interval(ISO-8601 duration, e.g.P1D)—plus optionalfull-scan-window(01:00-05:00, may wrap midnight such as22:00-04:00) andfull-scan-timezone(defaultUTC)—on acrawl-mode: search_deltajob so one run per interval automatically executes a full enumeration to detect and tombstone hard-deleted tickets (and tickets that leftbase-query), without scheduling a separatefulljob.ZammadCrawlStaterecordslastFullScanAtand the scan summary log states whether enumeration was incremental, full, or a periodic full scan.
- Configure
-
Scope-Keyed Crawl State & Seamless Granularity Switching:
-
ZammadCrawlStatefiles are keyed by the crawl scope (base-url,group-names,ticket-states,include-tickets,include-knowledge-base) rather than rendering options. Switchingarticle-granularity, article filters, identity rendering, or role labels updatesrenderSignatureunder the same state file, re-rendering tickets once and tombstoning obsolete document IDs (e.g.,/publicand/internalwhen moving toTICKET). Legacy state files written by earlier releases are automatically read and migrated on the first run.
-
-
Binary Ticket Attachments & Synthetic Body Deduplication:
- Streams article attachments via
GET /api/v1/ticket_attachment/{ticketId}/{articleId}/{attachmentId}into size-cappedTempFileInputStreamfiles (max-attachment-size-bytes, default 50 MiB), automatically deleting temporary files on close or cancellation. - Skips Zammad's synthetic
content-alternative: trueHTML email body attachments so article content is never indexed twice. - Attachment ACLs strictly inherit the parent document's visibility (internal article attachments and
TICKET-granularity attachments are strictly group-only).
- Streams article attachments via
-
Zammad 7.x Knowledge Base Ingestion & Scheduled Publishing:
- Loads KB structure via
POST /api/v1/knowledge_bases/init(with automaticGETfallback on HTTP404/405) and fetches localized translation contents viaGET /api/v1/knowledge_bases/{kbId}/answers/{answerId}?include_contents=<comma-joined-content-ids>. - Resolves
KnowledgeBaseLocale.system_locale_idagainstLocaleassets and reads Zammad's computedpermissions_effective(falling back to KB-levelpermissions_effectiveor inlinepermissions). - Evaluates
published_at,internal_at, andarchived_atagainst Zammad's server clock (CanBePublished::StateMachine), so drafts scheduled to publish in the future are never prematurely exposed and answers scheduled to archive in the future are not prematurely tombstoned.
- Loads KB structure via
-
Hardened Pagination, Rate-Limit Resilience & Cooperative Cancellation:
- Enforces Zammad's real server pagination caps (
max 100on/api/v1/tickets,max 200on/api/v1/tickets/search) and paginates metadata lookups (/groups,/roles,/ticket_states,/ticket_priorities), stopping ticket pagination only on empty or duplicate pages to prevent silent truncation and false tombstones. - Retries HTTP
429,502,503,504and transientIOExceptions honoringRetry-After(both integer seconds and RFC 1123 HTTP-date) with jittered exponential backoff. - Closes owned
HttpClientpools ondisconnect(), clears per-scan user/organization lookup caches onconnect(), and honors downstreamFluxcancellation/interruption without advancing incremental state.
- Enforces Zammad's real server pagination caps (
-
Pre-Crawl Discovery & On-Demand Fetching:
- Implements
DiscoverableRepositoryConnector(explore,dryRun,crawlStart) andDocumentFetchingRepositoryConnector(fetchDocument) with page-by-pageVisibilityFilterand selection-filter enforcement (supportinggroup:<id>and/Groups/<GroupName>references,<ticketId>:public|internal,zammad://.../tickets/<id>/public|internal, and underTICKETgranularityticket:<id>/zammad://.../tickets/<id>).
- Implements
Spring properties bind under spring.opencrawling.connector.zammad.* (kebab-case); Admin UI and connectors.json configurations accept camelCase (or kebab-case) keys.
| Spring Property | Config Key | Default | Description |
|---|---|---|---|
base-url |
baseUrl |
http://localhost:8080 |
Base URL of the Zammad instance (trailing / and /api/v1 stripped automatically) |
auth-type |
authType |
token |
Authentication mode: token (Token token=...), oauth2 (Bearer ...), or basic
|
access-token |
accessToken |
— | Zammad personal HTTP access token or OAuth2 bearer token (masked in Platform API) |
username |
username |
— | Username/email for auth-type: basic
|
password |
password |
— | Password for auth-type: basic (masked in Platform API) |
on-behalf-of |
onBehalfOf |
— | Optional X-On-Behalf-Of user login/email for API delegation |
group-names |
groupNames |
(all) | Comma-separated Zammad group names to include (case-insensitive; empty = all groups) |
ticket-states |
ticketStates |
(all) | Comma-separated ticket states to include (e.g. open,closed; empty = all states) |
include-tickets |
includeTickets |
true |
Enable crawling support tickets and articles |
include-internal-articles |
includeInternalArticles |
true |
Include internal agent notes (internal: true) with strictly agent-only ACLs (forced false when article-granularity=PUBLIC_ONLY) |
article-granularity |
articleGranularity |
SPLIT_PUBLIC_AND_INTERNAL |
SPLIT_PUBLIC_AND_INTERNAL, TICKET (alias CONVERSATION), PER_ARTICLE, or PUBLIC_ONLY
|
include-attachments |
includeAttachments |
true |
Stream and index binary ticket article attachments |
include-knowledge-base |
includeKnowledgeBase |
true |
Crawl localized Zammad Knowledge Base answers |
include-tags |
includeTags |
true |
Fetch ticket tags via GET /api/v1/tags?object=Ticket&o_id={id} (also fetched automatically if tag filters are set) |
crawl-mode |
crawlMode |
search_delta |
search_delta (updated_at:[<watermark> TO *], with optional periodic full scan) or full (full scan + hard-delete reconciliation) |
state-file |
stateFile |
data/zammad-state |
Path to a JSON state file or directory storing watermarks, lastFullScanAt, render signatures, content hashes, ACL fingerprints, and emitted document IDs. Jobs crawling the same instance with different selection filters must use distinct state-file values. |
max-attachment-size-bytes |
maxAttachmentSizeBytes |
52428800 |
Maximum attachment size in bytes (50 MiB; 0 = unlimited). Oversized attachments are skipped |
include-acls |
includeAcls |
true |
Enforce Zero-Trust ACL mapping (false marks all documents public) |
include-customer-in-public-acl |
includeCustomerInPublicAcl |
true (Admin UI new template: false)
|
Grant customer principal (user:<email> or user:zammad-user-<id> per customer-identity) read access on public ticket threads/articles/attachments. Forced false when privacy-profile=strict. |
include-shared-organization-in-public-acl |
includeSharedOrganizationInPublicAcl |
true (Admin UI new template: false)
|
Grant group:zammad-org-<org> read access on public ticket views when organization.shared == true. Forced false when privacy-profile=strict. |
allow-public-kb |
allowPublicKb |
false |
When true, published (non-internal) KB answers are marked public; when false, restricted to KB category roles |
excluded-article-senders |
excludedArticleSenders |
System |
Comma-separated article sender types to exclude (case-insensitive; e.g. System filters trigger notifications and escalations). Set to "" to keep all senders. |
excluded-article-types |
excludedArticleTypes |
(none) | Comma-separated Zammad article types to exclude (case-insensitive; e.g. sms, twitter status) |
skip-auto-replies |
skipAutoReplies |
true |
Exclude articles flagged with Zammad's preferences.is-auto-response: true and mailer-daemon@ / postmaster@ delivery bounce notices |
article-role-labels |
articleRoleLabels |
(default English labels) | Optional comma-separated role=label overrides for article headings (customer, agent, internal, system), e.g. customer=Kunde,agent=Mitarbeiter,internal=Interne Notiz,system=System (defaults: Customer, Agent, Internal note, System) |
created-after |
createdAfter |
(none) | Only crawl tickets with created_at >= this threshold. Accepts ISO date (2024-01-01), ISO-8601 instant (2024-01-01T00:00:00Z), or relative ISO-8601 period/duration (P2Y, P90D, PT36H) |
included-tags |
includedTags |
(all) | Comma-separated tags (case-insensitive); ticket must match at least one tag |
excluded-tags |
excludedTags |
(none) | Comma-separated tags (case-insensitive, e.g. spam, test, duplicate); takes precedence over included-tags
|
excluded-title-patterns |
excludedTitlePatterns |
(none) | Java regular expressions (one per line, or comma-separated when single-line) matched via find() against ticket titles (e.g. ^\[ALERT\]) |
priorities |
priorities |
(all) | Comma-separated Zammad priority names to include (case-insensitive, e.g. 2 normal, 3 high) |
organizations |
organizations |
(all) | Comma-separated Zammad organization names to include (case-insensitive) |
base-query |
baseQuery |
(none) | Optional Zammad search query (e.g. group.name:"Support" AND NOT tags:spam). Combined with watermark as (<base-query>) AND updated_at:[<watermark> TO *] in incremental runs; used on /api/v1/tickets/search in full enumerations |
privacy-profile |
privacyProfile |
standard |
standard (settings as configured) or strict (keeps e-mail addresses out of rendered text, metadata, and permission fields: upgrades email rendering to name, forces customer-identity: id, and disables customer/shared-org public ACLs) |
identity-rendering |
identityRendering |
email (Admin UI new template: name)
|
How customer, owner, and article authors are rendered in headers, headings, and metadata: email, name, id (zammad-user-<id>), or none
|
customer-identity |
customerIdentity |
email (Admin UI new template: id)
|
Customer principal format in public ticket ACLs: email (e-mail/login) or id (zammad-user-<id>) |
participant-metadata |
participantMetadata |
names |
Writes ticket participant metadata for downstream governance/anonymization hooks: names (zammad.participants), names-and-ids (zammad.participants + zammad.participantIds), or none
|
full-scan-interval |
fullScanInterval |
(disabled) | Optional ISO-8601 duration (e.g. P1D, PT12H, P7D) triggering a periodic full ticket enumeration inside a search_delta job to tombstone hard-deleted tickets |
full-scan-window |
fullScanWindow |
(anytime) | Optional 24-hour time window (HH:mm-HH:mm, e.g. 01:00-05:00 or wrapping midnight 22:00-04:00) during which a due periodic full scan may run |
full-scan-timezone |
fullScanTimezone |
UTC |
IANA timezone ID for evaluating full-scan-window (e.g. Europe/Berlin) |
page-size |
pageSize |
50 |
Requested page size (1..200; automatically capped at 100 for /api/v1/tickets and 200 for /api/v1/tickets/search) |
parallelism |
parallelism |
4 |
Max tickets processed concurrently on Java 25 Virtual Threads |
max-retries |
maxRetries |
5 |
Max retries for HTTP 429/502/503/504 and transient I/O errors |
initial-backoff-ms |
initialBackoffMs |
1000 |
Base delay (ms) for jittered exponential backoff |
timeout-seconds |
timeoutSeconds |
30 |
HTTP connect and request timeout in seconds |
Important
Upgrade & Compatibility Notes (PR #417):
-
Default article filtering: System articles (
excluded-article-senders: System) and automatic replies (skip-auto-replies: true) are excluded by default. Setexcluded-article-senders: ""andskip-auto-replies: falseto retain previous output. -
Internal note headings: Internal-note headings now display the role label (
[Internal note]by default) instead of the raw sender type ([Agent]). -
First run after upgrade: Legacy state files are automatically read and migrated to the new
scope-v2filename. Existing entries without arenderSignaturere-render once on the first run; subsequent runs skip unchanged rendered documents via SHA-256 content hashes. -
Distinct
state-fileper selection filter: Because the default state filename hashes the core scope (base-url,group-names,ticket-states,include-tickets,include-knowledge-base) and not selection filters, jobs crawling the same Zammad instance with different selection filters (created-after, tags, priorities, organizations,base-query, etc.) must configure distinctstate-filepaths. -
Saved vs. new Admin UI definitions: Existing saved definitions keep backward-compatible connector defaults (
identity-rendering: email,customer-identity: email,include-customer-in-public-acl: true,include-shared-organization-in-public-acl: true). New built-in definitions created in the Admin UI start privacy-preserving (identityRendering: name,customerIdentity: id,includeCustomerInPublicAcl: false,includeSharedOrganizationInPublicAcl: false).
spring:
opencrawling:
connector:
type: zammad
zammad:
base-url: "https://support.acme.com"
auth-type: "token"
access-token: "${ZAMMAD_ACCESS_TOKEN}"
group-names: "Tier-2-Identity-Security,Billing-Operations"
ticket-states: ""
article-granularity: "SPLIT_PUBLIC_AND_INTERNAL" # SPLIT_PUBLIC_AND_INTERNAL | TICKET | PER_ARTICLE | PUBLIC_ONLY
include-internal-articles: true
include-attachments: true
include-knowledge-base: true
include-tags: true
crawl-mode: "search_delta"
state-file: "/data/zammad-state"
max-attachment-size-bytes: 52428800
include-acls: true
include-customer-in-public-acl: true
include-shared-organization-in-public-acl: true
allow-public-kb: false
# Article filters (#412)
excluded-article-senders: "System"
excluded-article-types: ""
skip-auto-replies: true
article-role-labels: "" # e.g. customer=Kunde,agent=Mitarbeiter,internal=Interne Notiz,system=System
# Ticket selection (#413)
created-after: "" # 2024-01-01, 2024-01-01T00:00:00Z, or P2Y
included-tags: ""
excluded-tags: "" # e.g. spam, test, duplicate
excluded-title-patterns: "" # one regular expression per line
priorities: ""
organizations: ""
base-query: "" # Zammad search syntax, combined with watermark
# Privacy (#414)
privacy-profile: "standard" # standard | strict
identity-rendering: "email" # email | name | id | none
customer-identity: "email" # email | id
participant-metadata: "names" # names | names-and-ids | none
# Periodic full scan (#415)
full-scan-interval: "" # e.g. P1D
full-scan-window: "" # e.g. 01:00-05:00
full-scan-timezone: "UTC"
page-size: 50
parallelism: 4When starting a job via Admin UI, CLI, or Platform API, the crawl path (basePath) supports:
-
defaultor/— full configured scope (tickets + Knowledge Base) -
/Groups/<GroupName>orgroup:<id>— tickets in a single Zammad group -
kb-category:<id>or/KnowledgeBase— Knowledge Base scope only - Any valid Zammad search expression (e.g.,
state.name:open AND priority.name:"3 high")
| Entity | Canonical OIS Document ID | zammad.entityType |
|---|---|---|
Full ticket conversation (TICKET) |
zammad://<instance>/tickets/<ticketId> |
ticket_conversation |
Public ticket thread (SPLIT_PUBLIC_AND_INTERNAL / PUBLIC_ONLY) |
zammad://<instance>/tickets/<ticketId>/public |
ticket |
Internal agent notes thread (SPLIT_PUBLIC_AND_INTERNAL) |
zammad://<instance>/tickets/<ticketId>/internal |
ticket_internal |
Single article (PER_ARTICLE) |
zammad://<instance>/tickets/<ticketId>/articles/<articleId> |
ticket_article |
| Binary article attachment | zammad://<instance>/tickets/<ticketId>/articles/<articleId>/attachments/<attachmentId> |
ticket_attachment |
| Knowledge Base localized answer | zammad://<instance>/kb/<kbId>/answers/<answerId>/<locale> |
kb_answer |
Each ticket article rendered in HTML (TICKET, SPLIT_PUBLIC_AND_INTERNAL, PER_ARTICLE, and PUBLIC_ONLY) is wrapped in a machine-readable <section> element:
<section data-article-id="1003" data-internal="true" data-role="internal" data-visibility="internal" data-sender="Agent" data-type="note" data-author="Mark Agent" data-created="2026-10-18T10:00:00Z">
<h2>[Internal note] Mark Agent (2026-10-18T10:00:00Z)</h2>
<p>Root cause: DHCP lease of 60 minutes on the gateway.</p>
</section>| Metadata Key | Type | Description |
|---|---|---|
zammad.entityType |
STRING |
ticket, ticket_internal, ticket_conversation, ticket_article, ticket_attachment, or kb_answer
|
zammad.ticketId |
LONG |
Zammad numeric ticket ID |
zammad.ticketNumber |
STRING |
Human-readable ticket number |
zammad.title |
STRING |
Ticket or Knowledge Base answer title |
zammad.group / zammad.groupId
|
STRING / LONG
|
Owning Zammad group name and numeric ID |
zammad.state / zammad.priority
|
STRING |
Ticket state (new, open, closed, …) and priority (1 low, 2 normal, 3 high) |
zammad.customer / zammad.organization
|
STRING |
Customer (rendered per identity-rendering) and organization name (omitted on /internal agent note documents) |
zammad.owner |
STRING |
Assigned agent, rendered per identity-rendering
|
zammad.participants |
STRING (multi-valued)
|
Unique display names of the customer, owner, and article authors (controlled by participant-metadata; droppable after governance via opencrawling.governance.drop-metadata-after-gate) |
zammad.participantIds |
STRING (multi-valued)
|
Unique Zammad user IDs (zammad-user-<id>) of participants when participant-metadata=names-and-ids
|
zammad.hasInternalNotes |
BOOLEAN |
Whether a ticket_conversation document (TICKET granularity) contains internal notes |
zammad.articleRole |
STRING |
Article role on PER_ARTICLE documents: customer, agent, internal, or system
|
zammad.from |
STRING |
Article author on PER_ARTICLE documents, rendered per identity-rendering
|
zammad.tags |
STRING |
Comma-separated ticket tags |
zammad.internal |
BOOLEAN |
true for internal agent notes, internal attachments, internal KB answers, and ticket_conversation documents that include internal notes |
zammad.articleCount / zammad.articleId
|
INTEGER / LONG
|
Article count (thread/conversation view) or article ID (per-article & attachment documents) |
zammad.attachmentId, zammad.fileName, zammad.fileSize, zammad.parentId
|
LONG / STRING
|
Attachment identifiers, size, and parent OIS document ID |
zammad.kbId, zammad.answerId, zammad.categoryId, zammad.categoryTitle, zammad.locale, zammad.visibility
|
LONG / STRING
|
Knowledge Base hierarchy, locale code, and visibility (published / internal) |
zammad.createdAt, zammad.updatedAt, zammad.closeAt
|
TIMESTAMP |
ISO-8601 timestamps |
zammad.webUrl |
STRING |
Direct Zammad Web UI deep link (/#ticket/zoom/<id> or /#knowledge_base/<kbId>/locale/<locale>/answer/<id>) |
zammad_identity_users / zammad_identity_groups
|
STRING |
Flat comma-separated resolved OIS user and group principals |
ZammadSecurityMapper enforces fail-closed Zero-Trust access boundaries across every entity type:
| Zammad Entity | Effective OIS SecurityConfig Rules |
Notes |
|---|---|---|
Ticket conversation (article-granularity: TICKET) & its attachments |
group:zammad-group-<group> only
|
Always fail-closed to the owning group because internal notes are included in the conversation document. |
| Public ticket thread / public article / public attachment |
group:zammad-group-<group> + user:<customerEmail> or user:zammad-user-<id> (if enabled, per customer-identity) + group:zammad-org-<org> (only if organization.shared == true and enabled)
|
Non-shared organizations (shared: false) never grant access to other organization members. Under privacy-profile: strict, customer and shared-organization principals are disabled (group-only). |
| Internal agent notes / internal article / internal attachment |
group:zammad-group-<group> only
|
Customer and organization principals are strictly excluded; zammad.customer and zammad.organization metadata are also omitted. |
| Internal-only ticket (0 public articles, split modes) | Only /tickets/<id>/internal is emitted |
No /public document is emitted, preventing ticket title/tags/group metadata from leaking to customers. |
| Orphan / unresolvable group ticket |
Fail-closed (permissions: [], acl: "") |
Never defaults to public. |
| Published Knowledge Base answer |
public if allow-public-kb=true; otherwise group:zammad-role-<role> for each category reader/editor role |
Falls back to active Zammad agent groups only when no granular permissions exist. |
| Internal Knowledge Base answer |
group:zammad-role-<role> for each category reader/editor role (ignores allow-public-kb) |
Categories with explicit granular permissions denying all roles (access: "none") fail closed (permissions: []). |
All group, organization, and role names are normalized into stable identity tokens via ZammadSecurityMapper.sanitizeToken(...) (e.g. "ClientCorp Europe" → zammad-org-ClientCorp-Europe, "Tier-2 Identity Security" → zammad-group-Tier-2-Identity-Security).
To eliminate personal e-mail addresses and redact customer/agent names before embedding:
- Enable
privacy-profile: strict(or configureidentity-rendering: name/id/noneandcustomer-identity: idexplicitly). - Keep
participant-metadata: names(ornames-and-ids) so the connector passeszammad.participantsto the Pre-Embedding Governance Gate (opencrawling.governance.enabled=true). - Configure
OPENCRAWLING_GOVERNANCE_DROP_METADATA_AFTER_GATE=zammad.participants,zammad.participantIds(opencrawling.governance.drop-metadata-after-gate) inoc-runtimeso governance/anonymization hooks can read participant names while ensuring they are stripped before chunking and never stored in the vector index. Protected keys (acl,security,ois:*, etc.) cannot be dropped.
Runs REST API v1 contract smoke checks, all 43 unit/regression/over-HTTP tests in oc-zammad-repository-connector (including ZammadHelpdeskFeaturesTest covering ticket conversations, role labels, granularity-switch tombstones, article/ticket filters, strict privacy, ID-based customer identities, periodic full scans, legacy state migration, and the empty-enumeration guard; ZammadPropertiesTest; and tests against the local Python 3 stdlib emulator oc-zammad-repository-connector/zammad-mock/zammad_mock.py on port 18390 or an optional live Zammad instance via ZAMMAD_URL + ZAMMAD_ACCESS_TOKEN), and oc-runtime wiring & governance tests (ConnectorKindTest, ConnectorFactoryTest, ConnectorCheckerServiceTest, IngestionConsumerGovernanceTest):
# Default: starts local zammad_mock.py automatically and runs full contract + unit + over-HTTP + runtime tests
./scripts/test-zammad-connector.sh
# Optional: also probe a live Zammad instance before running the suite
ZAMMAD_URL=https://support.acme.com ZAMMAD_ACCESS_TOKEN=... ./scripts/test-zammad-connector.sh2. Full Decoupled Pipeline E2E Suite Against Real Zammad 7.2 + Elasticsearch 9.5 (scripts/test-zammad-decoupled.sh)
Boots a real multi-container Zammad 7.2.3-0008 + Elasticsearch 9.5.5 stack alongside the OpenCrawling decoupled pipeline (oc-zammad-repository-connector/docker/docker-compose-decoupled-with-zammad.yml):
# Full production Docker build + 10-step real Zammad E2E test
./scripts/test-zammad-decoupled.sh
# Fast local iteration using BuildKit ~/.m2 cache or existing images
./scripts/test-zammad-decoupled.sh --local
./scripts/test-zammad-decoupled.sh --skip-build --keepThe 10-step suite verifies:
- Real Zammad bootstrap (PostgreSQL, Elasticsearch
9.5.5, Redis, Memcached,zammad-init,zammad-railsserver,zammad-scheduler,zammad-websocket). - Seeding realistic Zammad groups (
Tier-2-Identity-Security,Billing-Operations), shared vs. non-shared organizations, multi-article tickets (public + internal + real binary attachments), tags, and Knowledge Base answers (published + internal + archived) via Zammad Rails & REST API v1. - Initial decoupled crawl through Kafka → Tika 4 ForkParser → Ollama
mxbai-embed-large→ PostgreSQLvector_store_1024. -
MCP Zero-Trust ACL Enforcement across 5 personas:
- Customer (
claire.dupont@clientcorp.eu): sees her public ticket & public attachment, blocked from internal notes, internal attachments, other customers, and internal KB. - Shared-org colleague (
zammad-org-ClientCorp-Europe): sees shared org public ticket, blocked from non-sharedPrivateBank AGand internal notes. - Non-shared org customer (
lukas.meyer@privatebank.ch): sees only his own public ticket. - Tier-2 agent (
zammad-group-Tier-2-Identity-Security): sees both public and internal notes/attachments for Tier-2, blocked fromBilling-Operations. - Security Auditor (
zammad-role-Security-Auditor): sees internal Security Advisories KB answer.
- Customer (
-
Live REST mutations + Incremental
SEARCH_DELTA: creates a new ticket, updates a ticket removing its internal note and attachments, archives a KB answer, and verifies upserts andDELETEtombstones propagate all the way topgvector. -
Hard-delete +
FULLreconciliation: deletes a ticket via REST API and verifiescrawlMode=fullemits theDELETEtombstone and purges its vectors frompgvector. -
Step 9 — Ticket-level conversations, filters, strict privacy & periodic full scans: switches the same job and state file to
article-granularity=TICKET,privacy-profile=strict,excluded-tags=spam,base-query='group.name:"Tier-2-Identity-Security"', andfull-scan-interval(PT1S, thenP1D), verifying against real Zammad 7.2 + Elasticsearch + MCP that:- Previous
/publicand/internalsplit documents are tombstoned and replaced by/tickets/<id>conversations; - System articles, automatic replies (
is-auto-response), and spam-tagged tickets are excluded; - Conversations carry
zammad.entityType=ticket_conversation,zammad.articleCount,zammad.hasInternalNotes,[Internal note]labels, display names with zero e-mail addresses,zammad.participants, and group-only ACLs (verified via MCPsecureVectorSearchandgetDocumentContent); - Periodic full scans inside
search_deltatombstone both a ticket subsequently taggedspamand a hard-deleted ticket; - Setting
full-scan-interval=P1Dswitches the next run back to incremental enumeration combining(<base-query>) AND updated_at:[... TO *]on Zammad's Elasticsearch.
- Previous
- Step 10 — Summary & teardown.
Both suites run automatically on pull requests and pushes to main via .github/workflows/integration-tests.yml (Zammad Connector Test Suite and Zammad Decoupled Pipeline Test Suite).
-
Knowledge Base binary attachments (
KnowledgeBase::Answer::Translation::Contentattachments) are not yet crawled; currently binary attachment streaming covers ticket articles only. -
Discovery (
explore/dryRun/fetchDocument) currently focuses on Zammad Groups and ticket documents (tickets/<id>underTICKET, ortickets/<id>/public|internalunder split granularities); Knowledge Base category discovery and on-demand KB/attachment fetching are planned for a follow-up iteration. -
Hard-deleted tickets in
SEARCH_DELTAmode: Zammad does not expose a deletion-event feed for hard-deleted tickets; configurefull-scan-interval(with optionalfull-scan-windowandfull-scan-timezone) on the incremental job—or schedule a separatecrawl-mode: fullreconciliation run sharing the samestate-file—to purge hard-deleted tickets. -
Local crawl state file & selection filters:
ZammadCrawlStatepersists JSON atomically on local disk (state-file). Horizontally scaled crawler replicas should mount a persistent volume per job scope or coordinate single-writer execution, and jobs crawling the same instance with different selection filters must configure distinctstate-filepaths.
- Home
- User & Installation Guide
- Quick Start
- VM & Bare-Metal Distribution Installation
- Local Development & Fast Docker Builds
- User Guide
- Architecture
- Zero-Trust Security & ACL Management
- Security and Identity Management
- Document Authorization & OpenFGA
- Enterprise Multi-Agent Integration
- Zero-Trust Data Provider Agent & Active Ingestion
- Document Lifecycle Actions & Tombstones
- Apache Tika 4 Text Extraction
- Configuration Guide
- Native Job Scheduler
- Platform API (
/api/v1) - Platform API: Traceability
- Connectors Reference
- Alfresco Content Services (ACS) Connector
- Alfresco Process Services (APS) Connector
- Flowable Repository Connector
- Camunda Repository Connector
- OASIS CMIS Repository Connector
- Doxis Repository Connector
- M-Files Repository Connector
- SharePoint Online Repository Connector
- Google Drive Repository Connector
- Atlassian Confluence Repository Connector
- IBM FileNet P8 Repository Connector
- OpenText Documentum Repository Connector
- Zammad Repository Connector
- Hyland OpenArch Integration (Repository, Output & MCP)
- Hyland Content Innovation Cloud (CIC) Connector
- Relational Database (JDBC) Connector
- Web Repository Connector
- Apache StormCrawler Repository Connector
- gRPC Push Repository Connector
- Elasticsearch Output Connector
- Apache Solr Output Connector
- Luxir Output Connector
- Apache SeaTunnel Output Connector
- Apache Ozone Output Connector (Migration Mode)
- AWS S3 Output Connector (Migration Mode)
- Alfresco Content Services Output Connector (Migration Mode)
- Apache Iceberg Output Connector
- Doxis Output Connector
- Connector Execution Modes
- Model Context Protocol
- Apache Ozone Claim Check Store
- Lakehouse Audit Reporting & Ledger
- AI-Powered Observability (AIOps)
- Auto-Narrativization Copilot
- Maven Connector Archetypes
- Java Client SDK
- Python Client SDK
- OpenCrawling CLI
- OpenAPI Specification & Swagger UI
- gRPC Internal Transport
- Migration Guide (ManifoldCF)
- Integration Test — Decoupled Pipeline
- M-Files Integration Test
- SharePoint Online Integration Test
- Google Drive Integration Test
- Confluence Integration Test
- IBM FileNet Integration Test
- OpenText Documentum Integration Test
- Zammad Integration Test
- Hyland OpenArch Integration Test
- Hyland CIC Integration Test
- Web Crawler Test
- Elasticsearch Integration Test
- Local Docker Build Test
- Root CLI Runner Scripts Test
- Secure MCP Server Integration Test
- Narrativization Integration Test
- Reporting Integration Test
- AWS S3 Migration Integration Test
- Alfresco Migration Integration Test
- OpenFGA Authorization Integration Test
- CLI Decoupled Integration Test
- Java Client SDK Decoupled Test
- Apache Iceberg Decoupled Test
- Doxis Output Integration Test
- Platform API v1 Test Suite