Mail Collection Developer Guide

Mail collection - Developer Guide

How inbound email becomes tickets: who starts a check and how often, what one check does from start to finish, and where it all gets recorded. For setting up a mailbox, see Mailbox authentication and Basic IMAP mailboxes.

The one thing to know first: mail is collected only while somebody has the ticket inbox open. There is no scheduled task for mail. If nobody is signed in - overnight, at weekends, on a quiet install - email waits in the mailbox until somebody opens Tickets. A scheduled task is planned; see section 9.

---

1. The files

File	What it does
tickets/index.php (the inline script near the end)	The poller: checks every mailbox when the inbox opens and every 60 seconds after
api/tickets/check_mailbox_email.php	One check of one mailbox, start to finish. Everything in sections 3-6 is here unless named otherwise
includes/mailbox_graph.php	Microsoft Graph helpers: /me or /users/{address}, app-only tokens, identity checks, folder lookup
includes/gmail.php	Google: tokens, listing, fetching, mark read, trash
includes/mailbox_imap.php	Basic IMAP: connect, search, fetch (attachments ride along), after-import actions
includes/attachment_limits.php	The size check (oversize*()), see Attachment size limits
includes/tenancy.php → resolveTicketTenantForEmail()	Which company a new emailed ticket belongs to
includes/ticket_numbering.php	Ticket numbers, and finding a ticket by any number it has had
includes/mailbox_health.php	The warnings shown on each mailbox row
api/tickets/get_inbox_activity.php	The inbox's "has anything new arrived?" number, used to refresh the list
tickets/settings/index.php	Tickets → Settings → Mailboxes: the Check button and the activity viewer

The rest of the inbound family: chat channels (WhatsApp, Telegram, Slack, Teams, Mattermost) arrive by webhook through includes/messaging/ingest.php, and portal replies through api/self-service/reply_ticket.php. Neither is polled, so neither is covered here.

---

2. Timing: who starts a check, and when

The poller

Every open ticket inbox runs its own poller (checkMailboxes() in tickets/index.php):

1. On page load, and then every 60 seconds (POLL_INTERVAL) on that tab's own timer.
2. It asks api/tickets/get_mailboxes.php for the mailboxes, keeps those that are active and signed in, and posts each one's id to check_mailbox_email.php one after another, not in parallel.
3. The timed polls send auto: true. Clicking the mail icon in the inbox sends auto: false.
4. Afterwards it reads the activity marker (section 7) and refreshes the list if anything new arrived.

A polling flag stops one tab starting a second round while its first is still running. Nothing coordinates separate tabs or analysts: each has its own timer, started whenever that page was opened.

Many inboxes, one mailbox

So with five analysts in Tickets, each mailbox gets about five check requests a minute, at times that drift around each other. Since #183, the server turns that into at most one real check a minute:

A request arrives and...	What happens
another check of this mailbox is running	stands down: {"success": true, "skipped": "busy"}
it is automatic, and any check of this mailbox finished less than 50 seconds ago	stands down: {"success": true, "skipped": "recent"}
otherwise	a real check runs

• "Running" is a MySQL named lock, GET_LOCK('freeitsm_mbx_<12 hex of SHA1(database name)>_<mailbox id>', 0). The database name is in it because one MySQL server can host several installs. The lock is released automatically when the request's database connection closes, so a crashed check cannot leave a mailbox locked.
• "Finished" is target_mailboxes.last_checked_datetime, written at the end of every real check, successful or failed. So a failing mailbox is also tried about once a minute, not once per open inbox.
• 50 seconds, not 60, so that the inbox whose own timer is due still gets a turn every minute, despite drift.
• A manual check is never throttled: the mail icon, and Check in Tickets → Settings → Mailboxes (which sends no auto). It still waits its turn behind the lock; it stands down if a check is mid-run.
• A stand-down changes nothing. No last-checked time, no last error, no log row.

How fast mail arrives

• Best case: within a minute of arriving, while at least one inbox is open.
• Each check takes at most max_emails_per_check emails (per mailbox, default 10). A backlog of 200 takes 20 minutes to clear at one check a minute.
• The order differs by provider. Microsoft Graph asks for the newest unread first ($orderby=receivedDateTime DESC); IMAP takes the oldest first. So on Microsoft, during a backlog, the oldest mail waits longest.
• Nobody signed in, no mail. The mailbox's Not checked for N days warning appears after 24 hours without a check (section 7).

---

3. One check, step by step

check_mailbox_email.php for one mailbox:

1. Who may run it: a signed-in analyst with the Tickets module (requireModuleAccessJson('tickets')).
2. Load the mailbox (getMailboxConfig(), secrets decrypted). Inactive → error.
3. Lock and stand-down (section 2).
4. Get a token for the provider (section 4). If that fails, failMailboxCheck() records the error on the mailbox and stops.
5. Microsoft delegated only - the right inbox? mailboxBackfillIdentity() and mailboxIdentityMismatch() refuse to read a mailbox whose signed-in account is confirmed to be a different one (issue #26).
6. Fetch unread mail from the configured folder: getEmails() (Graph, isRead eq false), gmailGetEmails() (is:unread within the folder's label) or imapGetEmails() (UNSEEN). Every provider's messages are reshaped to the Graph shape (from.emailAddress.address, body.content, ...), so everything after this step is provider-blind.
7. For each email:
   1. Allow list. If the mailbox has any allow-list entries (domains or addresses, mailbox_email_whitelist) and the sender matches none → rejected, the rejected action runs, logged, next email.
   2. Size check (GH #170). Attachments are measured before importing, because "reject the whole email" has to be decided before a ticket exists. Too large under reject → rejected, the sender and/or the addresses in the settings are told. Too large under strip → imported without the offending attachments, which are listed in a note on the message. The fetched attachments are passed on so nothing downloads twice.
   3. Import (saveEmailToDatabase(), section 5).
   4. After-import action (section 6).
   5. New ticket only: the new ticket template email to the requester.
   6. Log it (section 7).
8. Record the outcome (recordMailboxCheckOutcome()): last-checked time always; last_error set to the joined per-email errors, or cleared. ⚠️ A check that found mail and failed to save it has not worked, even though the response says success: true, so per-email errors become the mailbox's last error too.

---

4. Tokens, per provider

Provider	Kind	How the token is got
Microsoft, delegated	somebody signed in once	token_data holds an access and refresh token. Refreshed when within 5 minutes of expiry (getValidAccessToken()), and the new pair saved. Reads /me
Microsoft, app-only	client credentials	mailboxAppOnlyToken(): a client-credentials token, cached in token_data and reused until 5 minutes before expiry. Reads /users/{target mailbox}
Google	somebody signed in once	gmailGetValidAccessToken(), the same refresh pattern
Basic IMAP	username and password	No token. A placeholder 'imap-session' keeps the shared "is there a token?" checks happy; IMAP never makes a web call

A failed token request carries the provider's own words (oauthTokenErrorMessage()) into last_error, so the mailbox row can say "the client secret has expired" without anybody pressing Check. That gap once went unnoticed for eighteen hours; see the comments in the file.

Sending mail (api/tickets/send_email.php, template emails) uses the same stored tokens, so a mailbox's sending and collecting stand or fall together. See Issue 67 - App-only email sending.

---

5. Import: a new ticket or a reply

saveEmailToDatabase():

1. Already imported? The de-dup key is emails.exchange_message_id: the Message-ID header where the provider gives one (internet_message_id; IMAP UIDs repeat across mailboxes), else the provider's own id. Found → returns ['already_imported' => true, 'ticket_id' => ...] and nothing is saved. The check logs it as Already imported with that ticket, does not count it as saved, and still runs the after-import action so the email stops coming back.
2. A reply? extractTicketReference() looks for the [SDREF:...] tag that every email FreeITSM sends carries in its subject. It captures whatever is in the tag and lets the database decide if it is real. It never pattern-matches the number's shape, because installs choose their own format (GH #71). findTicketByNumber() then finds the ticket by any number it has ever had (ticket_number_history) and follows merges to the surviving ticket.
   • Found → a reply: the ticket's updated time is touched, a finished ticket is reopened (if Tickets → Settings allow), a snoozed one is woken, and the quoted thread is stripped (stripInboundThread()).
   • Not found, or no tag → a new ticket.
3. The sender becomes a requester (getOrCreateUser()): found by address, or created.
4. New ticket:
   1. Company first - resolveTicketTenantForEmail(): a mailbox pinned to a company decides outright; a shared mailbox routes by the sender's exact address, then their domain, else the ticket goes to triage. A single-company install always uses the Default company. See Multi-tenancy email routing.
   2. Then the number - TicketNumbering::next() for that company. 🔴 The order matters: per-company numbering draws from the company's own counter, and numbering first gave every emailed ticket the Default company's number.
   3. Status and priority are the configured defaults, never the literal names Open or Normal. Source is the mailbox's default origin (#79).
5. The message is stored in emails (direction = 'Inbound', is_initial = 1 for the one that opened a ticket).
6. New ticket: confidential-mailbox defaults are applied before announcing it (ticketSensitivityApplyDefaults(), discussion #62), then ticket.created is dispatched with no actor. Reply: ticket.reply_received goes to the workflow engine instead, never both for one email.
7. Attachments are saved (saveAttachment(); rejected and quarantined files are listed in a note on the message, never dropped silently), cid: images are rewritten to stored files, and the ticket is re-indexed for search now that its attachments exist.
8. An email_import row goes to system_logs (section 7).

---

6. After importing: the mailbox's action

Per mailbox, separately for imported and rejected mail:

Setting	Microsoft	Google	IMAP
Delete	Graph DELETE on the message	Trash	delete + expunge
Move to Deleted Items	move to deleteditems	Trash	delete + expunge
Mark as read	mark read	mark read	flag \Seen
Move to folder	move to that folder	⚠️ mark read (no arbitrary folders)	move + expunge

Every fetch asks only for unread mail, so each action takes the email out of the next check's view. If the action fails, the email stays unread and is fetched again next time. The de-dup then catches it, logs it as Already imported, and tries the action again. The failure itself is in the log's post_action step and in the mailbox's last error.

---

7. Where it is all recorded

Where	What	Shown in
mailbox_activity_log	One row per email handled: action (imported, already_imported, rejected), sender, subject, reason, ticket, and processing_log - a JSON list of every step (allow list, size check, import, post action, template email) with its result	Tickets → Settings → Mailboxes → a mailbox's activity; click a row for the full processing log
system_logs (log_type = 'email_import')	One row per real import: sender, subject, ticket, new ticket or reply, attachments	Reporting → System logs → Email imports
target_mailboxes.last_checked_datetime, last_error, last_error_datetime	The mailbox's last real check and whether it worked	The mailbox row's warnings (includes/mailbox_health.php)
PHP error log	Failures that must not stop an import: attachment processing, workflow dispatch, search indexing, logging itself	Only on the server

The mailbox row's warnings: Never checked for mail (signed in but no check yet), Not checked for N days (24 hours without a check, which is deliberately generous), and The last check for mail failed, quoting the provider. The last one is never dismissible and clears itself when a check succeeds.

The inbox's refresh. After its checks, the poller asks api/tickets/get_inbox_activity.php for one number: the id of the newest message on any ticket this analyst can see. When it goes up, the list and folder counts refresh. That catches mail collected by another analyst's inbox, and chat messages that arrive by webhook, neither of which this tab's own check would have reported. The first look only sets the baseline.

---

8. Traps

• TRAP: no scheduled task collects mail. Some older text said otherwise. Until section 9 ships, an install with nobody signed in collects nothing - including Docker, which runs no scheduler at all.
• TRAP: emails.exchange_message_id has no unique index. The de-dup is a SELECT before the INSERT, so only the per-mailbox lock stops two checks importing the same email twice. Anything new that imports mail must go through check_mailbox_email.php or take the same lock.
• TRAP: success: true does not mean mail was collected. A stand-down is a success, and so is a check whose individual emails failed. Read skipped, details.emails_saved and details.errors, or the mailbox's last_error.
• TRAP: the last-checked time is written by failed checks too. It means "last tried", not "last worked". The stand-down relies on that, so a broken mailbox is not hammered once per open inbox.
• TRAP: the reply tag is matched by content, not shape. Never tighten TicketNumbering::REF_PATTERN to the current number format: every email ever sent quotes the format of its day.
• TRAP: Gmail cannot move to a folder. Move to folder on a Google mailbox marks the email read and leaves it where it is.

---

9. What's planned

Mail collection is getting a scheduled task of its own (agreed alongside #182):

• a command-line script that checks every active mailbox through the same code and the same lock, so mail arrives with nobody signed in;
• while a scheduled run has been seen in the last few minutes, the inboxes' own polls stand down; if it stops, they notice and take over again, so mail never stops for long;
• it becomes one of the jobs on the background job health page.

This page will be updated when it ships.

---

Related pages

• Mailbox checks overlapped (#183) - why the lock and the stand-down exist
• Mailbox authentication · Basic IMAP mailboxes · Gmail mail collection - Developer Guide
• Mailbox folder verification - Developer Guide · Attachment size limits - Developer Guide
• Multi-tenancy email routing · Ticket numbering - Developer Guide · Merging tickets - Developer Guide
• Email send log - Developer Guide - the outbound half