Basic IMAP Client Developer Guide

Basic IMAP client - Developer Guide

How FreeITSM reads a Basic IMAP mailbox without PHP's imap extension: the files, the protocol conversation, the PHP that drives it, and how it plugs into the existing mail pipeline.

• Why it is built this way: Basic IMAP without the PHP extension
• Testing and verifying it: Basic IMAP client - Testing
• Issues: #257, #259, #50

---

1. The files

File	Role
includes/imap_client.php	New. The IMAP client: the ImapClient class (the connection), the reply parser, and the readers for BODYSTRUCTURE, ENVELOPE, encoded headers and filenames. Knows nothing about tickets
includes/mailbox_imap.php	Changed. The Basic IMAP provider: turns a mailbox row into a signed-in connection, reads unread mail into FreeITSM's message shape, applies the post-import action, verifies a folder - and sends replies over SMTP. Its public functions kept their names and signatures
api/tickets/check_mailbox_email.php	Unchanged. Calls imapGetEmails() (line ~162) and imapHandleAfterProcessing() (line ~832)
api/tickets/verify_mailbox_folder.php	Unchanged. Calls imapVerifyFolder()
includes/ssl.php	Unchanged. Its sslResolveCaBundle() now also serves the IMAP and SMTP sockets
tests/imap-client-parse.php, tests/imap-client-live.php	New. See Testing
docker/php-extensions.txt, setup/index.php	imap removed from both - nothing needs it

Nothing above mailbox_imap.php changed. The importer, whitelisting, company routing, threading and attachment storage all receive the same arrays as before.

---

2. Three layers

check_mailbox_email.php ─┐                      (unchanged)
verify_mailbox_folder.php┤
                         ▼
  mailbox_imap.php       imapStreamFor() · imapGetEmails() · imapNormaliseMessage()
  (the provider)         imapWalkStructure() · imapHandleAfterProcessing() · imapVerifyFolder()
                         ▼
  imap_client.php        3. message structure   imapBodyStructure() imapEnvelope()
                                                imapPartFilename() imapDecodeMimeHeader()
                         2. reply parser        imapParseValues() → arrays, strings, null
                         1. connection          ImapClient: TLS, tags, literals, NO/BAD
                         ▼
                         PHP: stream_socket_client(), OpenSSL, base64, mbstring/iconv

Each layer only talks to the one below. The connection does not know what a message is; the structure readers do not know there is a network.

---

3. The protocol, by example

IMAP is a text conversation over one connection. This is a real session with the Dovecot server used in the tests, recorded from the client, reading one email with an attachment (C = FreeITSM, S = server). The password is masked, long capability lists and the server's timings on tagged lines are shortened to ..., and nothing else is changed.

Connect and sign in

(TLS handshake on port 993 - certificate checked)
S: * OK [CAPABILITY IMAP4rev1 IMAP4rev2 LOGIN-REFERRALS ID ENABLE IDLE SASL-IR LITERAL+ AUTH=PLAIN] Dovecot ready.
C: A0001 LOGIN "wiki-5744" "********"
S: A0001 OK [CAPABILITY IMAP4rev1 IMAP4rev2 SASL-IR LOGIN-REFERRALS ID ENABLE IDLE SORT ...] ...
C: A0002 CAPABILITY
S: * CAPABILITY IMAP4rev1 IMAP4rev2 SASL-IR ... UIDPLUS ... MOVE ...
S: A0002 OK ...

• Every command starts with a tag (A0001, A0002...). The server answers with any number of untagged lines (* ...) and then one tagged line - A0002 OK, NO or BAD - saying whether that command worked. The client reads until its own tag comes back. That is how it knows a reply is complete; never "the next line".
• Capabilities are asked for again after sign-in, because servers commonly reveal MOVE and UIDPLUS only then. They decide which commands to use later.
• STARTTLS (encryption tls, port 143): the client connects in plain text, sends STARTTLS, upgrades the socket to TLS, then throws away the capabilities it learnt before the upgrade and asks again (RFC 3501 §6.2.1 - a man in the middle could have forged them).

Open the folder and find unread mail

C: A0005 SELECT "INBOX"
S: * FLAGS (\Answered \Flagged \Deleted \Seen \Draft)
S: * OK [PERMANENTFLAGS (\Answered \Flagged \Deleted \Seen \Draft \*)] Flags permitted.
S: * 1 EXISTS
S: * 1 RECENT
S: * OK [UNSEEN 1] First unseen.
S: * OK [UIDVALIDITY 1794081623] UIDs valid
S: * OK [UIDNEXT 2] Predicted next UID
S: A0005 OK [READ-WRITE] Select completed ...
C: A0006 UID SEARCH UNSEEN
S: * SEARCH 1
S: A0006 OK ...

(A0003 and A0004 were the test putting an email in and making a folder - APPEND and CREATE - so there was something to read.)

• * 1 EXISTS is the folder's message count - Verify's total.
• Everything is done by UID, which stays the same for a message in that folder, rather than by sequence number, which shifts whenever anything is deleted. The UID is what imapHandleAfterProcessing() later acts on.

Read one message - without marking it read

C: A0007 UID FETCH 1 (UID ENVELOPE BODYSTRUCTURE INTERNALDATE)
S: * 1 FETCH (UID 1 INTERNALDATE "07-Nov-2026 20:00:30 +0000"
     ENVELOPE (NIL "Printer jammed"
       (("=?UTF-8?B?SsO2cmc=?=" NIL "jorg" "example.com"))   ← from
       (("=?UTF-8?B?SsO2cmc=?=" NIL "jorg" "example.com"))   ← sender
       (("=?UTF-8?B?SsO2cmc=?=" NIL "jorg" "example.com"))   ← reply-to
       ((NIL NIL "support" "freeitsm.test"))                 ← to
       NIL NIL NIL "<wiki-1@example.com>")                   ← cc, bcc, in-reply-to, message-id
     BODYSTRUCTURE (
       ("text" "html" ("charset" "utf-8") NIL NIL "7bit" 20 0 NIL NIL NIL NIL)          ← part 1
       ("application" "pdf" NIL NIL NIL "base64" 12 NIL
          ("attachment" ("filename" "error.pdf")) NIL NIL)                              ← part 2
       "mixed" ("boundary" "b") NIL NIL NIL))
S: A0007 OK ...

(The reply is one line; it is broken up here, with ← notes, to read. The date is NIL because this test email had no Date header - FreeITSM then uses INTERNALDATE, when the server received it.)

This one reply is the whole of what used to need imap_fetch_overview() and imap_fetchstructure(). The server has already parsed the email. The client never sees a MIME boundary; it sees a tree: part 1 is HTML in UTF-8, part 2 is a base64 PDF called error.pdf, and the two sit in a multipart/mixed.

Then each part it needs, by number:

C: A0008 UID FETCH 1 BODY.PEEK[1]
S: * 1 FETCH (UID 1 BODY[1] {20}
S: <p>It is jammed.</p>)
S: A0008 OK ...
C: A0009 UID FETCH 1 BODY.PEEK[2]
S: * 1 FETCH (UID 1 BODY[2] {12}
S: JVBERi0xLjQ=)
S: A0009 OK ...

• BODY.PEEK, not BODY. Plain BODY[...] marks the message \Seen; PEEK does not. That is the old FT_PEEK, and it is why a message stays unread until the mailbox's post-import action decides what happens to it. Note the server echoes the section without .PEEK - BODY[1] - which is the key the client looks for.
• {20} is a literal: "the next 20 bytes are data". The client then reads exactly 20 bytes, whatever they contain - CRLFs, quotes, binary. This is how attachments arrive safely. After the literal the line continues ()), so the client keeps reading.
• Part numbers nest: in a multipart/mixed holding a multipart/alternative, the HTML might be 1.2 and an attachment 2. A single-part email's body is part 1.

Afterwards: mark read, move or delete

C: A0010 UID MOVE 1 "Processed"
S: * OK [COPYUID 1794081625 1 1] Moved UIDs.
S: * 1 EXPUNGE
S: * 0 RECENT
S: A0010 OK ...

Post-import action	Commands sent
Mark read	UID STORE 1 +FLAGS.SILENT (\Seen)
Move to folder	UID MOVE 1 "Folder" - or, on a server without MOVE: UID COPY, UID STORE +FLAGS.SILENT (\Deleted), then EXPUNGE
Delete / move to deleted	UID STORE 1 +FLAGS.SILENT (\Deleted), then UID EXPUNGE 1 with UIDPLUS, else EXPUNGE

TRAP: plain EXPUNGE removes every message flagged \Deleted in the folder, not just ours. With UIDPLUS the client expunges only the one UID. Without it, it does what the old extension did on every delete and move (imap_expunge). Both test servers have UIDPLUS; the fallback is tested by hiding it.

Folders, and goodbye

C: A0011 LIST "" "*"
S: * LIST (\HasNoChildren) "/" Processed
S: * LIST (\HasNoChildren \Drafts) "/" Drafts
S: * LIST (\HasNoChildren \Junk) "/" Junk
S: * LIST (\HasNoChildren \Sent) "/" Sent
S: * LIST (\HasNoChildren \Trash) "/" Trash
S: * LIST (\HasNoChildren) "/" INBOX
S: A0011 OK ...
C: A0012 LOGOUT
S: * BYE Logging out
S: A0012 OK Logout completed ...

LIST gives each folder's flags, its hierarchy delimiter (/ here, . on many cPanel servers - hence "INBOX.Support") and its name. Entries flagged \Noselect are namespace placeholders, not folders, and are left out of Verify's list.

Literals going the other way

The client sends literals too. Any value containing a line break, a NUL or a non-ASCII byte is sent as {n} + the bytes, after the server's go-ahead:

C: A0001 LOGIN "wiki-lit" {10}
S: + OK
C: pässwörd                       (exactly 10 bytes - ä and ö are two bytes each in UTF-8)
S: A0001 NO [AUTHENTICATIONFAILED] Authentication failed.

(Recorded with a deliberately wrong password, so the last line also shows what a refusal looks like - NO, with the server's reason, which is what the client puts in its error.)

ImapClient::sendCommand() splits the command at each {n}\r\n, waits for the + line, sends exactly n bytes, and carries on. A tagged NO instead of + means the server refused, and the client stops.

---

4. Layer 1 - the connection (ImapClient)

Method	Sends	Returns
connect()	TLS handshake (ssl), or plain + STARTTLS (tls); reads the greeting	-
login($user, $pass)	LOGIN, then CAPABILITY	-
capability() / hasCapability('MOVE')	CAPABILITY	the list, upper-cased
listFolders()	LIST "" "*"	[['name', 'delimiter', 'flags'], ...], names in UTF-8
select($folder)	SELECT	the message count (EXISTS)
uidSearch('UNSEEN')	UID SEARCH	UIDs, ascending (oldest first)
uidFetch($uid, $items)	UID FETCH	['ENVELOPE' => [...], 'BODY[2]' => '...']
uidAddFlags, uidMove, uidDelete	as in the table above	-
logout()	LOGOUT	-
command($text)	anything, tagged	the untagged lines; throws on NO/BAD with the server's own words

How command() reads:

1. Send A0007 <command> (splitting at literals, above).
2. Read logical lines: a line, and if it ends in {n}, exactly n more bytes and the rest of the line, repeatedly. A logical line can therefore contain CRLFs.
3. * ... lines are collected (and scanned for CAPABILITY). A line starting with our tag ends the command: OK returns the collected lines, anything else throws ImapException.

Unrequested lines are normal in IMAP: a server may add * 3 EXISTS when mail arrives, or a * FETCH for a flag change on another message. uidFetch() keeps only the FETCH whose UID matches the one asked for; other commands look only for the lines they need.

5. Layer 2 - the reply parser

imapParseValues($text) turns IMAP data into PHP values:

IMAP	PHP
(a b (c))	['a', 'b', ['c']]
"say \"hi\""	'say "hi"'
{5}\r\nhello	'hello' (a literal)
NIL	null
UID, \Seen, BODY[1.2], BODY[HEADER.FIELDS (SUBJECT)]	the string as it is (brackets belong to the atom, spaces inside them too)

It never loops forever on bad input: a truncated list returns what it has, and a literal shorter than declared stops at the end of the data.

6. Layer 3 - message structure

imapBodyStructure($bs) turns the parsed BODYSTRUCTURE into a tree of arrays - the shape imap_fetchstructure() gave, minus the numeric type codes:

['type' => 'application', 'subtype' => 'pdf', 'params' => [], 'id' => null,
 'encoding' => 'base64', 'disposition' => 'attachment',
 'dparams' => ['filename' => 'error.pdf'], 'parts' => []]

A multipart node has type => 'multipart' and its children in parts. The fields sit at fixed positions, with one twist: where the disposition (attachment/inline and its filename) sits depends on the type, because some types carry extra fields first:

Part type	Fields before the extension data	Disposition at
text/*	type, subtype, params, id, description, encoding, size, lines	index 9
message/rfc822 (an attached email)	... size, envelope, body, lines	index 11
anything else	... size	index 8
multipart/*	children, subtype, params	after the subtype + 2

TRAP: get those positions wrong and attachments silently become body text, or lose their names. tests/imap-client-parse.php has a case for each row, and the live test sends a real attached email.

imapEnvelope($env) reads date, subject, from, to, cc and Message-ID. Addresses arrive as (name route mailbox host); the client joins mailbox@host and decodes the name. Group markers ((NIL NIL "group" NIL) ... (NIL NIL NIL NIL)) carry no address and are skipped.

Decoding - all with functions PHP already has

What	Example	How
Transfer encoding	base64, quoted-printable	base64_decode(), quoted_printable_decode() - imapDecodeTransfer()
Character set	ISO-8859-1, Windows-1252, Shift_JIS	mb_convert_encoding(), then iconv() - imapCharsetToUtf8(). ISO-8859-1 is read as Windows-1252, its superset, because mail labelled Latin-1 very often contains curly quotes and €, which only exist in 1252
Encoded headers (RFC 2047)	=?UTF-8?B?SsO2cmc=?= → Jörg	imapDecodeMimeHeader() - B and Q forms; spaces between two encoded words are dropped, as the RFC says
Encoded filenames (RFC 2231)	filename*=utf-8''%C3%9Cbersicht.pdf, split as filename*0*, filename*1*	imapPartFilename() / imapRfc2231Param()
Folder names (modified UTF-7)	Übergabe ↔ &ANw-bergabe	mb_convert_encoding(..., 'UTF7-IMAP') - imapFolderEncode() / imapFolderDecode()

TRAP: when both filename and filename* are present, filename* wins. Mail programs - FreeITSM's own mimeBuildMessage() included - send an ASCII fallback (filename="Bericht f__r Sie.pdf") and the real name. Reading the plain one first turned "Bericht für Sie.pdf" into "Bericht f__r Sie.pdf". Each side looked right on its own; only the send-then-read round trip in the live test found it.

---

7. How it plugs into what already exists

The client is new; nothing around it is.

One message shape for every provider. Microsoft (includes/mailbox_graph.php), Google (includes/gmail.php) and Basic IMAP all hand the importer the same Graph-like array. imapNormaliseMessage() builds it exactly as before:

[
  'id' => '1',                                   // the UID - post-actions use it
  'internet_message_id' => '<wiki-1@example.com>', // de-duplication; 'imap-<mailbox>-<uid>' if none
  'subject' => 'Printer jammed',
  'from' => ['emailAddress' => ['name' => 'Jörg', 'address' => 'jorg@example.com']],
  'toRecipients' => [...], 'ccRecipients' => [...],
  'receivedDateTime' => '2026-11-07T20:00:30Z',
  'body' => ['contentType' => 'html', 'content' => '<p>It is jammed.</p>'],
  'hasAttachments' => true,
  'attachments_inline' => [ /* Graph fileAttachment arrays, bytes already fetched */ ],
  'importance' => 'normal', 'isRead' => false,
]

• attachments_inline is the existing convention (from the Gmail connector) for "the bytes are already here". saveEmailToDatabase() in check_mailbox_email.php stores them directly instead of calling Graph. IMAP needs this because a part cannot be fetched again once the connection closes.
• imapStreamFor() - the same function, the same per-request cache keyed on mailbox and folder. It now returns a signed-in ImapClient with the folder selected, instead of an extension stream. The fetch and every post-action in one poll share one connection.
• imapVerifyFolder() - opens INBOX first (so a failure means credentials or TLS, not a bad folder name), lists folders, matches case-insensitively and with / for the server's delimiter, then opens the folder by the server's own name and reports EXISTS and the UNSEEN count.
• TLS policy - imapStreamContext() applies the same rule as every cURL call (includes/ssl.php): verify unless the install-wide SSL_VERIFY_PEER is off, and use PHP's configured CA bundle, or on Windows the bundled includes/cacert.pem. SMTP sending now uses the same context. Before this, imapSmtpSend() verified with no CA bundle, so on Windows PHP - which has none of its own - verifying a real certificate depended on php.ini.
• Sending is unchanged - imapSmtpSend() and includes/mime_message.php.

---

8. Security

Risk	Guard	Where
A value ending the string early and starting a command (a password with " or a line break)	Quoted strings escape " and \; anything with CR, LF, NUL or non-ASCII is sent as a literal, whose length is declared first - its content cannot be read as a command	imapString() - TRAP comment
Search criteria from outside	uidSearch() sends its argument as-is, so only ever pass a fixed key from code ('UNSEEN')	uidSearch() - TRAP comment
Password sent on an unencrypted connection	If the server advertises LOGINDISABLED, the client refuses to send it and says to choose SSL or STARTTLS	login()
Forged capabilities before STARTTLS	Discarded after the upgrade and asked for again	connect()
Unverified server	Certificates verified by default with the app's CA policy; turning verification off is the install-wide SSL_VERIFY_PEER switch, not a per-mailbox one	imapStreamContext()
A hostile or broken server announcing a huge literal	MAX_LITERAL = 128 MB; larger is refused with an error	ImapClient::MAX_LITERAL
A server that stops talking	Connect and every read time out (30 s); the error says so	readLogicalLine()
Errors leaking the password	Error messages contain the server's reply text, never the command sent	command()

Unencrypted connections are still allowed when the server allows them (encryption "none"), exactly as before, because the setting exists and some installs use it with a local mail server. A server that refuses passwords on an unencrypted line gets a clear message instead of a failed sign-in.

---

9. What behaves differently from the old extension

Deliberate, and each one tested:

	Before	Now
Needs	the imap extension	nothing
Folder names with accents (move-to folder, the mail folder)	passed as typed; IMAP expects modified UTF-7, so such names could fail to open or move	encoded properly
Attachment names in RFC 2231 form (filename*=)	only filename/name were read, so such files could arrive as "attachment"	decoded, continuations joined, filename* preferred
ISO-8859-1 mail	converted as Latin-1	converted as Windows-1252, so € and curly quotes survive
An unreadable Date header	strtotime() false → 1 Jan 1970	falls back to when the server received it, then to now
Verify's folder list	included \Noselect placeholders	real folders only
Delete / move on a UIDPLUS server	expunged every \Deleted message in the folder	expunges only that message
LOGINDISABLED	sign-in simply failed	explained: choose SSL or STARTTLS
SMTP on Windows	verified with no CA bundle	the bundled cacert.pem, as everywhere else

Error messages keep their old prefixes (IMAP connection failed, IMAP move to "X" failed), so logs and the mailbox screen read the same.

---

9b. Sending: errors by step, port rules, and the Test button (4.11.0)

Added after a real Zoho mailbox was set to STARTTLS on port 465: it collected mail, and every reply failed with SMTP error: expected 220, got "".

File	What
includes/mail_ports.php	New. mailPortExpectedEncryption() and mailPortMismatch() - SMTP 465 = SSL, 587 = STARTTLS; IMAP 993 = SSL, 143 = STARTTLS. Nothing else is judged. none on a STARTTLS port is allowed; none on an SSL port is not
includes/mailbox_imap.php	imapSmtpSend() names the failing step - no greeting, STARTTLS refused, SMTP sign-in failed, will not send as, refused the recipient - and appends the port hint. imapSmtpExpect() throws SmtpReplyException, whose ->reply is the server's own words ('' for silence)
includes/imap_client.php	connect() does the same for IMAP: no greeting, or a failed TLS handshake, gets the port hint
includes/mailbox_health.php	Two dismissible warnings on Basic IMAP mailboxes: smtp_port_encryption, imap_port_encryption
includes/mailbox_send.php	mailboxTestResolve() - which settings and which password the Test button uses
api/tickets/test_mailbox_send.php	New. The Test button. Same gate as Save and Verify (TICKETS_MAILBOXES). Sends through mailboxSendSimple() - the provider routes replies use - to the analyst who clicked, route mailbox_test in the send log
tests/mailbox-send-test.php	New. 36 checks - see Testing

Why the test goes to the analyst, and only them: an address field would turn the button into a way to send email as the company to anyone.

TRAP: the stored password is only reused for the server and username it was saved for. Basic IMAP is tested with the form's unsaved values, and a blank password means "use the saved one" - exactly as Save treats it. But Test sends that password to whatever SMTP server is typed. Reusing it for a changed server would let anybody with mailbox settings type a server of their own, click Test, and collect a password they are not allowed to see. So mailboxTestResolve() reuses it only when the SMTP server (case and spaces ignored) and the username are unchanged; otherwise it asks for the password to be typed. The same rule covers a separate SMTP login. tests/mailbox-send-test.php section 3 checks it, and was proved able to fail by running it against a copy with the rule removed - the stored password then went to smtp.attacker.example.

10. Traps

• BODY.PEEK is echoed back as BODY. Look up BODY[1.2], not BODY.PEEK[1.2].
• Never assume the next line is the answer. Read to the tag.
• Literals contain anything, CRLFs included. Read by count, never by line.
• EXPUNGE without UIDPLUS is folder-wide.
• An open folder only learns of new mail when it next talks to the server (NOOP is "anything new?"). The poll never meets this, because it opens a fresh connection each time.
• Docker's port forwarder accepts connections before the server inside is ready. A test against a just-started container can get an empty greeting; wait for the container's log.
• Port and encryption go together (includes/mail_ports.php). STARTTLS on 465 opens a plain connection to a server waiting for TLS; neither speaks, and it looks like a dead server.
• Do not reintroduce an imap_*() call. tests/docker-php-extensions.php fails if one appears, because imap is neither installed in nor excluded from the image.

Related

• Basic IMAP without the PHP extension - the decision
• Basic IMAP client - Testing
• Basic IMAP / SMTP Mailboxes - setting one up
• RFCs: 3501 (IMAP4rev1), 6851 (MOVE), 4315 (UIDPLUS), 2047 (encoded headers), 2231 (encoded parameters)