Reading Long Tickets Developer Guide

Reading Long Tickets — Developer Guide

The three automatic behaviours in the reading pane: shortening a long message, folding the older part of a long ticket, and flagging a message that has arrived before. All from discussion #104.

The AI half — the summary and the briefing — is in Ticket AI Reading — Developer Guide. The user-facing page is Reading Long Tickets.

🔒 THE INVARIANT: nothing on this page ever deletes content

Every behaviour documented here is presentation only. Read this before changing any of it:

• Nothing is removed from the database. No DELETE, no UPDATE of a body, no rewriting of emails.body_content. The three behaviours add a wrapper element, move a node, or set a flag on a response.
• Nothing is withheld from the browser. The full body of every message — including one flagged as a duplicate — is in the JSON and in the DOM. Collapsing sets a CSS class; folding calls appendChild into a hidden <div>. Both are one tap from reversed.
• Duplicate detection only ever flags. It sets same_as_id / same_kind on the response and never filters the array.
• Everything survives the feature being switched off. Turn all three settings off and every message renders exactly as it did before any of this existed, because none of it changed anything that was stored.

Why so absolute: deciding where quoted history ends is genuinely hard. Every mail client marks a quoted chain differently, plenty do not mark it at all, and a forwarded message can look identical to a reply. Any approach is right most of the time and wrong some of the time — so a destructive rule is wrong every so often, permanently and invisibly, while a presentational rule that is wrong costs one click. That is the entire reason this is built the way it is, and it is why a change here that starts dropping content is a change to the design, not an optimisation.

⚠️ The one boundary, so this guide does not overstate itself. The invariant covers everything from the stored message onwards. It does not describe ingestion: stripInboundThread() in api/tickets/check_mailbox_email.php cuts the quoted chain off an inbound reply before it is stored, so what reaches the database is the new part of the message only. That is a deliberate, long-standing choice and is out of scope here — but if you are chasing "where did the rest of the email go?", it is there and not in anything on this page.

The display-side twin, stripQuotedThread() in get_ticket_thread.php, is non-destructive like everything else here: it shapes the response, not the row.

---

Files

File	What it holds
includes/ticket_display.php	The eight settings, with defaults and clamping
assets/js/inbox.js	mcApply / mcSweep / mcOpened / mcRemember, and tgGroupOlder
assets/css/inbox.css	.mc-wrap, .mc-toggle, .tg-fold, .tg-toggle, .dup-note
api/tickets/get_ticket_thread.php	Near-duplicate detection, server-side
tickets/settings/manifest.php	The eight keys on the general tab
tickets/settings/index.php	The controls, plus load and save
tickets/index.php	Emits window.MESSAGE_COLLAPSE

---

🔑 Collapse by RENDERED HEIGHT, not by line count

The feature request asked for a line threshold. Measuring real inbound mail says that is the wrong unit.

On a live install: 122 inbound emails, averaging 8,337 characters of which only about 1,800 are visible text. 38 of them were laid out with <table> and 26 carried their own <style>. A vendor notification — "Your Microsoft invoice is ready" — was 57,750 characters, a handful of source lines, and rendered about a metre tall.

Any line-based rule gets that message exactly backwards. So the trigger is the height the browser actually lays out:

/* ⚠️ Measured AFTER the body is in the document. The HOST is what to
   measure: its height comes from its shadow content, so it reports the
   real rendered height while the shadow root itself has no scrollHeight
   and its first child is the injected <style> — which is 0 tall and would
   quietly make every message look short enough to leave alone. */
const full = Math.max(host.scrollHeight, host.offsetHeight);
const limit = MC.collapse_px || 264;

That comment is the bug that nearly shipped. Message bodies are isolated in a shadow root (see emailBodyHost), and measuring the shadow root gives you its first child — the injected <style> element, zero pixels tall. Every message would have measured as short and nothing would ever have collapsed, with no error anywhere.

Lines in the UI, pixels in the code

collapse_px is derived, never stored. The setting stays in lines because "collapse after about 12 lines" is a sentence an administrator can reason about and "collapse after 264 pixels" is not. One constant does the conversion:

/** Roughly what one line of message text occupies once rendered. */
const TICKET_COLLAPSE_LINE_PX = 22;

// …at the end of ticketDisplaySettings():
// The one place lines become pixels.
$defaults['collapse_px'] = $defaults['collapse_lines'] * TICKET_COLLAPSE_LINE_PX;

Settings are clamped, not trusted

$defaults[$k] = in_array($k, ['collapse_lines', 'group_show'], true)
    // Clamped, not trusted. 4 lines is a peephole and 80 is no
    // collapsing at all; both are worse than the default.
    ? max($k === 'group_show' ? 2 : 4, min(80, (int)$row['setting_value']))
    : (int)(bool)(int)$row['setting_value'];

The whole array is wrapped in try/catch returning defaults, because a reading pane that cannot reach the settings still has to render.

The settings reach the browser as one object

Emitted server-side rather than fetched, so the first paint is already correct:

// tickets/index.php
window.MESSAGE_COLLAPSE = <?php echo json_encode(ticketDisplaySettings(connectToDatabase())); ?>;

One object, read from includes/ticket_display.php, so the browser cannot disagree with the server about what the settings say.

The choke point

hydrateEmailBodies() is where every message body enters the DOM, so that is where both sweeps hang. Nothing else needs to know:

// Isolate each thread body in a shadow root (see emailBodyHost).
hydrateEmailBodies(container);
tgGroupOlder(container);

and mcSweep() walks the same hosts:

function mcSweep(root) {
    if (!MC.collapse_enabled || !root || !root.querySelectorAll) return;
    const hosts = root.querySelectorAll('.thread-message-body, .email-body-content');
    hosts.forEach((h, i) => mcApply(h, {
        newest: i === hosts.length - 1,
        id: h.closest('[data-email-id]') ? h.closest('[data-email-id]').getAttribute('data-email-id') : ''
    }));
}

newest is positional — the last host in document order — rather than a date comparison, because the thread is already rendered in order and re-deriving it from timestamps is one more thing that can disagree.

Guards worth keeping

if (host.dataset.mcDone) return;              // rendered twice; measure once
…
if (!forced && full <= limit + 40) return;    // a shade over is not worth a control

The + 40 slack matters: without it a message one line over the limit gets a Show more button that reveals one line, which is worse than no control.

Remembering what was opened

Per device, in localStorage, never on the server:

function mcRemember(id, open) {
    if (!MC.collapse_remember || !id) return;
    try {
        const set = mcOpened();
        open ? set.add(id) : set.delete(id);
        // Keep the last 400: a reading position from six months ago is not
        // worth carrying, and localStorage has a hard quota that throws.
        localStorage.setItem(MC_KEY, JSON.stringify([...set].slice(-400)));

Both the read and the write are in try/catch — localStorage throws in a private window and when the quota is hit, and a reading position is never worth breaking a page for.

---

Folding the older part of a long ticket

🔑 A long MESSAGE and a long TICKET are different problems

Eighty short messages defeat a per-message limit completely. Every one of them is under it, mcApply returns early on all eighty, and the ticket is still unreadable. This needs its own answer.

Move the whole message, not just the meta

older.forEach(meta => {
    const parts = [];
    let n = meta.previousElementSibling;
    if (n && n.classList.contains('thread-separator')) parts.push(n);
    parts.push(meta);
    n = meta.nextElementSibling;
    while (n && !n.classList.contains('thread-meta') && !n.classList.contains('thread-separator')) {
        const next = n.nextElementSibling;
        parts.push(n);
        n = next;
    }
    parts.forEach(el => fold.appendChild(el));
});

A "message" in the rendered thread is a .thread-meta block plus everything up to the next one — separator, meta, any duplicate note, and the body. Moving the meta alone leaves the bodies behind, orphaned under a heading that has gone.

⚠️ Note const next = n.nextElementSibling; before pushing. appendChild moves the node, so reading nextElementSibling afterwards reads it from its new parent and the walk stops after one element.

Folding one message saves nothing

const metas = [...container.querySelectorAll('.thread-meta')];
if (metas.length <= show + 1) return;          // folding one message saves nothing

The control states its count

btn.textContent = t('tickets.reading.older_messages').replace('{n}', older.length);
btn.setAttribute('aria-expanded', 'false');
fold.parentNode.insertBefore(btn, fold);
fold.hidden = true;

hidden rather than display: none, so it is one property to toggle and screen readers treat it correctly.

---

Flagging a message that has arrived before

Why server-side

It is O(n²) on the message count and get_ticket_thread.php already has the whole thread in hand. Doing it in the browser would mean shipping every body to compare it against every other body.

Fingerprint the VISIBLE text

$text = strtolower(trim(preg_replace('/\s+/u', ' ',
    html_entity_decode(strip_tags((string)($email['body_content'] ?? '')), ENT_QUOTES | ENT_HTML5, 'UTF-8'))));
$len  = mb_strlen($text);

Markup is not content. Two deliveries of the same message routinely differ in their HTML — a tracking pixel, a different Message-ID in a footer, one hop's worth of style rewriting — while reading identically. Comparing the rendered text is comparing the thing a human would call "the same message".

Two kinds of match

// Too short to say anything useful. "Thanks" is not a duplicate of
// "Thanks" in any sense worth acting on.
if ($len >= 120) {
    $hash = md5($text);
    $head = mb_substr($text, 0, 300);
    foreach ($fingerprints as $prev) {
        $kind = null;
        if ($prev['hash'] === $hash) {
            $kind = 'identical';
        } elseif ($prev['head'] === $head && $prev['len'] > 0
                  && abs($len - $prev['len']) / $prev['len'] <= 0.03) {
            $kind = 'near';
        }
        if ($kind !== null) {
            $email['same_as_id']   = $prev['id'];
            $email['same_as_time'] = $prev['time'];
            $email['same_kind']    = $kind;
            break;                       // the FIRST match is the original
        }
    }

near is the one that earns its keep. The common real case is somebody resending after hearing nothing, with "Resending as I have not heard back" added at the bottom — same opening, length within a few percent, and a completely different md5. An exact hash misses every one of those.

break on the first match is deliberate: the earliest matching message is the original, and the note should point at it rather than at the most recent copy.

It only ever flags

The message is returned in full. The browser folds it away regardless of height:

/* A message flagged as one that has arrived before is folded away whatever
   its height: its length is not the reason it is noise. */
const forced = host.classList.contains('mc-force');
if (!forced && full <= limit + 40) return;
…
const startOpen = !forced && ((opts.newest && MC.collapse_expand_newest) || mcOpened().has(id));

mc-force overrides both the height test and startOpen — including the "newest message is always open" rule, since a duplicate arriving last is exactly the case worth folding.

Two genuinely different messages that happen to open identically are a mild annoyance. A hidden one is not — hence flag, never remove.

---

Testing it

A clean database is the worst place to test any of this. The emails that break a reading pane are the ones a real service desk receives.

scripts/insert_messy_thread.sh                 # newest open ticket
scripts/insert_messy_thread.sh 109             # a specific ticket id
scripts/insert_messy_thread.sh --clean         # remove everything it made

It builds Outlook top-posts with no <blockquote> (which our quote stripping is known to miss), an EXTERNAL EMAIL banner, a four-paragraph disclaimer, a mobile signature, an out-of-office, a bounce with raw headers, plus an exact duplicate and a near-duplicate resend. Every row carries a [MESSY-TEST] subject prefix.

⚠️ Two things the script itself had to learn:

# ⚠️ sed, not `grep -P`: this environment reports "-P supports only unibyte
# and UTF-8 locales" and refuses.
DBPASS="$(sed -n "s/.*DB_PASSWORD'[^']*'\([^']*\)'.*/\1/p" c:/wamp64/db_config.php 2>/dev/null | head -1)"

# ⚠️ stderr is filtered, NOT discarded. Swallowing it made a failed INSERT
# look like a silent success under `set -e` — the exact trap this script is
# meant to help find in other people's data.
run() { "$MYSQL" -u "$DBUSER" -p"$DBPASS" "$DB" -sN -e "$1" 2>&1 | grep -v "insecure" || true; }

What to assert

Measuring "is anything wider than the screen?" is not enough — see Mobile Friendly Techniques §28. Drive the pane and check:

• .tg-fold exists, is hidden, and holds total − group_show messages
• the button reads "{n} older messages" and opens the fold on click
• .dup-note appears on both the exact duplicate and the resend
• the duplicates are inside .mc-wrap.mc-collapsed
• with the settings saved as 0: no fold, every message visible, no notes

That last one is the important one. A toggle that persists but does not take effect looks identical to a working one in the database.

Related pages

• Reading Long Tickets — the user-facing page
• Ticket AI Reading — Developer Guide
• Merging Tickets — Developer Guide
• Mobile Friendly Techniques