Group messages into person+subject conversations

[Hohnik]

Parent

Spec: MailChat MVP — email-as-chat

What to build

As a user, my synced messages are grouped into chat-style conversations keyed by (person, subject) rather than shown as a flat list — replacing the flat per-account view from the previous ticket with a real conversation list and thread view, reading like an ongoing chat with each contact.

Acceptance criteria

• A Contact model exists, keyed by exact email address (case-insensitive), with display name kept up to date from the most recent message's From/To header; no automatic cross-address alias merging.
• A Conversation model exists, keyed by (contact, normalized_subject), with no account component (the same contact across multiple connected accounts merges into one conversation).
• The grouping logic resolves the primary correspondent from the relevant header (From for inbound, To for outbound — direction read from the Message.direction column) by parsing it with email.utils.getaddresses, lowercasing each extracted address, and excluding every address that matches a connected account (Account.address, also lowercased). Cc is never consulted for this resolution — CC'd participants are stored on the message but never affect grouping. If exactly one non-account address remains, it is the primary correspondent. If more than one remains (e.g. an outbound message to two external people), the first non-account address in header order is used — no fan-out to multiple conversations, no group-chat model. If zero remain (mail exchanged only between your own connected accounts), the message groups under one global sentinel Contact — address me@mailchat.local, display name "Me" — rather than being skipped, still split into separate conversations by subject like any other contact. Excluded/unselected addresses are not otherwise persisted beyond the raw to_header/cc_header/from_header already stored on Message. (Decided in Decide primary-correspondent resolution for multi-address To/Cc headers.)
• Message gains a direction column ("inbound" / "outbound"), set by sync from the source IMAP folder — INBOX → inbound, the account's configured Sent folder → outbound — not inferred from headers at read time. (Decided in Decide reply send-from account and how sent replies re-enter the conversation.)
• Message gains a read-state column is_read (bool) plus the IMAP addressing needed to write the flag back: imap_uid (int), imap_folder (str), and imap_uidvalidity (int), all set at ingest. Conversation gains no read column — conversation-level read state is always derived from its messages, so there is one source of truth that cannot drift. (Decided in Decide mark-read granularity and trigger.)
• The IMAP fetch changes from (RFC822) to (UID FLAGS BODY.PEEK[]) and keeps readonly=True — belt and braces, because the pull must never set \Seen as a side effect of reading. A non-PEEK body item (RFC822, BODY[...]) makes the server set \Seen on exactly the messages fetched, so under a writable SELECT every new message would be marked read within one poll interval of arriving, before the user ever opened it — and since the server is authoritative, sync would be wiping MailChat's own unread state. readonly=True alone prevents this, but BODY.PEEK[] makes the safety a property of the fetch itself rather than something that breaks silently if readonly is ever flipped for an unrelated reason. Note the response item comes back keyed BODY[] rather than RFC822; the raw bytes are identical, so parse() is unaffected. imap_uidvalidity is read from the SELECT response. The server's \Seen flag is authoritative and Message.is_read is a cache of it, seeded at ingest. (Decided in Decide mark-read granularity and trigger #17.)
• normalize() no longer returns an already-known message untouched: for messages inside the fetch window it refreshes is_read from the pulled \Seen flag, overwriting the local value unconditionally — the server always wins and there is no pending/dirty state — and rewrites imap_uid/imap_uidvalidity so a UID left stale by a UIDVALIDITY change heals on the next poll. Known limitation: the refresh only reaches the messages each poll fetches per folder (currently the newest 10), so a message read in another client after it falls out of that window stays unread in MailChat indefinitely. (Decided in Decide mark-read granularity and trigger #17.)
• Subject normalization repeatedly strips Re/Aw/Fwd/Fw/Wg tokens (case-insensitive) followed by a colon, wherever they occur in the subject string (not anchored to the start), then collapses whitespace and trims; applied idempotently.
• Message-ID/In-Reply-To/References are parsed and stored on the Message but are not consulted by the grouping logic.
• The normalizer (from the previous ticket) now finds-or-creates the Contact and Conversation for each incoming message using this grouping logic, instead of leaving messages ungrouped.
• Contact/Conversation find-or-create is guarded by a single in-process asyncio.Lock covering the entire lookup-then-create sequence for both models within one normalize() call (not two separate locks), with the transaction committed before the lock releases — so the concurrent per-account sync tasks in sync/worker.py can never create duplicate Contact/Conversation rows for the same contact/conversation key. No DB-level unique constraint is required as a backstop; this is a single-process app. (Decided in Decide concurrency safety for cross-account Contact/Conversation creation.)
• The flat per-account message list is replaced with a chat-style conversation list (grouped by contact/subject) and an individual conversation thread view showing messages in chronological order, both htpy-rendered and updating via hx-trigger polling.
• The conversation list shows an unread count badge per conversation, counting that conversation's messages with is_read = 0 and direction = 'inbound'. Unread is an inbound-only concept in the UI — is_read still mirrors the server faithfully on outbound rows, but they never contribute to a badge, so a reply you sent can never make its own conversation look unread (some servers store sent mail without \Seen). (Decided in Decide mark-read granularity and trigger #17.)
• Tests: grouping-logic tests using real fixtures — a recreation of the multi-message Louis.de DL-4916597 thread (proving non-anchored subject stripping unifies all its variants into one conversation) and a message with a CC'd participant (e.g. the Horbach example, To one person / Cc another, proving CC doesn't affect grouping) — mocking only the IMAP connection boundary, asserting on resulting Conversation/Message/Contact rows in a real temporary/in-memory SQLite database; htmx route tests asserting the conversation list and thread view render the expected grouped output. Concurrency: a test proving two accounts syncing the same new contact at the same time still produce exactly one Contact and one Conversation row (mirrors the pattern already used for the timing/failure-isolation tests in tests/test_sync/test_worker.py). Read state: a test that ingest seeds is_read, imap_uid, imap_folder and imap_uidvalidity from the fetched flags/SELECT response; a test that a second poll overwrites is_read on an existing row when the server flag has changed since (proving the server wins and that normalize() no longer leaves known messages untouched); a test asserting the fetch is issued as (UID FLAGS BODY.PEEK[]) against a readonly=True SELECT — pinning both halves of the guard, since either alone would pass a weaker assertion while leaving syncing able to set \Seen as a side effect; and a conversation-list route test asserting the badge counts inbound-unread messages only and ignores an unread outbound one.

Blocked by

• Sync mail into a flat per-account message list
• Decide primary-correspondent resolution for multi-address To/Cc headers