feat: Mail into Matrix and optional filing through Archivist#38
Merged
Conversation
ADR-010 names the recurring architecture - sources -> ledger -> projections - event-sourcing-flavored but not full ES. Key guarantee: the machine-derived vault is reproducible from its sources, indexed by the Matrix ledger. Reprocessing replays the source, never the vault file - and for a chat filing the source is the whole thread: the original message plus its reply chain (corrections), folded in order. Only user hand-edits are irreducible. dev.md carries the short rule; framework deferred until a third source earns it. The email-tools design doc is that third source.
himalaya needs no host resource and is the credentialed, network-facing piece - so it runs in a small mail container (egress scoped, creds from secrets), and the classify/file/route logic reuses the bot-runner pipeline via a mail bot (shared Maildir handoff). Records the verified himalaya envelope-list JSON contract and flags the message-read body schema as pin-on-install.
Email is one more ingestion source. Add a pure email_to_source mapper (body -> text, subject -> title, Message-ID -> an RFC 2392 mid: pointer for dedupe/ reprocess) and a capture_email entry that files it as an email-kind capture through the existing classify->mirror pipeline - so an email gets the same briefing + extracted action items as any document, no new intelligence. This is the pure, fully-tested core (no himalaya, no mailbox, no bot); it proves the ADR-010 seam that a new source slots into SourceContent. The mail container, stack mail CLI, mail bot + room routing follow.
The mail bot reads himalaya's Maildir files (plain RFC822) directly rather than himalaya's rendered JSON: add parse_email (stdlib email, bytes in so UTF-8 bodies decode) extracting subject/from/message-id/date/body, the stable Message-ID included. capture_email now carries the email's real Date as captured_at. Pinned himalaya 1.2.0's actual contract against a fabricated local maildir: the README was wrong (flag is -o json not --json; envelope list is a bare array with from/to objects using addr and no message-id; maildir needs maildirpp=true). So ingestion stays on RFC822 + stdlib, not himalaya's output.
…ding - Paths: <bucket>/emails/YYYY/MM/...; bucket from the room binding (family for a shared mailbox, person for a personal one); merge by date, account in frontmatter, no per-inbox subfolders. - Threading: one file per thread, keyed by the thread root and folded per ADR-010 (re-fetch + re-fold); each new message appends; distinguish an inbound reply (append) from a Matrix correction (rewrite). Thread-keying is day-one. - Summary gate: keep the body always; skip the LLM summary for short mail but still extract action items + tags. - Transport: the homebrew himalaya has no IMAP->Maildir sync, so sync is mbsync or stdlib imaplib behind the Maildir seam; himalaya stays for send + CLI. The seam makes the transport swappable, so himalaya's single-maintainer bus factor is not our exposure.
Spins up a throwaway GreenMail (auth-disabled, fabricated data), sends via SMTP, fetches with stdlib imaplib, and asserts parse_email round-trips the message faithfully - including a UTF-8 body. Proves the chosen ingestion transport (imaplib + parse_email, no himalaya/mbsync) against a real IMAP server. Self -contained: manages and tears down its own container; skipped without Docker.
parse_email now resolves the thread root (References[0], else In-Reply-To, else own Message-ID) - the day-one identity every message in a conversation folds into per ADR-010. Add MailFetcher: read-only stdlib imaplib fetch of new messages, deduped by Message-ID against a caller-supplied seen set so re-runs are idempotent (blocking I/O, the bot calls it via asyncio.to_thread). Verified against GreenMail: fetch + dedup + idempotent re-run.
Every message in a conversation now lands in a single vault entry keyed by the thread root, instead of one file per message. A reply appends a dated section; persons and tags union across the thread so indexing spans the whole exchange. Re-filing a message already in the file is a no-op (each section carries a mid: marker, so idempotency lives in the file). Email keeps action items (a school's 'return the form by Friday' is a task worth surfacing) unlike bookmarks, which drop them.
Adds a mermaid flow of fetch → thread_root → fold (first/append/idempotent) to the email design doc, updates the threading section to what landed (per-message append, frontmatter union, mid: marker idempotency), and fixes the stale /mail/ path to /emails/. Section headings use the vault's middot separator, not an em dash.
Rewrites the email architecture around two roles split on the credential boundary: a mail gateway (holds creds, posts to the room, SMTP-sends drafts) and the archivist (no creds, classifies and folds from the room). The Matrix room replaces the shared-Maildir handoff and becomes the durable source of record. Adds postmoogle's email-thread to m.thread mapping (threadify / nothreads / stripify) and a standardized twofold ingest event: a rendered body plus a dev.famstack.source block carrying raw_content and per-source fields (from, message_id, thread_root).
Corrects the mail bot from 'silent gateway' to a normal MicroBot subclass that can converse for its own configuration; the real invariant is credential isolation, not muteness. Records that the twofold source-event plumbing (post, m.thread, raw_content, recognize) lives in the bot framework so the mail bot and archivist share it rather than each reimplementing it.
Records that raw_content (email-scoped for now) holds the verbatim body, and attachments are posted as Matrix media in the same thread so the whole message is captured. Attachments reuse the archivist's existing _on_file path (PDFs to Paperless, images to the vault); no new attachment pipeline. Flags the attachment-noise filter (skip signatures/inline/tiny, plus an on/off toggle) as a build-time concern.
Email is an ingestion channel, so its primitives belong in stack.* where every surface imports them: the mail bot (core), the archivist (docs), the host CLI, and tests. Moves ParsedEmail/parse_email to stack.email_message and the IMAP MailFetcher to stack.mail_fetcher; extractors re-exports the parser for back-compat. No behaviour change.
Framework plumbing for ingest sources: posts a human-readable body plus a dev.famstack.source block (raw_content + per-source fields), optionally threaded under an m.thread root, and returns the event id. Any source (email now, scanners/webhooks later) reuses it instead of reimplementing the wire format.
Adds the MailBot (a second core-stacklet bot, discovered via a *.bot.toml sibling): it polls IMAP, strips quoted history/signatures with email-reply-parser so only the new message is posted, and posts a twofold source event threaded under the conversation's Matrix thread. The archivist files it from the room — one capture path. Config comes from stack.toml [mail] (rendered to env); passwords from the secret store, never from chat. Seen Message-IDs and thread roots persist so restarts stay idempotent.
Closes the loop: the archivist now recognises a dev.famstack.source message in a room and folds it through capture_email — the same capture path a pasted URL takes. Email files to the institutional bucket (<shared_bucket>/emails/), and the filing envelope rides onto the timeline for the deriver/reprocess. Only bot senders may emit source events (a family member can't spoof an ingest); raw email is treated as untrusted data, hardened further in the email security review.
Wires the user-facing config road: stack.toml [mail] + [[mail.accounts]] now render into MAIL_ACCOUNTS_JSON (host/user/folder/room per account) via _build_template_vars, mirroring AI_MODELS_JSON. IMAP passwords come from the secret store (mail__<NAME>_IMAP_PASSWORD) and are embedded in the rendered JSON the same way other container secrets reach .env — never written to stack.toml. The mail bot reads the embedded password (env var stays a manual/test override).
On room join the mail bot posts which mailbox + folder it delivers there (self-explaining UX), or says so when no mailbox routes to the room. The archivist now ignores plain messages from -bot senders (only their dev.famstack.source events are actionable), so the welcome isn't mistaken for a capture and bot-to-bot loops can't form.
The mail bot was discovered but skipped at launch because its login password (core__MAIL_BOT_PASSWORD) was never generated. Add it to core's [env].generate list alongside the stacker bot so the bot runner can create the @mail-bot account.
Subject on its own line with an envelope marker, sender + date in a blockquote, body in its own paragraph. Blank lines force real breaks (single newlines collapse in Matrix markdown, which made the first cut run subject and sender together). Sender shows display name + address when both are present.
The poller was started in on_first_sync, which the framework gates behind a .welcomed marker to fire a single time — so after the first launch the bot joined rooms but never polled. Move it to register_callbacks, which runs on every launch inside start()'s event loop.
…kdown Thread fold: the thread file is keyed by the thread-root hash, but the slug came from each message's title and drifted, so a reply spawned a second file. Now the mirror looks up the existing thread file by its hash suffix and appends there (same pattern the document mirror uses for renamed docs). HTML mail: an HTML-only body is converted to Markdown via html2text (links and headings preserved, images dropped), so the room and the vault get clean text instead of raw tags. Plain parts are still preferred when present.
… as a person Emails now carry their provenance: the mail bot stamps account + folder onto the dev.famstack.source event, and the archivist tags the vault entry with Sender: <addr>, Mailbox: <account>, Folder: <folder> — so you can filter all mail from a sender, a mailbox, or a folder. Also: email no longer falls the sender (the mail bot) in as a person when the classifier names none, so entries stop showing persons: Mail-bot.
A phishing mail hides a hostile URL behind friendly link text. defang_links now reveals the real URL next to the label and wraps every URL in a code span so neither Matrix nor Obsidian auto-links it — the reader sees where it points and can't click by reflex. Applied to the room message and the vault body; the verbatim raw_content on the source event stays faithful for reproducibility.
…ed content Frame the captured text (email body, web page, pasted note) as untrusted data in the classifier prompt: it is summarized, never obeyed. An injected "ignore the above" lands as content to describe, not a directive. JSON-mode output already bounds the escape surface; this closes the instruction channel. Also defang links in the model's own summary/facts on the email render path, so a phishing URL the model quotes stays readable but non-clickable (the verbatim body was already defanged).
Each poll downloaded the whole folder (SEARCH ALL + RFC822 per message), which doesn't survive a Gmail-scale inbox. Track a per-folder UID watermark (FolderCursor: uidvalidity + last_uid) and fetch only UIDs above it. The fetcher advances the watermark to the folder's highest UID; the bot rolls it back below a message whose Matrix post failed, so the failure and everything after it is re-fetched next poll — no silent drop. Message-ID dedup stays as the idempotency backstop across UIDVALIDITY resets and re-fetched batches.
A fresh account had only two choices: backfill the whole folder on first poll or nothing. Add an optional [[mail.accounts]] since = "YYYY-MM-DD" that compiles to a server-side IMAP SEARCH ... SINCE, so existing mail is ingested from a chosen date instead of the entire history. ANDs with the UID watermark, so the floor bounds the first backfill and steady-state polls stay incremental. Date conversion is locale-independent; a malformed value falls back to no floor.
Email thread files were <slug>-<hash>.md, so a month folder sorted alphabetically by subject. Prefix the date (YYYY-MM-DD-<slug>-<hash>.md) like the documents path already does, using the thread's first-message date so it stays stable as replies append. Thread lookup keys on the -<hash>.md suffix, so existing un-prefixed threads are still found and appended to.
A message missing a Date header fell through to today() downstream, so backfilling an old folder would stamp old mail with the date the bot happened to run. Fetch INTERNALDATE alongside RFC822 and use it as the date when the header is absent — the date the server received the mail, read in its own offset (host-TZ-independent). today() remains only as a last resort IMAP can no longer reach.
The UID watermark advances past the newest message, so after a narrow backfill (e.g. since=last week) widening the floor (since=this year) would fetch nothing — the older in-range mail sits below the watermark. Record the floor the watermark was built against; when the configured floor moves earlier, reset the watermark and re-scan the wider window. Message-ID dedup skips the already-filed mail. The floor is persisted so the widen survives a restart. Supports the 'start narrow, then widen' backfill workflow.
Attachments were dropped — the parser only kept the body. Extract named attachments (bytes verbatim) and re-post each as an m.file/m.image under the email's thread, with the subject as the caption for context. A new MicroBot.send_file uploads to the media repo and sends the event (framework plumbing). The archivist's existing file path then files them — vault summary/text extraction by default. Promoting an attachment to Paperless via an emoji reaction is a later iteration.
A bot-posted attachment rode the generic file path and got attributed to the mail bot (persons: Mail-bot, filed under mail-bot/). Mark each attachment with dev.famstack.attachment carrying the email provenance; the archivist files it with default_person=False and seed tags [email, Sender: <from>], so the bot is never the person. The bucket comes from the room's topic binding (a 'Family Email' topic room routes them to the shared family space).
Email text hardcoded family/emails/; now it reads the room's topic binding (like attachments already do) and nests under emails/, defaulting to the shared bucket when the room has no binding. A 'Family E-Mails' topic room files text and attachments under one bucket; the plain family room still lands in family/emails/. One placement rule for everything from a mail room.
Derive a bot-posted email's bucket from who is in the room, not the bot sender: - count humans excluding ALL bots (was only excluding self, so an email room with mail-bot + archivist-bot + one person mis-scored as shared) - a personal-scope topic nests under the room's sole human, not the bot - a non-topic room scopes by membership too, so a DM/private room (one human) files under that person and the shared family room files under the shared bucket The '-bot' convention moves to MicroBot.is_bot_user as the one framework definition; the scope->bucket mapping stays in the archivist (its sole consumer). Human-sent captures keep their existing sender-based routing.
Add an Email section (and a how-do-I entry): mail arrives as a tidy message, links are shown as non-clickable plain text, attachments are filed like any capture, and where mail files depends on who is in the room (family room -> shared, your own room/DM -> your name, topic room -> the topic). Notes the admin can backfill existing mail from a chosen date.
Add an Email section under Configuration: stack.toml [mail] + [[mail.accounts]] fields, the password in .stack/secrets.toml (mail__<NAME>_IMAP_PASSWORD), invite both bots, room-membership scoping, app passwords, since-backfill, read-only fetch, and link defanging.
…> Mail Carrier Account provisioning sets the display name only for bots without a session, so a rename never reached an already-created account. Bots now sync their configured display name to their Matrix profile on every launch (best-effort, only when it differs). Rename the mail bot's display name to 'Mail Carrier'; the id stays mail-bot (account, secret, and the -bot convention depend on it).
Adds a diagnostic that logs in to each configured account (read-only) and lists the server's real folder names + flags, plus the configured folder's message count. IMAP folder names often differ from the webmail labels (Gmail's [Gmail]/All Mail, a localized Gesendet, nested paths), so this is how you pick the right 'folder' value and debug a wrong host/password before the bot polls. Host command docker-execs into the bot-runner; passwords stay in the container env. MAIL_ACCOUNTS_JSON parsing is factored into a shared account_from_entry (bot + CLI use one parser).
Auth failures are ambiguous: a secret-store key that doesn't match the account name renders a blank password, which looks identical to a wrong password. Print a masked preview per account (first/last 2 chars + length; empty called out loudly; short secrets length-only) so the admin can confirm the secret actually handed over before chasing a password problem.
The command reads the rendered env, not stack.toml live, so an edit needs a restart to show. Frame the output as the bound config and remind to run 'stack restart core' to apply stack.toml edits — so a stale reading isn't mistaken for the command ignoring the config.
html2text doesn't always leave a blank line before a table, and python-markdown's tables extension only recognizes one that starts a fresh block — without it the table showed as a paragraph of raw | pipes in Element. Normalize the converted markdown to insert the missing blank line before each table header. Renders correctly in both Element and the Obsidian vault.
The room message dumped the whole email (signature, footer, tracking links) into the timeline. Post a compact card instead — subject, sender, date, attachment count — and put the full body as the first threaded reply, beside the attachments and any conversation replies. The card carries the machine raw_content, so the archivist files from it untouched and the visible card is free to be small. The full body sends with line_breaks (nl2br) so the email's own newlines survive markdown. MicroBot._send gains thread + line_breaks.
A family brain shouldn't fill with marketing. parse_email flags a message as noise from headers only (never body text, which would false-positive on personal mail mentioning 'unsubscribe'): List-Unsubscribe or List-Id (a newsletter/mailing list), Precedence: bulk, Auto-Submitted, or a machine sender (noreply@, mailer-daemon@, bounce@). The mail bot drops flagged mail before the room (marking it seen) when [mail] filter_noise is on; default true, set false to ingest everything. Idea adapted from the Hermes email gateway.
Add an email troubleshooting section to the admin guide covering the real gotchas (auth/app-password, archivist must be in the room, restart-after-config, folder names via stack core mail, since flooding, filter_noise, Element X thread beta). Note SSL/993 is the default in the [mail] example (plaintext only for a local/test server). Add an email line to the README capability list so the feature is discoverable.
Emails carry footer/logo/signature images and tracking pixels that aren't real attachments; auto-filing them dumped junk into Paperless. Drop them at parse: an image whose Content-ID is referenced inline in the HTML (cid:) is display chrome, not a file; a size backstop catches stray small icons. And restrict attachments to the types the pipeline handles well (PDF, image, txt, md), dropping zip/office/calendar parts. Genuine attachments are untouched.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Delegate IMAP inboxes to your Matrix chat: point a mailbox at a room and new
mail, with its attachments, shows up there for the family to read. Filing it
into the brain is an optional layer on top. We use it to feed our brain with family mails.
Core (just
core):with the full body in the thread (tables rendered, line breaks kept, links
defanged for anti-phishing).
filter_noise(default on): drops newsletters, automated, and bulk mail,header-based (never body scanning).
sincebackfill: ingest existing mail from a chosen date; widen to pull more.stack core mail: diagnostic to test the connection and list real folder names.Optional (with
docs+code): the archivist files each email and itsattachments into the vault/Paperless, scoped by room membership (a shared room
files to the family bucket; a private room or DM to that person).
Configured in
stack.toml [mail]; the password lives in the secret store (neverin config or chat), SSL by default. Email is treated as untrusted: links
defanged, the classifier hardened against prompt injection.