Release v0.0.11 · bchoor/mailcellar

Key takeaways

[Feature] Import mail from an mbox file: In Settings ▸ Accounts ▸ Add mbox archive, you can upload an mbox file from your browser and have its mail imported with no Gmail link, then follow progress on the Backfill page.
[Feature] [Improvement] Reconnect a failed Gmail account: In Settings ▸ Accounts, an account whose Google connection has failed now shows a Reconnect button that signs you in again and keeps all the mail already copied.
[Fix] Messages Gmail won't hand over no longer leave months stuck on "Almost": On the Backfill page, a month where Gmail lists a message but refuses to return it now counts as complete and shows a note like "1 Gmail won't give".
[Fix] Copied counts now include new mail: On the Backfill page, the copied count for a month that is already indexed now includes mail picked up by new-mail sync, so it matches the Searchable count.
[Improvement] Overview history by month for newer mailboxes: On the Overview page, the history chart is titled "Messages per month" and buckets by month when your mail spans under two years.

Settings ▸ Accounts has a new Add mbox archive option, and a failed account now offers Reconnect instead of "remove and add again".
Error messages for a failed account now say Reconnect.
On the Backfill page, an mbox archive shows as Importing or Imported, with "in the file" counts per month.
The Overview history chart switches to months when the mail spans under 24 months. Longer histories stay in years.
The Backfill coverage map's Complete green and Almost amber are more muted in light and dark mode. The intermediate shades were lowered to stay below Complete.

Messages Gmail lists but answers 404 or 400 for are now recorded as unavailable instead of being dropped silently.
Unavailable messages are subtracted from the month's expected count, so the month turns green. The map, totals, meter, presets, "All missing" and the backfill-missing action all follow this.
Backfilling an affected month again fetches only the missing message, and the month lands marked unavailable.
Reconnect reruns Google's consent with the account suggested. The fresh token is saved over the old one, so all copied mail stays.
When an account stops, the message now includes Google's error code (such as invalid_grant) and points to Settings ▸ Accounts.

The mbox import builds the search index month by month, newest first.
Months that are already indexed now add new-mail messages to their copied count. This fixes a mailbox that showed 2,156 of 2,199 copied while Searchable showed all 2,199.
A one-time repair fixes recent indexed months, and the status row is rebuilt once so the map and totals pick up the corrected counts.

The 5-minute watchdog alarm now goes idle. A census that stopped on an error and the 6-hour recount start only at a sync slot, and a census held for the daily limit no longer counts as running.
A month left showing Indexing no longer keeps the alarm on. The watchdog drops months whose index is already built.
Right after a reconnect, the next alarm retries the census and restarts the stopped sync.
New-mail sync adds about one extra write per month per batch to keep copied counts right. No new Gmail calls or backfill are needed.

mbox files upload straight to raw files (R2 objects) in 64 MB parts, 3 at a time, with retries.
An interrupted upload resumes: choose the same file again and only the missing parts go up. The only size limit is R2's 10,000 parts (about 640 GB).
The import packs messages into raw files and appends Basin rows. Rows are staged first, so a failed append is retried.
The upload and the packed files count on the backfill spend meter, which counts all bytes in the bucket.

An mbox import runs as a chain of Workflow runs:
- Scan reads the file in 8 MB slices, finds messages and their months, and skips Spam and Trash.
- Copy packs the messages into raw files and appends the Basin rows.
- Index builds the search index for each month.
Each import run makes one Durable Object call for progress, plus the call that launches the next run. The copy path makes no other reads or writes there.
After Workers Paid ends, Free budgets apply and imports run more slowly.
While an account is stopped, the watchdog logs one account-error line per alarm with the account, email and error. That is about 4 a day when idle.
The gmail-sync log line now lists what keeps the alarm on (busy) for each account: sync, queued months, census or index. It also reports whether the alarm is watching.

#113 mbox archive mailboxes: upload, import and Backfill progress (@bchoor)
#112 Count new mail as copied in months already indexed (@bchoor)
#111 Accounts: Reconnect a failed account instead of removing it (@bchoor)
#110 Accounts: log a broken Google connection, retry right after it's added again (@bchoor)
#109 Overview: show messages per month when the mail spans under two years (@bchoor)
#108 Watchdog: a month left showing Indexing no longer keeps the alarm on (@bchoor)
#107 Watchdog: log what keeps the 5-minute alarm on (@bchoor)
#106 Watchdog: stop the idle 5-minute alarm (@bchoor)
#105 Backfill map: mute Complete green and Almost amber, light and dark (@bchoor)
#104 Backfill: messages Gmail won't give count as unavailable, not missing (@bchoor)