feat(plugin-email,platform-objects): sys_email 加 headers_json + 有界 attachments_json —— 自定义 header 与小附件取得队列投递资格 (#5177)

.changeset/sys-email-headers-attachments.md

@@ -0,0 +1,48 @@
+---
+"@objectstack/platform-objects": minor
+"@objectstack/plugin-email": minor
+---
+
+feat(plugin-email,platform-objects): `sys_email` carries headers and small attachments, so those messages become durably deliverable (#5177)
+
+Durable email delivery works from the **row**, not from the in-memory message:
+`send()` publishes an `{ rowId }` job (#5160), the boot sweep re-reads rows
+(#5161), and both end at `rowToNormalized`. So anything a `sys_email` row could
+not carry, a row-based delivery would have dropped — and custom headers and
+attachments were exactly that. The honest workaround was to refuse: a message
+with either was pushed back onto inline delivery so that it would at least go
+out whole, which closed the durable path to precisely the mail most worth
+making durable (a signed receipt, a `List-Unsubscribe` header, an invoice PDF).
+
+`sys_email` now has two columns, and those messages are queueable.
+
+**`headers_json`** — the custom headers, as a JSON object. Written in both
+delivery modes (it is audit evidence as much as delivery input) and rebuilt on
+read. Headers are no longer a reason to fall back to inline delivery.
+
+**`attachments_json`** — attachments as a JSON array of
+`{ filename, contentType?, size, hash, cid?, contentForm, inline?, storageKey? }`,
+content base64 in `inline`. Written when the **combined raw size of one
+message's attachments is within `SYS_EMAIL_ATTACHMENT_LIMIT_BYTES` (256 KiB,
+exported from `@objectstack/plugin-email`)** — worst case ~350 KB of base64, so
+a row stays bounded. Both arms of the declared `content: string | Buffer`
+contract round-trip as the arm they were sent as: restoring a text attachment
+as a Buffer would silently drop `charset=utf-8` from its MIME part and let the
+recipient's client mis-decode a UTF-8 file, so `contentForm` records which one
+it was. `cid` travels too — an inline `<img src="cid:…">` is unusable without
+it.
+
+**Over the limit, nothing changes.** The message is delivered inline exactly as
+before, whole, and the row stores no attachment content; the reason is stated
+at `info` (a bound, not a degradation — the worst outcome is today's
+behaviour). Out-of-row storage for large attachments is #5172; `storageKey` is
+declared now so that lands as a new *producer* rather than a data migration.
+
+Rows written before these columns exist read exactly as they did. A column that
+is present but does not describe what it claims — malformed JSON, a size or
+hash that disagrees with the content, a missing `contentForm` — is **rejected**,
+and the row lands at `failed` carrying the reason, rather than being delivered
+with a part quietly missing.
+
+The `sys_email` schema change is additive (two optional textarea columns); no
+migration is required and default inline delivery is unchanged.

packages/platform-objects/src/apps/translations/en.objects.generated.ts

@@ -2045,6 +2045,14 @@ export const enObjects: NonNullable<TranslationData['objects']> = {
       body_html: {
         label: "Body (HTML)"
       },
+      headers_json: {
+        label: "Headers (JSON)",
+        help: "Custom headers supplied to IEmailService.send, as a JSON object of name → value. Written in both delivery modes (it is audit evidence as much as delivery input). Absent on rows written before this column existed, which read back as \"no custom headers\"."
+      },
+      attachments_json: {
+        label: "Attachments (JSON)",
+        help: "Attachments as a JSON array of { filename, contentType?, size, hash, cid?, contentForm, inline?, storageKey? }, with content base64 in `inline`. Written only when the combined raw size is within the plugin-email budget (SYS_EMAIL_ATTACHMENT_LIMIT_BYTES, 256 KiB — ~350 KB of base64 at worst); a larger message is delivered inline and stores nothing here, so the row stays bounded. `storageKey` (out-of-row content) has no producer yet — objectstack#5172."
+      },
       status: {
         label: "Status",
         help: "Lifecycle state — queued by IEmailService.send before transport call",

packages/platform-objects/src/apps/translations/es-ES.objects.generated.ts

@@ -2045,6 +2045,14 @@ export const esESObjects: NonNullable<TranslationData['objects']> = {
       body_html: {
         label: "Contenido (HTML)"
       },
+      headers_json: {
+        label: "Cabeceras (JSON)",
+        help: "Cabeceras personalizadas facilitadas a IEmailService.send, como un objeto JSON de nombre → valor. Se escriben en ambos modos de entrega (son tanto evidencia de auditoría como entrada de la entrega). Ausentes en las filas escritas antes de que existiera esta columna, que se leen como «sin cabeceras personalizadas»."
+      },
+      attachments_json: {
+        label: "Adjuntos (JSON)",
+        help: "Adjuntos como un array JSON de { filename, contentType?, size, hash, cid?, contentForm, inline?, storageKey? }, con el contenido en base64 en `inline`. Solo se escribe cuando el tamaño bruto combinado está dentro del presupuesto de plugin-email (SYS_EMAIL_ATTACHMENT_LIMIT_BYTES, 256 KiB — ~350 KB de base64 en el peor caso); un mensaje mayor se entrega en línea y aquí no se almacena nada, de modo que la fila permanece acotada. `storageKey` (contenido fuera de la fila) todavía no tiene productor — objectstack#5172."
+      },
       status: {
         label: "Estado",
         help: "Estado del ciclo de vida; se pone en cola mediante IEmailService.send antes de la llamada al transporte.",

packages/platform-objects/src/apps/translations/ja-JP.objects.generated.ts

@@ -2045,6 +2045,14 @@ export const jaJPObjects: NonNullable<TranslationData['objects']> = {
       body_html: {
         label: "本文（HTML）"
       },
+      headers_json: {
+        label: "ヘッダー（JSON）",
+        help: "IEmailService.send に渡されたカスタムヘッダーを、name → value の JSON オブジェクトとして保持します。両方の配信モードで書き込まれます（配信の入力であると同時に監査証跡でもあるため）。この列が存在する前に書き込まれた行には値がなく、「カスタムヘッダーなし」として読み戻されます。"
+      },
+      attachments_json: {
+        label: "添付ファイル（JSON）",
+        help: "添付ファイルの JSON 配列で、要素は { filename, contentType?, size, hash, cid?, contentForm, inline?, storageKey? } の形をとり、内容は base64 で `inline` に格納されます。添付の合計生サイズが plugin-email の予算（SYS_EMAIL_ATTACHMENT_LIMIT_BYTES、256 KiB —— 最悪でも base64 で約 350 KB）に収まる場合にのみ書き込まれます。上限を超えるメッセージはインライン配信され、この列には何も保存されないため、行のサイズは有界に保たれます。`storageKey`（行外の内容）にはまだプロデューサーがありません —— objectstack#5172。"
+      },
       status: {
         label: "ステータス",
         help: "ライフサイクル状態 — トランスポート呼び出し前に IEmailService.send がキューに入れます",

packages/platform-objects/src/apps/translations/zh-CN.objects.generated.ts

@@ -2045,6 +2045,14 @@ export const zhCNObjects: NonNullable<TranslationData['objects']> = {
       body_html: {
         label: "正文（HTML）"
       },
+      headers_json: {
+        label: "邮件头（JSON）",
+        help: "传给 IEmailService.send 的自定义邮件头，以 name → value 的 JSON 对象存储。两种投递模式下都会写入（它既是投递输入，也是审计证据）。在该列出现之前写入的行没有此值，读回时按「无自定义邮件头」处理。"
+      },
+      attachments_json: {
+        label: "附件（JSON）",
+        help: "附件的 JSON 数组，元素形状为 { filename, contentType?, size, hash, cid?, contentForm, inline?, storageKey? }，内容以 base64 存放在 `inline` 中。仅当附件合计原始大小在 plugin-email 的预算内（SYS_EMAIL_ATTACHMENT_LIMIT_BYTES，256 KiB —— 最坏约 350 KB 的 base64）时才写入；超出上限的邮件改走内联投递，此列不落任何内容，因此行体积有界。`storageKey`（行外内容）目前还没有生产者 —— 见 objectstack#5172。"
+      },
       status: {
         label: "状态",
         help: "生命周期状态——在调用传输层之前由 IEmailService.send 排队",

packages/platform-objects/src/audit/sys-email.object.ts

@@ -108,6 +108,35 @@ export const SysEmail = ObjectSchema.create({
       group: 'Content',
     }),
 
+    // ── Message parts a row must carry to be deliverable (#5177) ─
+    // Delivery of a queued / stranded / app-inserted message happens FROM
+    // THIS ROW, not from the in-memory message: `send()` publishes an
+    // `{ rowId }` job (#5160) and the boot sweep re-reads rows (#5161). Any
+    // part of the message the row cannot carry is therefore a part a durable
+    // delivery silently drops — which is why messages with headers or
+    // attachments used to be pushed back onto inline delivery instead.
+    headers_json: Field.textarea({
+      label: 'Headers (JSON)',
+      required: false,
+      description:
+        'Custom headers supplied to IEmailService.send, as a JSON object of name → value. '
+        + 'Written in both delivery modes (it is audit evidence as much as delivery input). '
+        + 'Absent on rows written before this column existed, which read back as "no custom headers".',
+      group: 'Content',
+    }),
+
+    attachments_json: Field.textarea({
+      label: 'Attachments (JSON)',
+      required: false,
+      description:
+        'Attachments as a JSON array of { filename, contentType?, size, hash, cid?, contentForm, '
+        + 'inline?, storageKey? }, with content base64 in `inline`. Written only when the combined raw '
+        + 'size is within the plugin-email budget (SYS_EMAIL_ATTACHMENT_LIMIT_BYTES, 256 KiB — ~350 KB of '
+        + 'base64 at worst); a larger message is delivered inline and stores nothing here, so the row '
+        + 'stays bounded. `storageKey` (out-of-row content) has no producer yet — objectstack#5172.',
+      group: 'Content',
+    }),
+
     // ── Delivery state ───────────────────────────────────────────
     status: Field.select(
       ['queued', 'sent', 'failed'],

packages/plugins/plugin-email/src/email-plugin.outbox-sweep.test.ts

@@ -17,6 +17,7 @@ import { assertEngineDeleteDispatch } from '@objectstack/objectql';
 import { DbQueueAdapter } from '@objectstack/service-queue';
 import { EmailServicePlugin } from './email-plugin.js';
 import { EmailService, EMAIL_SEND_QUEUE } from './email-service.js';
+import { encodeAttachmentsForRow, encodeHeadersForRow } from './sys-email-payload.js';
 
 // ── harness ────────────────────────────────────────────────────────────────
 
@@ -172,8 +173,32 @@ async function boot(opts: BootOpts = {}) {
     created_at: createdAt,
   });
 
+  /**
+   * The same crash, for a message that carried custom headers and a small
+   * attachment (#5177) — the columns are written exactly as `send()` writes
+   * them, via the same encoders.
+   */
+  const strandWithParts = (id: string, createdAt: string) => {
+    const encoded = encodeAttachmentsForRow([
+      { filename: '对账单.txt', content: '金额:¥1.00', cid: 'stmt@inline' },
+    ]);
+    if (encoded.kind !== 'inline') throw new Error(`fixture is not storable: ${encoded.kind}`);
+    engine.seed('sys_email', {
+      id,
+      from_address: 'no-reply@example.test',
+      to_addresses: 'user@example.test',
+      subject: `Stranded ${id}`,
+      body_text: 'hello',
+      headers_json: encodeHeadersForRow({ 'X-Campaign': 'spring' }),
+      attachments_json: encoded.json,
+      status: 'queued',
+      attempt_count: 0,
+      created_at: createdAt,
+    });
+  };
+
   return {
-    plugin, ctx, engine, adapter, clock, transport, strand,
+    plugin, ctx, engine, adapter, clock, transport, strand, strandWithParts,
     service: () => services.email as EmailService,
     sysEmail: () => engine.rows('sys_email'),
     jobs: () => engine.rows('sys_job_queue'),
@@ -221,6 +246,22 @@ describe('boot sweep — inline delivery', () => {
     expect(errorLines(h.ctx).join('\n')).toMatch(/never reached a recipient/);
   });
 
+  it('re-delivers a stranded row WITH its headers and attachment, inline too (#5177)', async () => {
+    // Inline mode reaches the transport straight from the sweep, so this is
+    // the shortest path from a persisted row to the wire — and the one that
+    // proves the columns, not the queue, are what carries the parts.
+    const h = await boot();
+    h.strandWithParts('row-rich-inline', ago(min(30)));
+
+    const swept = await h.ready();
+
+    expect(swept).toMatchObject({ scanned: 1, sent: 1 });
+    expect(vi.mocked(h.transport.send).mock.calls[0][0]).toMatchObject({
+      headers: { 'X-Campaign': 'spring' },
+      attachments: [{ filename: '对账单.txt', content: '金额:¥1.00', cid: 'stmt@inline' }],
+    });
+  });
+
   it('does not touch a row that was inserted seconds ago', async () => {
     // Another instance is delivering it right now; a boot must not race it.
     const h = await boot();
@@ -259,6 +300,24 @@ describe('boot sweep — durable queue delivery', () => {
     expect(h.jobs()[0]).toMatchObject({ status: 'completed' });
   });
 
+  it('re-delivers a stranded row WITH its headers and attachment (#5177)', async () => {
+    // A crash must not silently downgrade the message. Before #5177 the row
+    // had nowhere to keep either part, so a swept row was necessarily sent
+    // stripped — the loss looked exactly like a successful delivery.
+    const h = await boot({ queue: true, plugin: { queueDelivery: true } });
+    h.strandWithParts('row-rich', ago(min(30)));
+
+    await h.ready();
+    await h.adapter.pollOnce();
+
+    expect(h.sysEmail()[0]).toMatchObject({ id: 'row-rich', status: 'sent' });
+    expect(h.transport.send).toHaveBeenCalledTimes(1);
+    expect(vi.mocked(h.transport.send).mock.calls[0][0]).toMatchObject({
+      headers: { 'X-Campaign': 'spring' },
+      attachments: [{ filename: '对账单.txt', content: '金额:¥1.00', cid: 'stmt@inline' }],
+    });
+  });
+
   it('collapses onto an existing pending job instead of racing a second worker', async () => {
     // The other half of a crash: the row AND its job survived (the process died
     // between publishing and the worker running). Re-publishing a second job

packages/plugins/plugin-email/src/email-plugin.queue-delivery.test.ts

@@ -225,6 +225,43 @@ describe('queue delivery — round trip', () => {
     expect(h.jobs()[0]).toMatchObject({ status: 'completed', attempts: 1 });
   });
 
+  it('carries custom headers and a small attachment all the way to the worker (#5177)', async () => {
+    // The capability #5177 adds, end to end: these messages used to be pushed
+    // back onto inline delivery because a row could not rebuild them, so the
+    // durable path was closed to exactly the mail most worth making durable.
+    const h = await boot({ plugin: { queueDelivery: true } });
+    await h.ready();
+
+    const res = await h.service().send({
+      to: 'a@b.com',
+      subject: '对账单',
+      text: 'hello',
+      headers: { 'X-Campaign': 'spring', 'List-Unsubscribe': '<mailto:u@example.test>' },
+      attachments: [
+        { filename: '对账单.txt', content: '金额:¥1.00' },
+        { filename: 'logo.png', content: Buffer.from([0x89, 0x50, 0x4e, 0x47]), contentType: 'image/png', cid: 'logo@inline' },
+      ],
+    });
+
+    expect(res.status).toBe('queued');
+    expect(h.transport.send).not.toHaveBeenCalled();
+    expect(h.sysEmail()[0].headers_json).toBeTruthy();
+    expect(h.sysEmail()[0].attachments_json).toBeTruthy();
+
+    await h.adapter.pollOnce();
+
+    expect(h.sysEmail()).toHaveLength(1);
+    expect(h.sysEmail()[0]).toMatchObject({ id: res.id, status: 'sent' });
+    const delivered = vi.mocked(h.transport.send).mock.calls[0][0];
+    expect(delivered.headers).toEqual({ 'X-Campaign': 'spring', 'List-Unsubscribe': '<mailto:u@example.test>' });
+    // Both content forms come back as the arm they were sent as.
+    expect(delivered.attachments[0]).toEqual({ filename: '对账单.txt', content: '金额:¥1.00' });
+    expect(delivered.attachments[1]).toMatchObject({
+      filename: 'logo.png', contentType: 'image/png', cid: 'logo@inline',
+    });
+    expect(Buffer.compare(delivered.attachments[1].content, Buffer.from([0x89, 0x50, 0x4e, 0x47]))).toBe(0);
+  });
+
   it('SMTP 535: the queue retries with backoff, exhausts, and DLQs — one row throughout', async () => {
     // The regression the issue names: the old subscriber called `send()`, so
     // each redelivery INSERTED a new sys_email row. Five attempts, five rows,