feat(plugins): encoding and byte order mark options for CSV export

[datlechin]

Fixes #2534.

CSV export had no concept of output encoding. Every byte it wrote went through String.toUTF8Data(), a no-argument PluginKit extension hardcoded to .utf8 and shared by seven plugins, so a file for a tool that expects Windows-1252 could not be produced and the byte order mark Excel on Windows needs to read UTF-8 was never written. A value the encoding could not represent had nowhere to be reported.

What this adds

An Encoding picker on the CSV export options (UTF-8, ISO Latin 1, Windows-1252, spelled the way the CSV import picker already spells them), an Include byte order mark checkbox below it, and a warning that names the characters the encoding could not represent instead of writing ? in silence.

The default is unchanged: UTF-8, no mark. CSVExportBytesTests.defaultExportIsPlainUTF8 asserts the stock export is byte-for-byte what it was.

The warning arrives through ExportFormatResult.warnings, which already reaches an NSAlert that retitles itself "Export completed with warnings", switches to .warning style, and withholds the "Do not show this again" checkbox. XLSXExportPlugin, SQLExportPlugin and ParquetExportPlugin already use it.

Measured facts the design rests on

Each of these was measured with a swiftc harness on this toolchain, not read off a doc page.

• String.data(using:) is all or nothing: one unrepresentable scalar returns nil for the whole string. allowLossyConversion writes the substitute and reports nothing.
• Sweeping every scalar from U+0020 to U+10FFF for both isoLatin1 and windowsCP1252, the only byte a lossy conversion ever emits is 0x3F, and it is never a CSV metacharacter. A substitution cannot corrupt field structure.
• Foundation composes before it converts: a followed by U+0301 encodes to 0xE1 in ISO Latin 1 while U+0301 alone does not. The scan therefore walks grapheme clusters, not scalars, or it would name characters the file kept intact.
• .utf16, .utf32 and .unicode prepend a mark to the result of every data(using:) call ("A" gives FF FE 41 00, then "B" gives FF FE 42 00). A row-per-call writer using one of those would write a mark before every row. Only explicit-endian variants are safe, which is why PluginTextEncoding documents that constraint.
• NSString.getBytes(...remaining:) converts the longest encodable run and reports where it stopped, so the scan skips whole runs instead of testing one character at a time. On a 200 KB field holding one unrepresentable character: 0.33 ms against 26 ms for a per-character walk, with identical results on all 22 cross-check cases.

Why no UTF-16

The reporter's BOM line names UTF-16, and it was deliberately left out. Compiled against the real CSVDialect and CSVStreamingParser sources, a UTF-16 LE CSV with a mark parses back as mojibake (name reads as 渀愀洀攀) plus a spurious empty row, because the parser is byte-oriented: it ends a row on a single 0x0A and a field on a single delimiter byte, applying the encoding only when decoding the resulting run. UTF-16 BE round-trips clean. Shipping UTF-16 LE export would mean writing files TablePro's own CSV inspector and auto-detecting importer misread, and shipping BE alone is incoherent. The parser defect predates this change (the inspector already offers both variants) and is written up below.

The repair that had to land first

CSVExportOptions was the only export options struct in the repo with no custom init(from:). Measured: adding any non-optional field makes JSONDecoder throw keyNotFound on every stored payload, PluginSettingsStorage.load answers a throwing decode with nil, and loadSettings leaves the defaults. Shipping the two new fields without that init would have silently reset every user's saved delimiter, quote, line break and decimal choice. SQLExportModels, MarkdownExportModels, JSONExportModels, HTMLExportModels, XMLExportModels and ParquetExportModels all already had one.

Two released defects fixed on the way

Both are in the file and view this change edits, and leaving either would have shipped a new control that behaved differently from the one beside it.

• The Quote menu never localized. It rendered Text(handling.rawValue), which resolves to the verbatim String overload; the displayName that would have localized it was dead code and pointed at Bundle(for: CSVExportPlugin.self), a .tableplugin that ships no catalog and returns the key with no fallback. The three translations were already in the catalog and were never asked for.
• The # Table: comment line in a multi-table export hardcoded \n while every other write used the chosen line ending, so a CRLF export carried one lone LF per table and a reader splitting on CRLF joined the comment onto the header row.

Review

Codex was out of credits (You've hit your usage limit ... try again at Sep 7th), so /code-review high read the diff instead. It returned seven findings and all seven were acted on:

• The export snapshotted the encoding for the mark and the warning but re-read settings per line. One plugin instance serves every window, so changing the picker in another window's dialog could switch encoding mid-file. The whole options struct is now captured once, which also closes the same hole for the delimiter and the line ending.
• The unrepresentable scan ran on every line with nothing to stop it: roughly 3 minutes of pure scanning on a million-row CJK table, long after the report held every character it would ever list. The report now saturates and the encoder takes detectingUnrepresented: to stop paying for an answer it has.
• The warning counted distinct characters while reading as a count of values. It names the characters and no longer counts.
• An unrepresentable control character was appended with no glyph, ending the alert at the colon. Unprintables render as U+0081.
• The five new user-facing strings were missing from Localizable.xcstrings, which would have shipped English into the ko, tr, vi, zh-Hans and zh-Hant builds. They are in the catalog and translated; StringCatalogIntegrityTests passes on them and coverage holds at 5078/5080, the same two format-only strings as before.
• The test harness documented an exclusive window it did not have, since an actor is reentrant across await; a second call could restore the first call's test options into the developer's real settings. Runs are now chained.

What was built and tested

Step	Result
verify.sh build	PASS
verify.sh build CSVExport	PASS
verify.sh test (5 suites, incl. StringCatalogIntegrityTests)	PASS, 36 of 36
verify.sh lint (absolute paths)	PASS, 0 violations
verify.sh docs	PASS
verify.sh abi vs merge base	48 additions, 0 removals

The ABI diff is purely additive: a new enum, two new structs and a new caseless enum, with no existing signature touched. No currentPluginKitVersion bump and no registry re-release. CSVExportPlugin is bundled, so it always ships against a matching PluginKit, and no registry plugin references the new symbols.

verify.sh plugins fails locally on macro expansion @TaskLocal:1:2: error: unknown attribute 'usableFromInlinenonisolated' from oracle-nio. That is the only error in 8,730 lines of log, it is a known incompatibility between the vendored fork and the local Xcode 27 beta, and it blocks the aggregate for any change under Plugins/. The changed plugin target was built on its own instead; CI runs the aggregate on its own toolchain.

No TableProUITests coverage was added. The export sheet needs a live connection, and none of the four existing CSV export options has UI automation either, so this stays consistent with the rest of the pane rather than adding a lone flaky case.

Before / after

The CSV options pane before, then with the Encoding picker and the byte order mark checkbox, then with Windows-1252 selected so the checkbox dims because that encoding has no mark. Screenshots are attached in a follow-up comment.

One consequence worth flagging: the two new rows push Decimal and Reset to Defaults below the fold of the options pane, which is a ScrollView and scrolls to reach them. Say the word if you would rather the new controls sat last instead.

Other defects found while investigating

Verified, not fixed here, and not blocking this change.

1. CSV export ignores the per-object row scope. CSVExportPlugin.swift:76 calls the legacy streamRows(table:databaseName:); only streamRows(for:) reads PluginExportTable.rowScope. Set a filter and a row limit on a table, export as CSV, and every row and column is written while the filter icon stays lit and its accessibility value still reports the narrowing. HTML, JSON, SQL, Markdown, XML and Parquet all use streamRows(for:). XLSX and MQL have the same bug. Small: one call site per plugin.
2. The CSV inspector's writer drops any row the target encoding cannot represent, and still writes its line ending. CSVInspectorPlugin/CSVWriter.swift:98 appends the encoded line inside if let and the line ending outside it. Open a Windows-1252 CSV, edit a cell to hold 東京, save: the document reports a clean save, the unsaved dot clears, and that row is now an empty line. CSVWriter.WriteError.encodingFailed is declared and never thrown. This is the same class of defect this PR fixes for export. Small.
3. CSVStreamingParser is byte-oriented, so UTF-16 LE is unreadable. Measured above. Shared by CSV import and the CSV inspector, and the inspector's own picker already offers UTF-16 LE and BE. Fixing it means making indexRows, parseRow and field code-unit aware. Medium, and it is what would unblock UTF-16 export.
4. CSV delimiter detection is nondeterministic on a tie. CSVDialect.swift:117 ends with counts.max(by:) over a [UInt8: Int], and Dictionary iteration order depends on the per-process hash seed. Measured over 14 processes on a;b\n1,5;2,5\n (2 commas, 2 semicolons): 8 runs chose ;, 6 chose ,. When comma wins, the header parses as one field and the second column is silently dropped from the import. A single-column CSV is a four-way tie at zero, and the inspector then writes back whichever delimiter that launch picked. Small: iterate a fixed-order candidate array.
5. A third spelling of Latin-1 exists in the UI. Import and this new export picker both say "ISO Latin 1"; TablePro/Views/Inspector/CSVPropertyOptions.swift:35 says "Latin-1". Pre-existing, cosmetic.

Dropped after verification: the claim that PluginExportError.encodingFailed is a reachable defect. Its message does hardcode UTF-8 and it is thrown from one site, but String.data(using: .utf8) cannot return nil for a Swift String, so it has never reached a user. This change does not route through it.

https://claude.ai/code/session_01JH7gTeN4YSSASrHFeFJ9xT

[mintlify]

Preview deployment for your docs. Learn more about Mintlify Previews.

Project	Status	Preview	Updated
TablePro	🟢 Ready	View Preview	Sep 6, 2026, 7:10 AM

💡 Tip: Enable Automations to automatically generate PRs for you.

[datlechin]

Screenshots for the Before / After section. gh cannot upload images, so these are the three states as captured; the PNGs are on the authoring machine at scratchpad/shots/pane-before.png, pane-utf8.png and pane-cp1252.png.

Before — the options pane ends at Delimiter, Quote, Line break, Decimal, with Reset to Defaults visible.

After, UTF-8 — an Encoding row opens the picker group, with Include byte order mark below it, enabled because UTF-8 carries one.

After, Windows-1252 — the same pane with the encoding changed. Include byte order mark is dimmed, because that encoding has no mark. The stored choice survives the detour, so switching back to UTF-8 restores it.