license	apache-2.0
language	en de fr es pt it nl ru ja zh ko ca sv pl tr
tags	email token-classification sequence-labeling quoted-text-removal signature-detection multilingual onnx
pipeline_tag	token-classification
library_name	unquote
datasets	unquote-zones
metrics	f1

Unquote

Strip the quoted history off an email without ever eating the part you wrote.

Quoted replies, signatures and legal boilerplate are most of a mail thread and none of its meaning. Removing them before you pay a model to read it is easy money — right up until the cleaner deletes a paragraph the sender actually wrote, and you have no way to know.

Unquote is a small CPU model that tags every line of an email as one of eight zones. It is built around one asymmetry: leaving quoted text in costs you tokens; taking body text out costs you the message. So every decision is biased toward keeping.

flowchart TD
    E["raw email"] --> F["LineFeaturizer<br/>88 structural features per line<br/>quote depth · script · wrap width · distance from end"]
    F --> M["ZoneTagger<br/>line encoder → document transformer → CRF"]
    M --> D["constrained decode<br/>Viterbi over (zone, seen_trailer)"]
    D --> G["ConservativeGate<br/>below threshold → BODY"]
    G --> O["8 zones"]

    O --> B["BODY"]
    O --> Q["QUOTE"]
    O --> A["ATTRIBUTION"]
    O --> S["SIGNATURE"]
    O --> C["DISCLAIMER"]
    O --> MF["MOBILE_FOOTER"]
    O --> AR["AUTOREPLY"]
    O --> FH["FORWARD_HEADER"]

    B --> K["kept"]
    Q --> R["removable"]
    A --> R
    S --> R
    C --> R
    MF --> R
    AR --> R
    FH --> R

    style B fill:#1f6feb,color:#fff
    style K fill:#1f6feb,color:#fff
    style R fill:#8b949e,color:#fff
    style G fill:#2a9d8f,color:#fff

Loading

Three lines

from unquote import ZoneTagger
tagger = ZoneTagger.from_pretrained("models/unquote-small")   # or ZoneTagger.heuristic()
print(tagger.clean(raw_email))                                # body only

unquote clean mail.txt --keep body,attribution

What this does and does not promise

It promises it will not delete your body text, and it buys that promise in three places you can inspect:

Mechanism	Where	Strength
BODY is not in the removable set; --keep quote still keeps body	labels.py	structural — no caller can opt out
The gate only ever moves a label toward BODY, never away	calibrate.py	structural — property-tested
At gate threshold 1.0, nothing is removed at all	calibrate.py	mathematical
Per-zone thresholds fitted for zero body loss on a calibration split	calibrate.py	empirical

That last row is the honest one. A confidence gate cannot stop a confidently wrong prediction. On deliberately damaged synthetic mail at threshold 0.999 we measured mean body loss 0.0005 and worst-case 0.0625, affecting 1 message in 120. So the claim is "fitted to zero body loss and measured on held-out data", not an impossibility proof. tests/test_robustness.py holds us to the measured level; if a change makes the tagger confidently wrong more often, that test fails.

Choosing an operating point

The gate ships fitted for zero body loss, which is what the project promises and what the default gives you. That is one point on a curve, and the curve is measured, so pick a different point if your risk tolerance differs:

target body loss	threshold	measured body loss	token reduction
0.00%	0.9999	0.000%	34.7%
0.05%	0.99	0.045%	52.0%
0.20%	0.915	0.193%	53.4%
0.50%	0.705	0.499%	54.2%
1.00%	0	0.728%	54.7%

_{Measured on 1,500 held-out calibration messages; models/unquote-small/sweep.json holds all 206 points and figures/reduction_vs_loss.png plots them.}

Move along it with --threshold, or refit with scripts/train.py --target-body-loss 0.002. Be aware of what the strict end costs: at zero body loss the gate rescues signatures the model was 89% sure about, so SIGNATURE F1 falls from 0.897 ungated to 0.429 gated. That is the trade working as designed — the whole point is that an uncertain line survives — but it means the strict default leaves more signature text in place than the model is actually capable of removing.

It does not promise a perfect split. Leftover quoted lines are an accepted outcome of the trade, not a bug. If you need every quoted byte gone and you can tolerate losing prose, this is the wrong tool.

The eight zones

Zone	What it is	Removable
BODY	what this author wrote	never
QUOTE	quoted/forwarded material, including its signature	yes
ATTRIBUTION	On … wrote: / Am … schrieb: / …さんは書きました:	yes
SIGNATURE	name, title, employer, phone, vcard tail	yes
DISCLAIMER	legal boilerplate	yes
MOBILE_FOOTER	Sent from my iPhone, iPhoneから送信	yes
AUTOREPLY	out-of-office, vacation responders	yes
FORWARD_HEADER	forward banner + From:/Sent:/To:/Subject: block	yes

Two conventions the whole project inherits from unquote/compose.py:

1. Everything inside a quoted message collapses to QUOTE — including its signature. A quoted signature is not this author's signature.
2. The introducer keeps its own zone at every nesting depth. An attribution line three levels deep is still ATTRIBUTION.

Why a model instead of regexes

Because the rule-based assumption — a quoted line starts with > — fails constantly on real mail, and fails hardest outside English.

Measured on 4,000 real reply/parent pairs from Apache, GNU and Python archives:

1,572 of 4,000 replies (39%) contain at least one quoted line whose > marker was destroyed, almost always by hard-wrapping at 72–80 columns. Those lines are recoverable only by matching them against the parent message.

And the rule tables themselves are English-shaped. Verified on this machine: talon has no attribution pattern for CJK, so a Japanese inline reply passes through it essentially untouched — see the demo, where talon removes nothing from the Japanese example.

Results

Measured on 1,200 held-out messages from data/processed/test.jsonl (synthetic + real public-archive mail). Cost figures use claude-sonnet-5 input pricing.

Overall

System	body loss ↓	msgs w/ loss	macro F1 ↑	SIGNATURE F1	token reduction	ms/msg
talon (rule-based standard)	4.178%	119/1200	0.592	0.455	38.2%	0.54
email-reply-parser (rule-based)	8.207%	140/1200	0.719	0.367	47.5%	0.10
Unquote (fp32, gated)	0.008%	2/1200	0.844	0.429	35.9%	17.65
Unquote (int8 ONNX)	0.008%	2/1200	0.844	0.429	35.9%	6.43
Unquote (fp32, gate off)	1.176%	111/1200	0.951	0.897	42.3%	18.98
Unquote heuristic (no weights — not the model)	0.243%	23/1200	0.337	0.123	37.2%	2.13

body loss is the fraction of true BODY lines a system would delete — the number that decides whether a cleaner is safe to run unattended. Everything else is secondary to it.

By language (synthetic split)

System	en	fr	de	ja	ru	pt	ko	es	nl	it
talon (rule-based standard)	0.510	0.568	0.459	0.480	0.467	0.458	0.459	0.387	0.523	0.299
email-reply-parser (rule-based)	0.700	0.756	0.600	0.751	0.632	0.747	0.612	0.669	0.662	0.677
Unquote (fp32, gated)	0.823	0.873	0.874	0.906	0.914	0.899	0.866	0.846	0.873	0.879
Unquote (int8 ONNX)	0.825	0.873	0.879	0.905	0.914	0.899	0.866	0.846	0.874	0.878
Unquote (fp32, gate off)	0.955	0.973	0.969	0.997	0.984	0.973	0.948	0.964	0.975	0.989
Unquote heuristic (no weights — not the model)	0.229	0.226	0.266	0.411	0.252	0.265	0.194	0.255	0.256	0.230
messages	31	37	36	35	34	31	30	27	27	25

Rule-based extractors are tuned on English rule tables; the structural model has no such centre of gravity.

Plotted on the synthetic split on purpose: its labels are correct by construction, whereas non-English real mail in this corpus has pattern-derived gold for the trailer zones, which would be circular to score a rule-based baseline against. The all-messages version is figures/lang_bars_all.png.

The English gap

System	English macro F1	non-English macro F1	gap
talon (rule-based standard)	0.510	0.462	+0.049
email-reply-parser (rule-based)	0.700	0.680	+0.021
Unquote (fp32, gated)	0.823	0.882	-0.059
Unquote (int8 ONNX)	0.825	0.883	-0.058
Unquote (fp32, gate off)	0.955	0.975	-0.020
Unquote heuristic (no weights — not the model)	0.229	0.264	-0.035

Under damage

Hard-wrapping at 72 columns, destroyed and multiplied quote markers, HTML-to-text artefacts, mojibake, Unicode confusables.

System	clean macro F1	damaged macro F1	drop	damaged body loss
talon (rule-based standard)	0.634	0.316	+0.318	1.226%
email-reply-parser (rule-based)	0.722	0.626	+0.096	0.000%
Unquote (fp32, gated)	0.832	0.805	+0.027	0.060%
Unquote (int8 ONNX)	0.831	0.806	+0.026	0.060%
Unquote (fp32, gate off)	0.951	0.949	+0.002	1.544%
Unquote heuristic (no weights — not the model)	0.413	0.201	+0.212	1.366%

Synthetic vs real mail

System	synthetic macro F1	real macro F1	real body loss
talon (rule-based standard)	0.460	0.625	5.442%
email-reply-parser (rule-based)	0.675	0.709	11.270%
Unquote (fp32, gated)	0.880	0.436	0.000%
Unquote (int8 ONNX)	0.880	0.433	0.000%
Unquote (fp32, gate off)	0.972	0.726	1.332%
Unquote heuristic (no weights — not the model)	0.252	0.487	0.072%

⚠︎ On real mail, SIGNATURE/DISCLAIMER/ATTRIBUTION/MOBILE_FOOTER/AUTOREPLY/FORWARD_HEADER gold labels are pattern-derived and therefore circular for rule-based comparison; QUOTE/BODY come from parent-message alignment and are not. See DATA_PROTOCOL.md.

Exactly what ran for each system

• talon — talon 1.4.4: talon.quotations.extract_from_plain for quote removal + talon.signature.bruteforce.extract_signature for signatures. Import shims required: cchardet [shimmed -> chardet 7.6.0], sklearn.externals.joblib [shimmed -> joblib 1.5.2], sklearn.linear_model.stochastic_gradient [aliased -> sklearn.linear_model._stochastic_gradient], sklearn.svm.classes [aliased -> sklearn.svm._classes]. talon.init() ok. ML signature probe: talon.signature.extract returned signature=None on a message that plainly has one; talon logged 'ERROR when extracting signature with classifiers' -> AttributeError: 'NDArrayWrapper' object has no attribute 'T'; therefore the ML signature classifier is NOT available, so talon.signature.bruteforce.extract_signature -- talon's own fallback -- is used. Zone mapping: kept prose -> BODY (talon has no DISCLAIMER / MOBILE_FOOTER / AUTOREPLY / FORWARD_HEADER zone, so lines it keeps stay BODY and lines it drops become QUOTE), signature -> SIGNATURE, everything discarded -> QUOTE; discarded lines matching talon's own quotations.SPLITTER_PATTERNS are labelled ATTRIBUTION (Unquote's pattern tables are never used to score talon); talon has no pattern for CJK attributions, so those stay QUOTE.
• email_reply_parser — ok | RAISED on 12/1200 messages (1.0%); those were scored as all-BODY, which costs it reduction but not body loss
• llm_prompt — NOT RUN: no cached response for this message (key d609f5b5c7bf) and no call_fn; record a run with call_fn set, or point cache_path at an existing JSONL cache. Provide --llm-cache with cached responses to include this system.
• unquote_fp32 — torch checkpoint models/unquote-small, WITH the fitted conservative gate (gate=yes)
• unquote_int8 — onnx graph models/unquote-small/model.onnx
• unquote_fp32_ungated — torch checkpoint models/unquote-small, gate DISABLED (shows raw model quality; not the shipped configuration)
• unquote_heuristic — weight-free fallback (NOT the trained model)

How to read the comparison fairly

• Each system is scored only on the zones it can emit. talon has no DISCLAIMER/MOBILE_FOOTER/AUTOREPLY/FORWARD_HEADER concept, so it is not blamed for missing distinctions it never claimed. results.json records the handicap per system in expressible_zones.
• talon's ML signature classifier does not load on modern scikit-learn. It raises 'NDArrayWrapper' object has no attribute 'T' internally, talon swallows the error and returns signature=None. We detect this, fall back to talon's own signature.bruteforce, and print exactly what ran in system_status. We do not quietly benchmark a crippled configuration.
• talon also needs import shims to load at all on Python 3.13 (cchardet is dead; sklearn.externals.joblib moved). unquote.baselines.install_talon_shims() provides them.
• A system that could not run is absent from the results, never scored zero. The LLM-prompt baseline runs only from a response cache; without one it is recorded as NOT RUN.

Data

What we use

Public mailing-list archives only, plus synthetic threads.

Source	Endpoint	Role
Apache	lists.apache.org/api/mbox.lua	real, English
Python	mail.python.org/pipermail/…	real, English
GNU	lists.gnu.org/archive/mbox/…	real, English
Debian localized lists	lists.debian.org/<list>/<YYYY>/<MM>/msgNNNNN.html	real, 15 languages
unquote.compose	—	synthetic, 11 languages × 8 clients

Training mixes both, and the splits are disjoint by thread.

An earlier design trained on synthetic threads only, so that no real-data number could possibly be fitted. That turned out to be the wrong trade, and the measurement is worth showing:

trained on	macro F1 (synthetic)	macro F1 (real)	body loss (real)
synthetic only	0.982	0.321	0.550

The model learned the generator, not the task. With 55% ungated body loss on real mail, fitting the gate for zero body loss forced every threshold to 1.0 — removing nothing at all. Withholding real data did not make the evaluation honest; it made the model useless. That checkpoint is kept as models/unquote-synthetic-only/ with its numbers, because it is the evidence for this design decision.

What actually protects the evaluation is thread-disjointness: a reply quotes its parent almost verbatim, so whole threads are assigned to a single split, and messages are deduplicated by body text (cross-posts share a body under different Message-IDs). tests/test_splits.py asserts zero thread overlap and zero verbatim-body overlap between splits.

What we deliberately do not use

Enron, and every corpus like it. The Enron dataset is the standard email corpus, it is freely downloadable, and it is a collection of private messages from people who never agreed to publication and were not public figures — they were employees whose mailboxes entered the record through litigation. Public availability is not consent. scripts/build_dataset.py carries an enforced deny-list covering Enron, Avocado, and the various stolen or FOIA-released private mailboxes.

Mailing-list posts are different in kind: the author wrote to a world-readable archive, knowing it was one.

Privacy

Real signatures contain real names, phone numbers and addresses. Concentrating thousands of them into one convenient file is a different act from linking to an archive, so every real message is scrubbed before it is written or pushed:

• addresses → @example.com/.org/.net (RFC 2606 reserved)
• phone numbers → the +1-555-01xx fictional range
• postal addresses → invented streets
• every participant's name → a synthetic name. Scrubbing only the sender is not enough: an attribution line (Alex Kestrel wrote:) and a quoted signature name a different person. The builder indexes every From display name in the corpus, finds which known participants each message actually mentions, and replaces all of them. The first version of the scrubber missed this and leaked third-party names on attribution lines; tests/test_splits.py now fails if any unscrubbed address reaches disk.

Replacements are structure-preserving — a scrubbed phone number is still phone-number-shaped and the line count never changes — so the line-tagging task is unaffected. Mailing-list and role addresses are kept, because they are infrastructure, not people.

Also scrubbed: addresses the archive itself obfuscated. Public archives rewrite a@b.com as a <at> b.com to defeat scrapers; ignoring that would make the anti-scraping measure the reason our corpus leaks addresses. Which forms to support was decided by counting them in this corpus rather than guessing:

form	messages containing it
(at)	21
AT … DOT (caps)	10
<at>	4
[at]	0
at … dot (lowercase)	0
at org.apache… (Java stack trace)	165

So the lowercase bare-word form is deliberately not matched: it never occurs, while the construct it collides with — the opening of every Java stack trace — occurs 165 times, on exactly the developer lists this corpus is drawn from.

Verified on the real corpus: 22,762 addresses, 59,657 name tokens and 2,307 phone numbers replaced; 0 non-reserved addresses surviving on disk; line counts preserved in every message. tests/test_splits.py re-checks the written corpus, and tests/test_scrub.py pins the false-positive cases (stack traces, index.html, "meet me at noon").

Every example in app.py and this README is synthetic.

Labelling real mail without hand-annotating it

QUOTE labels on real mail come from parent-message alignment, not from a regex: a reply's quoted region is recovered by matching its lines against the actual parent it replies to (In-Reply-To/References). 6,795 of the fetched messages have their parent in the corpus. This is what makes real-data QUOTE/BODY numbers independent evidence.

The other six zones have no such external signal, so their real-data labels are pattern-derived — which makes them circular for scoring a rule-based baseline. results.json flags those rows (circular_zones), and DATA_PROTOCOL.md sets out which numbers may be quoted for what. Synthetic labels carry no such caveat: they are correct by construction, because the composer records each line's zone as it emits it.

Model

Hierarchical line tagger — tokens are cheap, context is what matters:

tokens of one line → embedding → shift-mix → masked mean+max pool
concat 88 structural features
→ transformer over the LINE sequence          ← where the evidence actually is
→ per-line emissions (L, 8) + learned (8,8) transitions
→ constrained Viterbi

Position is encoded from both ends. Trailer zones are anchored to the end of a message, so the document encoder gets distance-from-end alongside distance-from-start; with absolute positions alone the model must infer message length before it can find the signature.

Preset	Parameters
tiny	6.4 M
small	33.9 M
base	78.5 M

The structural constraint is a guarantee, not a preference

Decoding runs Viterbi over the augmented state (zone, seen_trailer), where seen_trailer latches the moment a SIGNATURE/DISCLAIMER/MOBILE_FOOTER line is emitted. In the latched half, BODY is -inf. So no body line can follow a signature, no matter what the network scores — including via the SIGNATURE → QUOTE → BODY path that a pairwise transition ban cannot block. posterior_marginals runs the same constrained lattice, so the probabilities the gate calibrates are consistent with what the decoder can actually emit.

One constraint is deliberately soft: ATTRIBUTION → BODY occurs in 0.064% of real transitions (someone writes On X wrote: then types unquoted prose). Forbidding it would force the decoder to relabel one of the two lines, and the cheaper relabel is BODY → QUOTE — body loss. A constraint that can cause body loss is the wrong trade here, so it lives in the transition prior instead.

Install

Inference only — no torch, no downloads, works immediately:

pip install "unquote @ git+https://github.com/NagaYu/unquote"

That gives you the CLI and ZoneTagger.heuristic(). To run the trained model you also need the inference extra and the checkpoint:

pip install "unquote[export] @ git+https://github.com/NagaYu/unquote"

from unquote import ZoneTagger
tagger = ZoneTagger.from_pretrained("NagaYu/unquote-small")   # pulls from the Hub
print(tagger.clean(raw_email))

Optional extras: train (torch, tokenizers, datasets), bench (email-reply-parser, matplotlib, scikit-learn), app (gradio), dev (pytest). For everything:

git clone https://github.com/NagaYu/unquote && cd unquote
pip install -e ".[train,export,bench,app,dev]"

Installing talon, to reproduce the comparison

talon is not in any extra, because it cannot be installed normally on a current Python. It depends on cchardet, whose C extension includes longintrepr.h — a header removed in Python 3.12 — so the build fails outright. Install it without its own dependency resolution:

pip install --no-deps talon
pip install "unquote[bench]"

unquote.baselines.install_talon_shims() then supplies cchardet (backed by chardet) and the sklearn.externals.joblib alias at import time.

That this dance is necessary is not an aside — it is part of what the benchmark measures. The rule-based standard is unmaintained, and its ML signature classifier no longer loads at all on a modern scikit-learn.

from_pretrained falls back to the weight-free heuristic backend, with a logged warning, if a checkpoint cannot be loaded. Check prediction.meta["backend"] if you need to be certain which one ran.

Reproduce

python scripts/build_dataset.py --synthetic 12000 --real     # fetch + compose + scrub
python scripts/train_tokenizer.py                            # byte-level BPE
python scripts/train.py --preset small --epochs 3            # train + calibrate the gate
python scripts/export.py --onnx --quantize dynamic-int8      # portable CPU inference
python benchmarks/evaluate.py --limit 1500                   # all systems, all strata
python figures/make_figures.py                               # the two figures
pytest                                                       # the four guarantees

On GGUF

scripts/export.py --gguf refuses, and says why. GGUF is llama.cpp's container and llama.cpp implements a fixed set of decoder-only LLM architectures. Unquote is a hierarchical encoder with a CRF head that also consumes an 88-dimensional engineered feature vector per line; there is no llama.cpp graph that would load these tensors. A .gguf here would be a file no runtime can execute, so we ship int8 ONNX instead.

Limitations

• Throughput is tens to hundreds of messages/second, not thousands. Measured single-process on 6 threads of an 8-core CPU, end to end (featurise, tokenise, forward pass, constrained decode, gate):

  configuration	ms/message	messages/second
  Unquote fp32 (torch)	17.8	56
  Unquote int8 (ONNX)	6.4	156
  heuristic fallback (no weights)	1.9	539

  The original target was "thousands per second on CPU". A 31M-parameter transformer over a line sequence does not reach that on one core, and saying so is more useful than quietly redefining the target. int8 quantisation buys 2.8x with identical F1 and body loss (see the results table), and throughput scales with processes. If you genuinely need thousands/sec, the weight-free backend is the honest option, at a large accuracy cost.

• Token counts from the default estimator are ±11% against a real tokenizer (fitted on 1,144 real multilingual samples). Use --tokenizer tiktoken for exact counts.

• Cost figures are input tokens only and assume the removed text would otherwise have been sent verbatim. With prompt caching, savings on a repeated quoted prefix are much smaller.

• Real-data gold for SIGNATURE/DISCLAIMER is pattern-derived; see the circularity note above.

• HTML-only messages are dropped at ingest — without line structure there is nothing to line-tag.

• Real mail is in the training set. Splits are thread-disjoint and body-deduplicated, which is what makes the held-out numbers meaningful, but it is a weaker guarantee than never having seen real mail at all — and the synthetic-only ablation shows why that stronger guarantee was not affordable.

• Non-English real mail comes from Debian lists only, and has no parent message to align against, so its trailer-zone gold is pattern-derived. The non-circular multilingual evidence is the synthetic split.

Layout

unquote/      labels types features compose damage model tokenization
              decode calibrate savings scrub baselines cli
scripts/      build_dataset train_tokenizer train export
benchmarks/   evaluate.py → results.json
figures/      make_figures.py → lang_bars, reduction_vs_loss
tests/        the four guarantees + contract enforcement
app.py        Gradio demo (synthetic examples only)
CONTRACT.md   internal interface contract
DATA_PROTOCOL.md  how real mail is labelled, and where that is weak

Every public function carries a Substantiates: line naming which claim it backs — multilingual, body-loss-zero, token-reduction, speed, robustness, structure. tests/test_docstring_claims.py enforces it, so a claim with no code behind it fails the build.

License

Apache-2.0. Archive content is quoted under the terms of the public archives it came from; see DATA_PROTOCOL.md.