LatentPress

Context Compression Beyond Text and Vision Zhengze Zhou¹*, Hejian Sang²* ¹Cornell University ²Iowa State University

Compressed context is usually carried as human-readable text or as a rendered image that must be decoded, even when its only consumer is a language model. LatentPress writes conversational histories and long documents into a third representation instead: continuous memory tokens that a frozen decoder reads directly through its input-embedding interface, with no text reconstruction at inference time. A small reader-matched writer (an adapter, not the decoder) compresses 4–16x while training only ~0.1% of the decoder's parameters (4.2M–26.2M).

On LongMemEval, LatentPress reaches 0.504 accuracy at 7.70x compression, against 0.490 for uncompressed evidence, and beats text summaries (0.184) and OCR-based compression (0.426 → 0.312 as compression increases). Writing takes 43 ms per conversation — roughly an order of magnitude faster than summarization or OCR — and reading a compressed prefix is 5–9x faster than reading raw context or cached OCR.

Figure 1 — LatentPress overview, shown for conversational context. (A) A long, heterogeneous history containing segments with different information value. (B) LatentPress compresses the context into a short sequence of continuous soft tokens in a single, near-real-time forward pass. (C) The frozen LLM reads one concatenated sequence (the soft-token context followed by the question) and decodes the answer directly, with no text-reconstruction step. The same direct-read interface generalizes to long documents beyond conversational memory.

Interface

LatentPress separates context use into two operations:

• WRITE — maps text (or a document) to a compact continuous state.
• READ — supplies that state directly to a frozen decoder's input-embedding layer for downstream QA. No decoding back to text.

This differs from the closest prior mechanisms in what is trained, at what scale, and whether the representation is reconstructed before the decoder reads it:

Method	What is trained	Trainable scale	Representation	Reconstructed at inference?
Gist (Mu et al., 2023)	whole decoder (FT, masked attn.)	decoder-scale	KV-cache	no
AutoCompressor (Chevalier et al., 2023)	LLM (recursive summary)	LLM-scale	input (summary)	no
ICAE (Ge et al., 2024)	LLM encoder (LoRA)	LLM-scale (LoRA)	input slots	yes (autoencoder)
xRAG (Cheng et al., 2024)	projector only (LLM frozen)	small projector	input (1 token)	no
DeepSeek-OCR (Wei et al., 2025)	vision model	vision-model-scale	image → text	yes (OCR)
Glyph (Cheng et al., 2025)	vision-text model	vision-model-scale	image → text	yes (OCR)
AgentOCR (Feng et al., 2026)	RL-driven visual compression	vision-model-scale	image → text	yes (OCR)
LatentPress (ours)	small reader-matched adapter	~0.1% of decoder	input (soft tokens)	no

Local reference clones of each baseline above live as sibling directories next to this repo (../gisting, ../AutoCompressors, ../icae, ../xRAG, ../DeepSeek-OCR, ../Glyph, ../AgentOCR) for side-by-side comparison.

Evaluated Context Representations

The main experiments compare five ways of getting conversational context in front of a frozen reader model:

Method	Context representation
raw	Original text
summary	Model-generated factual summary
dsocr	Text rendered as images, reconstructed by DeepSeek-OCR
softtoken simple	Uniform learned pooling
softtoken role-aware	User tokens preserved; assistant tokens pooled

Benchmarks: LongMemEval (oracle-evidence conversational memory QA) and LongBench-QA (long-document QA, cross-domain and in-domain adapted).

Main Results

Table 2 — LongMemEval, Qwen2.5-7B reader (500 oracle-evidence questions, Llama-3.1-70B-Instruct judge). LatentPress is mean ± std over five seeds; baselines are deterministic.

Method	Compression	Overall	User-fact
Uncompressed evidence	1.0x	0.490	0.946
LatentPress, k_a=8	4.62x	0.476 ± 0.014	0.938 ± 0.007
LatentPress, k_a=16	6.27x	0.478 ± 0.020	0.891 ± 0.015
LatentPress, k_a=32	7.70x	0.504 ± 0.024	0.938 ± 0.010
ICAE	4.12x	0.452 ± 0.017	0.548 ± 0.019
ICAE	8.96x	0.318 ± 0.022	0.381 ± 0.023
ICAE	17.28x	0.174 ± 0.029	0.209 ± 0.031
DeepSeek-OCR	2.33x	0.426	0.797
DeepSeek-OCR	5.97x	0.390	0.672
DeepSeek-OCR	9.34x	0.312	0.594
Text summary	12.06x	0.184	0.297

Table 3 — cross-backbone generalization, zero-shot LongMemEval overall accuracy (UltraChat-trained, only the reader-specific compressor head retrained; borrowed encoder layers frozen):

Reader	LatentPress k_a=8	k_a=16	k_a=32	DeepSeek-OCR (2.33x / 5.97x / 9.34x)	Text summary
Qwen2.5-7B	0.476 ± 0.014	0.478 ± 0.020	0.504 ± 0.024	0.426 / 0.390 / 0.312	0.184 (12.1x)
Qwen3-8B	0.506 ± 0.015	0.514 ± 0.020	0.494 ± 0.025	0.542 / 0.506 / 0.408	0.348 (11.3x)
Qwen3-1.7B	0.434 ± 0.018	0.424 ± 0.024	0.416 ± 0.028	0.264 / 0.236 / 0.156	0.106 (28.7x)

Figure 2 — LongMemEval accuracy–compression frontiers. Role-aware LatentPress (orange) stays stable across compression rates on all three readers. Its relationship to the uncompressed oracle-evidence baseline (gray diamond) is reader-dependent: role-aware matches raw on Qwen2.5-7B, exceeds it on the weaker Qwen3-1.7B, and stays below the stronger raw baseline on Qwen3-8B. DeepSeek-OCR (blue) is competitive on Qwen3-8B at low compression but degrades as compression increases, and text summarization (red) is the weakest point on every reader. All results use the same 500 questions.

Table 4 — in-domain LongBench-QA (official overall score, %; writer trained on the target-domain training splits):

Reader	raw (1x)	f4	f8	f16
Qwen2.5-7B	43.80	49.06 ± 2.30	43.77 ± 2.83	37.78 ± 3.46
Qwen2.5-14B	47.93	57.99 ± 2.35	52.18 ± 2.82	40.30 ± 3.51
Qwen3-8B	30.80	39.62 ± 2.31	36.93 ± 2.82	26.12 ± 3.33

In-domain adaptation beats the uncompressed baseline at mild compression (4x, and 8x on the larger readers) but falls below it at the most aggressive 16x rate on every reader.

Figure 3 — LongBench-QA accuracy–compression frontiers. Overall score for Qwen2.5-7B, Qwen2.5-14B, and Qwen3-8B under cross-domain (green) and in-domain (orange) writer training. Gray diamonds mark uncompressed performance at 1x, and the blue DeepSeek-OCR curve spans base_size 1024/512 (~2.6/9.9x). The red text-summary baseline (one point per reader, at 14–20x) is the weakest on every reader. In-domain adaptation exceeds the uncompressed result at the milder rates but drops below it at 16x on all three readers.

Table 5 — efficiency (warm-loaded inference, seconds/example, LongBench-QA, 30 examples; LatentPress at f8, cached DeepSeek-OCR at base_size=640):

Reader	Raw context	LatentPress	Cached OCR
Qwen2.5-7B	2.44	0.49	2.71
Qwen2.5-14B	4.14	0.49	4.34
Qwen3-8B	3.97	0.43	4.03

LatentPress is 5.0–9.2x faster than raw inference and 5.5–9.4x faster than cached OCR at read time. On the write side, encoding takes 43 ms per conversation (Qwen3-8B, batches of 8, H100 80GB), versus ~934 ms average for batched DeepSeek-OCR reconstruction (22x longer), 407–645 ms for text summarization (9–15x longer), and 350–700 ms for ICAE (8–15x longer).

Reproduce

Full reproduction instructions — Docker build, pinned model revisions, and every table above — live in experiments/vtc_memory_validation/README.md. Quick map from result to command:

What	Command
Build + verify the environment	experiments/vtc_memory_validation/docker/build.sh then docker/run.sh python docker/verify_environment.py --require-gpu
One Table 2 row (role-aware, k_a=8, 4.62x)	see Reproduce One Main Result — 1 GPU
Tables 2 and 3 in full	scripts/reproduce_tables_2_3_qwen25.sh — 8 GPUs
Full main table (raw/summary/SoftMem/DeepSeek-OCR, all rates)	scripts/reproduce_main_table.sh — 8 GPUs
A single method (raw, summary, uniform/role-aware SoftMem, OCR, LongBench)	scripts/{raw_summary,softtoken_simple,softtoken_role_aware,deepseek_ocr,longbench,longbench_raw}.sh

experiments/vtc_memory_validation/docker/build.sh
experiments/vtc_memory_validation/docker/run.sh \
  python docker/verify_environment.py --require-gpu

The supported reproduction path uses only Docker, the NVIDIA Container Toolkit, public container images, public datasets, and public Hugging Face model repositories — no cluster scheduler or organization-specific service. Experiments are plain Bash scripts, paths are detected from each script's location, missing public benchmark data is downloaded automatically, and model aliases resolve to public Hugging Face repositories. Validated on NVIDIA B200 GPUs; other recent NVIDIA GPUs with enough memory should also work.

Related Repositories

• HJSang/OPSD_OnPolicyDistillation — on-policy distillation training, a separate research line by one of the authors; its README links back here from its "Related Repositories" section.

Citation

@article{zhou2026latentpress,
  title   = {LatentPress: Context Compression Beyond Text and Vision},
  author  = {Zhou, Zhengze and Sang, Hejian},
  journal = {arXiv preprint arXiv:2609.01507},
  year    = {2026}
}

Acknowledgements

Baselines and comparisons build on public releases of Gist Tokens, AutoCompressors, ICAE, xRAG, DeepSeek-OCR, Glyph, and AgentOCR. Evaluation uses the public LongMemEval and LongBench-QA benchmarks and the UltraChat dataset.