[Bug]: Desktop: severe text truncation/corruption during streaming with Portuguese (accented) text

[VingadorMode]

Bug Description

1. Severe text truncation during streaming

Words and even whole syllables are dropped from streaming responses. This is NOT limited to accented characters — plain text between accents also gets eaten. Examples observed:

Expected	Rendered
renderização	render
componentes-chave	componentesave
Vou investigar o pipeline de renderização de texto	Vou investigização de texto
no Hermes Desktop	noHermes (space also removed)
backend e o useSmoothReveal	backendveal

2. Words fused together (spaces removed)

Adjacent words lose their separating spaces: noHermes, backendveal, componentesave.

3. Thinking/reasoning blocks leaking

Internal thinking blocks (model's chain-of-thought) are appearing in the chat surface. These should be filtered out by preprocessMarkdown's REASONING_BLOCK_RE or handled by the streaming pipeline, but they leak through to the user.

4. Only affects streaming text — static text renders correctly

A test phrase with accented words sent as a static message rendered perfectly:
«Açúcar, coração, órgão, atenção, exceção, balão, canção, noção, poção, feição, benção.»

This isolates the bug to the streaming pipeline, not the final markdown renderer.

5. Not a visual glitch

The corrupted text is what arrives on screen — it's not a CSS/font clipping issue. The raw text content itself is damaged before or during rendering.

Likely Cause: useSmoothReveal .slice() on broken UTF-16 boundaries

The streaming pipeline is:

Backend → WS → message.delta events
  → useMessagePartText() accumulates text
  → SmoothStreamingText → useSmoothReveal() does .slice(0, length) progressively
  → preprocessMarkdown() + tailBoundedRemend()
  → Streamdown parseMarkdownIntoBlocks()
  → React render

### Steps to Reproduce

## Steps to Reproduce

1. Open Hermes Desktop
2. Use DeepSeek provider with `deepseek-v4-pro` model
3. Have a conversation in Portuguese (Brazilian) with accented text
4. Ask for a detailed/long response
5. Observe the streaming text — words will be truncated, fused, or missing

### Expected Behavior

## Symptoms

### 1. Severe text truncation during streaming

Words and even whole syllables are dropped from streaming responses. This is NOT limited to accented characters — plain text between accents also gets eaten. Examples observed:

| Expected | Rendered |
|---|---|
| `renderização` | `render` |
| `componentes-chave` | `componentesave` |
| `Vou investigar o pipeline de renderização de texto` | `Vou investigização de texto` |
| `no Hermes Desktop` | `noHermes` (space also removed) |
| `backend e o useSmoothReveal` | `backendveal` |

### 2. Words fused together (spaces removed)

Adjacent words lose their separating spaces: `noHermes`, `backendveal`, `componentesave`.

### 3. Thinking/reasoning blocks leaking

Internal `thinking` blocks (model's chain-of-thought) are appearing in the chat surface. These should be filtered out by `preprocessMarkdown`'s `REASONING_BLOCK_RE` or handled by the streaming pipeline, but they leak through to the user.

### 4. Only affects streaming text — static text renders correctly

A test phrase with accented words sent as a static message rendered perfectly:
`«Açúcar, coração, órgão, atenção, exceção, balão, canção, noção, poção, feição, benção.»`

This isolates the bug to the **streaming pipeline**, not the final markdown renderer.

### 5. Not a visual glitch

The corrupted text is what arrives on screen — it's not a CSS/font clipping issue. The raw text content itself is damaged before or during rendering.

## Likely Cause: `useSmoothReveal` `.slice()` on broken UTF-16 boundaries

The streaming pipeline is:

Backend → WS → message.delta events
→ useMessagePartText() accumulates text
→ SmoothStreamingText → useSmoothReveal() does .slice(0, length) progressively
→ preprocessMarkdown() + tailBoundedRemend()
→ Streamdown parseMarkdownIntoBlocks()
→ React render

**Primary suspect:** `useSmoothReveal` in `apps/desktop/src/components/assistant-ui/markdown-text.tsx:420`:

```typescript
shownRef.current = targetRef.current.slice(0, shownRef.current.length + add)

String.prototype.slice() operates on UTF-16 code units, not code points. While Portuguese accented characters (á, é, ç, ã) are BMP and occupy 1 code unit each, the add value is computed from character counts that may not account for:

1. Surrogate pairs in any embedded Unicode beyond BMP
2. Combining character sequences (e.g., emoji with ZWJ, variation selectors)
3. Token-boundary misalignment — DeepSeek's tokenizer may emit partial multi-byte sequences across delta events, and when add lands on a partial boundary, the .slice() splits a code point, producing a corrupted string that cascades through the remaining pipeline

Once the string is corrupted by a bad .slice() boundary, preprocessMarkdown and tailBoundedRemend operate on damaged input, potentially amplifying the corruption (e.g., scrubBacktickNoise regexes matching across broken character boundaries, removing text that coincidentally matches fence-like patterns in the corrupted byte stream).

Secondary Suspects

scrubBacktickNoise in markdown-preprocess.ts

The fenceNoiseRe = /{3,}/greplacement runs on text *outside* protected ranges. If the accumulated text is already corrupted from a bad.slice()`, backtick-like byte sequences in multi-byte character fragments could cause aggressive removal of adjacent text.

tailBoundedRemend in remend-tail.ts

The remend library is applied to the tail block of streaming text. If the tail contains corrupted character boundaries, remend's internal parsing may produce unexpected results that eat content.

Thinking block leak

The REASONING_BLOCK_RE in markdown-preprocess.ts:4 filters <think>...</think> blocks. During streaming, if a chunk arrives with </think (incomplete close tag) and the .slice() cuts it, the regex won't match, causing thinking content to leak into the rendered output. Additionally, the DeepSeek provider may use a different thinking wrapper format that doesn't match the regex.

Affected Files

apps/desktop/src/components/assistant-ui/markdown-text.tsx  — useSmoothReveal, SmoothStreamingText
apps/desktop/src/lib/markdown-preprocess.ts                   — preprocessMarkdown, scrubBacktickNoise, REASONING_BLOCK_RE
apps/desktop/src/lib/remend-tail.ts                           — tailBoundedRemend, findRemendWindowStart
node_modules/@assistant-ui/react-streamdown                   — parseMarkdownIntoBlocks, StreamdownTextPrimitive
node_modules/remend                                           — incomplete markdown repair

### Actual Behavior

Symptoms

### 1. Severe text truncation during streaming

Words and even whole syllables are dropped from streaming responses. This is NOT limited to accented characters — plain text between accents also gets eaten. Examples observed:

| Expected | Rendered |
|---|---|
| `renderização` | `render` |
| `componentes-chave` | `componentesave` |
| `Vou investigar o pipeline de renderização de texto` | `Vou investigização de texto` |
| `no Hermes Desktop` | `noHermes` (space also removed) |
| `backend e o useSmoothReveal` | `backendveal` |

### 2. Words fused together (spaces removed)

Adjacent words lose their separating spaces: `noHermes`, `backendveal`, `componentesave`.

### 3. Thinking/reasoning blocks leaking

Internal `thinking` blocks (model's chain-of-thought) are appearing in the chat surface. These should be filtered out by `preprocessMarkdown`'s `REASONING_BLOCK_RE` or handled by the streaming pipeline, but they leak through to the user.

### 4. Only affects streaming text — static text renders correctly

A test phrase with accented words sent as a static message rendered perfectly:
`«Açúcar, coração, órgão, atenção, exceção, balão, canção, noção, poção, feição, benção.»`

This isolates the bug to the **streaming pipeline**, not the final markdown renderer.

### Affected Component

CLI (interactive chat)

### Messaging Platform (if gateway-related)

_No response_

### Debug Report

```shell
--- hermes dump ---
version:          0.18.2 [6142203b] (2026-07-11)
os:               Windows 10 AMD64
python:           3.11.15
openai_sdk:       2.24.0
profile:          default
hermes_home:      ~/.hermes
model:            deepseek-v4-pro
provider:         deepseek
terminal:         local

api_keys:
  openrouter           not set
  openai               set (shell only — not in .env; managed/desktop backend may not see it)
  anthropic            not set
  anthropic_token      not set
  nous                 not set
  google/gemini        set
  gemini               set
  glm/zai              not set
  zai                  not set
  kimi                 not set
  minimax              not set
  deepseek             set
  dashscope            not set
  huggingface          not set
  nvidia               not set
  opencode_zen         not set
  opencode_go          set
  kilocode             not set
  firecrawl            not set
  tavily               not set
  browserbase          not set
  fal                  not set
  elevenlabs           not set
  github               not set

features:
  toolsets:           hermes-cli
  mcp_servers:        0
  memory_provider:    built-in
  gateway:            stopped (manual process)
  platforms:          none
  cron_jobs:          0
  skills:             49

config_overrides:
  agent.max_turns: 25
  compression.threshold: 0.4
  display.streaming: True
  display.show_reasoning: False
--- end dump ---
Hermes One Desktop
Hermes One Desktop

Operating System

Windows 10.0.19045 N/A compilação 19045

Python Version

Python 3.14.6

Hermes Version

V018.2

Additional Logs / Traceback (optional)

## Suggested Fix Direction

1. **Audit `useSmoothReveal` for code-point-safe slicing** — use `[...str]` spread or `Array.from()` to split by code points before `.slice()`, or use the `Intl.Segmenter` API for grapheme-cluster-safe slicing. This is the most likely root cause.

2. **Add integration test** with Portuguese streaming text containing accented characters, emoji, and mixed scripts to catch regressions.

3. **Investigate thinking block leak separately** — the DeepSeek provider's reasoning format may need its own cleanup before text hits the Streamdown pipeline.

4. **Consider defensive `.normalize('NFC')` on accumulated text** before it enters the markdown pipeline to collapse combining character sequences that may have been split across delta boundaries.

Root Cause Analysis (optional)

No response

Proposed Fix (optional)

No response

Are you willing to submit a PR for this?

• I'd like to fix this myself and submit a PR