Agents crash with Jason.EncodeError on non-ASCII WORKFLOW.md prompts — \R regex in Workflow.split_front_matter/1 splits UTF-8 multibyte characters at byte boundaries

[ivybug-test]

Summary

With a WORKFLOW.md whose prompt body contains non-ASCII text (Chinese, in our case), every agent dispatch fails and the orchestrator retries in a backoff loop forever. The agent task dies with:

** (Jason.EncodeError) invalid byte 0xE5 in <<228, 189, 160, ...>>
    (jason 1.4.4) lib/jason.ex:164: Jason.encode!/2
    (symphony_elixir 0.0.3) lib/symphony_elixir/codex/app_server.ex:1031: SymphonyElixir.Codex.AppServer.send_message/2
    (symphony_elixir 0.0.3) lib/symphony_elixir/codex/app_server.ex:344: SymphonyElixir.Codex.AppServer.start_turn/7
    (symphony_elixir 0.0.3) lib/symphony_elixir/agent_runner.ex:105: SymphonyElixir.AgentRunner.do_run_codex_turns/8

• Environment: symphony-v0.0.3-linux_x86_64 Burrito release; tracker: GitHub Issues; Codex CLI 0.154.0.

Root cause

elixir/lib/symphony_elixir/workflow.ex:86:

lines = String.split(content, ~r/\R/, trim: false)

~r/\R/ compiles without the u (unicode) flag, so Erlang's :re (PCRE) runs in byte mode. In byte mode, \R matches the single byte 0x85 (NEL). But 0x85 is also a legal UTF-8 continuation byte — e.g. 待 U+5F85 = E5 BF 85, 内 U+5185 = E5 86 85. The regex split cuts those characters apart mid-sequence, and the subsequent Enum.join(prompt_lines, "\n") glues the halves back together with a literal newline, producing invalid UTF-8. Later, Jason.encode! validates the rendered prompt as UTF-8 and raises.

Byte-level confirmation on our template: it contains 21 occurrences of byte 0x85, each inside a CJK character, and the rendered prompt's first invalid position is E5 BF 0A — i.e. 待 with its final byte 0x85 replaced by \n (the text was "…不要等待人工执行…" from the unattended-session instructions). ASCII-only WORKFLOW.md files are unaffected, which is why the bundled one works.

Minimal reproduction

Reproduced standalone with the same pinned deps (solid 1.2.2, jason 1.4.4), replicating Workflow.parse → PromptBuilder.build_prompt → AppServer.start_turn 1:1. Any Elixir ≥ 1.14:

Mix.install([{:solid, "1.2.2"}, {:jason, "1.4.4"}])

content = File.read!("WORKFLOW.md")            # any body containing e.g. 待 (E5 BF 85)
lines = String.split(content, ~r/\R/, trim: false)
["---" | tail] = lines
{front, rest} = Enum.split_while(tail, &(&1 != "---"))
["---" | prompt_lines] = rest
prompt = Enum.join(prompt_lines, "\n") |> String.trim()

IO.puts("template valid utf8? #{String.valid?(prompt)}")  # => false — already corrupted before Solid

Rendering this template and encoding it like start_turn does (Jason.encode!(%{"input" => [%{"type" => "text", "text" => rendered}]})) raises the exact error above.

Verified fix

Either add the unicode flag:

lines = String.split(content, ~r/\R/u, trim: false)

or (preferred — binary patterns are codepoint-safe and faster):

lines = String.split(content, ["\r\n", "\r", "\n"], trim: false)

Both variants produce valid UTF-8 through the whole pipeline and Jason.encode! succeeds end-to-end (verified with the same deps, using a Chinese WORKFLOW.md and Chinese issue fields).

Scope

• Affects the v0.0.3 reference Elixir implementation for any WORKFLOW.md containing characters whose UTF-8 encoding includes the byte 0x85 — very common in CJK text, also possible in emoji and some other scripts.
• grep -rn '\\R' elixir/lib/ shows workflow.ex:86 is the only occurrence of this pattern.

Current workaround for affected users: keep WORKFLOW.md ASCII-only (English prompt body), since ASCII contains no 0x85 bytes.

[ivybug-test]

We've verified and shipped this fix locally. The one-line patch plus a regression test is available on our fork:

• Branch: https://github.com/ivybug-test/symphony/tree/fix/workflow-split-multibyte-utf8
• Commit: ivybug-test@b3180e6 (PR creation to this repo appears to be disabled, so linking the branch here — happy to send a proper PR if there's a preferred channel)

Diff summary:

• elixir/lib/symphony_elixir/workflow.ex:86: ~r/\R/ → ~r/\R/u
• elixir/test/symphony_elixir/core_test.exs: new regression test workflow load keeps multibyte UTF-8 prompts valid across line splitting (CJK prompt body containing 待 U+5F85 = E5 BF 85)

Verification: mix format --check-formatted clean; mix test 294 tests with 0 new failures on elixir:1.19-otp-28 (2 pre-existing assert_due_in_range timing failures reproduce identically on unpatched main); end-to-end, a Chinese WORKFLOW.md that crashed at dispatch on v0.0.3 now dispatches and drives the agent to a merged PR.