Support special characters in job names - #658
Merged
Merged
Conversation
elias-ba
changed the title
elias-ba
force-pushed
the
446-allow-diacritics
branch
from
August 31, 2026 03:05
8d35905 to
b6cccff
Compare
2 tasks
Apollo stripped every non-ASCII character out of a step name, so Café became Cafe and 患者確認 became nothing at all. The rule now lives in one place, services/name_rules.py, and everything that states or enforces it reads from there: the sanitizer, the fuzzy lookup, the prompt shown to the model, and the judges. Two modes, chosen by APOLLO_UNICODE_STEP_NAMES. Off, the default, folds accents to ASCII and matches what Lightning validates today. On, anything but a control character survives as typed, ready for Lightning#4577. Both reject the same control set, normalise to NFC and cap at 100 graphemes. The tables are generated from the Elixir that Lightning runs, so tools/unicode_parity probes both sides and fails CI if the two Unicode versions drift apart.
elias-ba
force-pushed
the
446-allow-diacritics
branch
from
August 31, 2026 09:16
b6cccff to
ff7a1cd
Compare
elias-ba
marked this pull request as draft
August 31, 2026 18:15
The hand-written composer existed to reproduce Erlang 27's normaliser, because Lightning runs on it and the two have to agree on how a name is spelled. OpenFn/lightning#5109 moves Lightning to Erlang 28, which fixes the bug. Copying it stops being the right thing at that point. Measured against Erlang 28 over the 72,269-row corpus, restricted to inputs that could be a step name: the standard library disagrees on 16,622 and the hand-written composer on 16,898. So it is now marginally worse than one line. What remains in both is a Hangul shape that only a half-finished IME produces. Real Korean, Japanese, French and Vietnamese words normalise identically under Erlang 28, Python and ICU. Sixteen rows of a realistic corpus change, all Bengali and Tamil two-part vowels. They are exactly the rows where Erlang 27 and Erlang 28 disagree, and the new output matches Erlang 28 on every one. That is the bug going away. The grapheme clusterer, the trim set and their parity checks stay. They have nothing to do with composition and they guard a real rejection. One thing this does not do. The grapheme tables are still generated from Erlang 27, so they will need regenerating against 28 once Lightning actually ships it. Doing that now would make Apollo disagree with the Lightning that is running.
elias-ba
marked this pull request as ready for review
September 1, 2026 07:36
hanna-paasivirta
approved these changes
Sep 1, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Short Description
Apollo enforced its own ASCII-only rule on job names, so a name with an accent was folded before Lightning ever saw it.
Vérifier l'étatcame back asVerifier letat.The new rule is switchable on
APOLLO_UNICODE_STEP_NAMESand defaults to off, matching Lightning's current rule. Deploy this first with the default, then flip the flag once OpenFn/lightning#5106 ships. Turning it on first means Apollo emits names Lightning rejects.Fixes #446
Stacked on #660. A leak-hardening pass came out of this work because it lives in the same files. It is split out so it can be reviewed and merged on its own, and this branch is based on it, so the diff here is the naming work only.
Implementation Details
The grapheme clusterer is hand-written, with no third-party dependency, because Python has nothing that counts characters the way Elixir's
String.length/1does. That matters because Apollo truncates a name at 100 and Lightning rejects it at 100, so if the two count differently we emit names Lightning refuses.tools/unicode_parity/probes real Elixir for every table it depends on and fails CI if the two disagree.Normalising uses Python's standard library. It used to be hand-written too, to reproduce a bug in Erlang 27's normaliser, since Lightning runs on it and the two have to agree on how a name is spelled. OpenFn/lightning#5109 moves Lightning to Erlang 28, which fixes that bug, so the copy went. That is where most of this PR's shrinkage comes from.
One bug fixed on the way.
normalize_for_lookupwas ASCII-only, so every non-Latin name folded to the empty string and therefore matched the first non-Latin job in the workflow.Two things left alone on purpose.
services/global_chat/subagent_caller.py:60and:128still log the first 120 characters of a subagent response; they are untouched by this branch and that logger streams back to the caller who owns the code. And the grapheme tables are still generated from Erlang 27, so they need regenerating once Lightning actually ships 28. Doing that now would make Apollo disagree with the Lightning that is running.AI Usage
Please disclose whether you've used AI in this work (it's cool, we just want to
know!):
You can read more details in our
Responsible AI Policy