Repository navigation
Exit 0, zero valid amounts: testing a local code-only job with 16 assertions #17144
FenjuFu
started this conversation in
Show and tell
Replies: 0 comments
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Uh oh!
There was an error while loading. Please reload this page.
A 29-cent conversion looked like a small enough job for a local coding model. The headless run exited 0, returned a complete JavaScript module—and that module rejected all five valid amounts, including
"0.29". Giving it the failed tests and clarifying the prompt produced the same code again.This is a small, reproducible failure case using Qwen Code, Spark-X2.6-1.7B-GGUF, and a local llama.cpp CPU server. The useful part is a gate for code-only batch jobs: keep tool permissions and functional correctness as separate checks.
What actually happened
--bare --safe-mode --max-tool-calls 0, the first attempt still tried one tool call. Qwen stopped it with exit 55, before tool execution. Safe mode removes customizations; in this run it did not remove the six built-in tools.--exclude-tools read_file run_shell_command get_goal update_goal edit notebook_editlefttools: []in the initialization event. The next run finished with exit 0 and zero tool calls.Number(s) * 100passed 4/5 valid conversions and 0/11 rejection checks. In particularNumber('0.29') * 100was28.999999999999996. It is a diagnostic baseline, not a proposed implementation.Reproduce the local setup
Tested on Windows 11 (
10.0.26200), Intel Core Ultra 5 125H, 31.5 GiB RAM, Node24.16.0, Qwen Code 0.27.2 (sourcef42fe0f3251172c1174d00382ece522eafb3c6c2), llama.cpp b12336 (cfb9d4b754616ca47d165fa1ff8a7a80c30847e3). CPU only, eight threads, one slot, 8192-token context, temperature 0, seed 42, reasoning off, prediction cap 1024. Qwen warned that this Node version can use extra temporary memory in shell safety checks; no shell tool was executed here.Model revision c4bc25be0e537d55266d90a08bdef931f4cf580a, quantization Q4_K_M, file SHA-256 7abd9859c10924be3c583945456fa9b25d0569484876105a7b7376f7938fffb7. Full-file hash matched the official model repository LFS metadata. No remote inference service is needed for this reproduction.
In a second terminal, from an empty working directory, save the prompt below as UTF-8
prompt.txtand run:The initial prompt was exactly:
There is an escaping ambiguity in that original regex wording; this is part of the experiment's limitation. The feedback run replaced it with the following plain-English contract and actual failures (a new session, same local model and settings):
Exact feedback prompt
The returned module and the independent check
Save this returned code unchanged as
generated.cjs:The first capture contains an entire decimal amount. For
"0.29", the optional second capture is absent, soparseInt(undefined, 10)becomesNaNand triggersRangeError. The pattern also fails to implement the required optional leading minus. The same module came back after feedback; it was not manually fixed for the figures.Save this as
check.cjs, then runnode check.cjs:Observed output: all five
FAIL valid ... RangeErrorlines; all elevenPASS rejection ...lines. Each row below comes from the saved execution results.Practical limits and next step
This used two tool-free generations of one task, with the same 16 synthetic tests; the feedback run saw those tests. It is a functional debugging example, not a model ranking, a financial parser ready for use, or evidence of general security. The original ambiguity, quantization, serving template, and model behavior have not been isolated from one another. Tool exclusion solved the tool-call boundary in this run; it did not solve the code defect.
For a code-only batch job I would keep three separate gates: expected Qwen exit status, zero permitted tool execution, and both valid-input and rejection assertions. Here the third gate correctly rejects the artifact. A useful follow-up is a held-out set, plus a controlled template/precision comparison; no successful parser is claimed from this run.
All reactions