Conversational answers with no generated code/YAML are not parsed correctly in Lightning

[hanna-paasivirta]

When the services only answer with a text answer and don't generate any code, the answers are not parsed correctly:

Workflow chat (higher priority):
Structured output parsing error:

It then re-generates and the output is broken:

Job chat:
Most of the time the "apply changes" button appears even if it seems like the model has not generated any code, which is confusing. However, sometimes the button doesn't appear, so it might be marking empty job code keys inconsistently:

[hanna-paasivirta]

1. workflow_chat: Model puts ``` in yaml field and yaml.safe_load fails
2. Invalid JSON Model outputs literal newline in yaml value → json.loads fails → retry loop. Retry sometimes produces empty response.
3. job_chat: changes event fires with empty code_edits: [] → "apply changes" button appears in front-end

[hanna-paasivirta]

The streaming of JSON is quite tricky to fix. It seems that the model behaviour changed when we reversed the order of streaming to have the code arrive first, and we now get more inconsistent output formats than before. I tried a more intuitive way of marking empty code keys with None instead of empty strings, and adding in more examples, but this is still not reliable enough. I tried switching to Sonnet 4.6 with structured outputs instead of answer pre-filling, and this seems to solve the problems entirely and doesn't seem to have the bugs I previously experienced with the beta version #310