[Bug] Codex JSONL parser silently truncates output when a line exceeds 1 MiB

[2190829778]

Description

When the Codex engine emits a JSONL record larger than 1 MiB, skill-up silently stops parsing the output and retains an earlier assistant message as SessionResult.FinalMessage.

For an agent_judge run, this can produce a misleading error:

agent_judge failed to parse agent output:
no valid JSON found in agent output (length=...)

The Judge may have produced valid final JSON later in the same stdout.json, but skill-up never reaches it.

This is especially easy to trigger when the Judge inspects a large transcript with rg, grep, or another command that writes many matches into one command_execution.aggregated_output field.

Steps to reproduce

1. Configure an evaluation using:

   engine:
     name: codex

   and an agent_judge.

2. During the Judge run, execute a command whose output is large enough for one Codex JSONL record to exceed 1,048,576 bytes.

   For example, search a large transcript without limiting the number or size of matching lines.

3. Let the Judge emit a valid final JSON response after that command:

   {
     "results": [
       {
         "criterion": "example criterion",
         "passed": true,
         "evidence": "example evidence"
       }
     ]
   }

4. Run the evaluation:

   skill-up run evals/eval.yaml

5. Inspect:

   <report>/<case>/with_skill/outputs/judge/run/stdout.json
   

6. Compare:

   • the first JSONL line larger than 1 MiB;
   • the last assistant message before that line;
   • the valid final agent_message after that line;
   • the length reported by no valid JSON found.

Expected behavior

One of the following should happen:

1. skill-up successfully reads the complete Codex output and uses the final Judge JSON; or
2. skill-up returns an explicit output-scanning error identifying the oversized JSONL record.

If --output-last-message produced a valid last-message artifact, that artifact should be used as the authoritative final response instead of an earlier progress message parsed from incomplete stdout.

Actual behavior

skill-up stops scanning at the first JSONL line larger than 1 MiB.

The scan error is not returned or logged. The parser keeps the last assistant progress message seen before the oversized line and treats it as the final response.

agent_judge then attempts to parse that progress message as JSON and reports:

no valid JSON found in agent output

This makes a valid Judge result appear to be an evaluation infrastructure failure.

Observed evidence

I reproduced this in two independent Judge runs.

Run A

• First oversized physical JSONL line: line 20

• Line size: 1,117,390 UTF-8 bytes

• Record type: completed command_execution

• aggregated_output size: 942,696 characters

• Valid final Judge JSON: line 35

• Final Judge JSON size: 4,780 characters

• Last assistant message before the oversized line: line 17

• That earlier message size: exactly 199 UTF-8 bytes

• Reported error:

  no valid JSON found in agent output (length=199)
  

Run B

• Oversized physical JSONL lines:

  • line 16: 1,142,738 UTF-8 bytes
  • line 17: 1,140,287 UTF-8 bytes

• Corresponding aggregated_output sizes:

  • 1,007,299 characters
  • 983,543 characters

• Valid final Judge JSON: line 36

• Final Judge JSON size: 6,867 characters

• Last assistant message before the first oversized line: line 12

• That earlier message size: exactly 243 UTF-8 bytes

• Reported error:

  no valid JSON found in agent output (length=243)
  

All physical records in both stdout.json files are valid JSON when read with a parser that supports lines larger than 1 MiB.

The final Judge agent_message in both runs is also valid JSON.

As a control, another run whose largest JSONL line was 954,948 bytes completed Judge parsing normally.

Suspected root cause

At commit c6ca37aeb6c36eca1141101d0632401414957705, parseCodexOutput uses a bufio.Scanner with a hard 1 MiB maximum:

scanner := bufio.NewScanner(strings.NewReader(output))
const maxTokenLen = 1024 * 1024
buf := make([]byte, maxTokenLen)
scanner.Buffer(buf, maxTokenLen)

for scanner.Scan() {
    // ...
}

Source:

skill-up/internal/agent/codex.go

Lines 683 to 704 in c6ca37a

	func parseCodexOutput(ctx context.Context, output string) codexOutputParseResult {
	state := codexParseState{
	commandExecutions: make(map[string]codexCommandState),
	mcpToolCalls: make(map[string]int),
	}
	scanner := bufio.NewScanner(strings.NewReader(output))
	const maxTokenLen = 1024 * 1024
	buf := make([]byte, maxTokenLen)
	scanner.Buffer(buf, maxTokenLen)
	for scanner.Scan() {
	line := strings.TrimSpace(scanner.Text())
	if line == "" || !strings.HasPrefix(line, "{") {
	continue
	}
	applyCodexEventsFromLine(ctx, &state, line)
	}
	return codexOutputParseResult{
	transcript: state.messages,

There is no scanner.Err() check after the scan loop.

parseCodexSessionFile has the same 1 MiB limit and also does not propagate the scanner error:

skill-up/internal/agent/codex.go

Lines 929 to 978 in c6ca37a

	func parseCodexSessionFile(sessionFile string) codexSessionParseResult {
	file, err := os.Open(sessionFile)
	if err != nil {
	return codexSessionParseResult{}
	}
	defer file.Close() //nolint:errcheck
	var messages transcript.Transcript
	var finalMsg string
	var inputTokens, outputTokens int
	currentTurn := 0
	scanner := bufio.NewScanner(file)
	const maxTokenLen = 1024 * 1024
	buf := make([]byte, maxTokenLen)
	scanner.Buffer(buf, maxTokenLen)
	for scanner.Scan() {
	line := strings.TrimSpace(scanner.Text())
	if line == "" || line == "[]" || line == "{}" {
	continue
	}
	var event codexSessionEvent
	if err := json.Unmarshal([]byte(line), &event); err != nil {
	continue
	}
	if event.Payload == nil {
	continue
	}
	switch event.Type {
	case "response_item":
	turnMsgs, final, turn := parseCodexSessionResponseItem(*event.Payload, currentTurn)
	if turnMsgs != nil {
	messages = append(messages, turnMsgs...)
	}
	if final != "" {
	finalMsg = final
	}
	currentTurn = turn
	case "event_msg":
	msgs, final, in, out := applyCodexSessionEventMsg(*event.Payload, currentTurn)
	if msgs != nil {
	messages = append(messages, msgs...)
	}
	if final != "" {
	finalMsg = final
	}
	inputTokens = max(inputTokens, in)

There is an additional fallback issue in resolveCodexLastMessage:

if finalMsg != "" || lastMessagePath == "" {
    return finalMsg, trans, generatedFiles, func() {}
}

Source:

skill-up/internal/agent/codex.go

Lines 508 to 532 in c6ca37a

	func resolveCodexLastMessage(ctx context.Context, rt Runtime, artifactDir, lastMessagePath, finalMsg string, trans transcript.Transcript, generatedFiles []string) (string, transcript.Transcript, []string, func()) {
	if finalMsg != "" || lastMessagePath == "" {
	return finalMsg, trans, generatedFiles, func() {}
	}
	artifactPath, registeredPath, cleanup, ok := downloadSessionArtifact(ctx, rt, artifactDir, lastMessagePath)
	if !ok {
	return finalMsg, trans, generatedFiles, func() {}
	}
	if registeredPath != "" {
	generatedFiles = append(generatedFiles, registeredPath)
	}
	if data, err := os.ReadFile(artifactPath); err == nil {
	finalMsg = strings.TrimSpace(string(data))
	if finalMsg != "" {
	trans = append(trans, transcript.Message{
	Role: transcript.RoleAssistant,
	Content: finalMsg,
	Turn: max(codexTurns(trans), 1),
	})
	}
	}
	return finalMsg, trans, generatedFiles, cleanup
	}
	func buildCodexRunCmd(instruction, model string, provider codexProviderConfig, sandboxFlag string) string {

Because an earlier progress message was successfully parsed before the oversized line, finalMsg is non-empty. Therefore, the valid file produced through --output-last-message is not read.

Suggested fix

Please consider addressing all three parts:

1. Detect and propagate scanner.Err() from both:

   • parseCodexOutput
   • parseCodexSessionFile

2. Increase or remove the fixed 1 MiB physical-line limit.

   A streaming approach that supports arbitrarily large JSONL records would be safer than relying on another fixed upper bound.

3. Prefer the --output-last-message artifact as the authoritative final message when it exists, even when stdout parsing found an earlier assistant message.

   Alternatively, only trust the stdout-derived final message when parsing reached a complete terminal event without a scanner error.

Suggested regression test

Add a Codex parser test containing:

1. an assistant progress message;
2. a valid command_execution JSONL record larger than 1 MiB;
3. a valid final agent_message containing Judge JSON;
4. a terminal turn.completed event.

The expected result should assert that:

• the oversized record does not silently truncate parsing;
• the valid final message is returned;
• scanner errors are never ignored;
• agent_judge receives the final JSON rather than the earlier progress message.

A second test should verify that --output-last-message recovers the correct final response when stdout parsing is incomplete.

Environment

• skill-up version: skill-up version 0.7.0
• Go version: go1.26.5 windows/amd64
• OS: Microsoft Windows NT 10.0.26200.0
• Architecture: X64
• PowerShell: 5.1.26100.8875
• Engine: Codex

Additional context

The current main branch still has the same internal/agent/codex.go and internal/judge/agent_judge.go file contents as release v0.7.0, so updating from v0.7.0 to the current main branch does not appear to avoid this failure.