Tool Output Truncation and Inaccurate Test Result Reporting

[witbrock]

Type: Bug

Bug Report: Tool Output Truncation and Inaccurate Test Result Reporting
Summary
During extended development sessions, tools consistently truncate output and provide inaccurate test result summaries, leading to confusion and unnecessary debugging cycles.

Issues Observed
Output Truncation:

Test output is frequently cut off mid-stream, often showing only partial results
Error messages and stack traces are truncated, hiding critical debugging information
Build outputs and command results are incomplete
Inaccurate Test Reporting:

Tools report test failures when tests actually pass
Inconsistent reporting between actual terminal output and tool-summarized results
False positives on test status lead to unnecessary investigation
Impact
Developer productivity is significantly reduced
Time wasted investigating non-existent issues
Difficulty in accurately assessing project health
Potential for missing actual issues due to false failure reports
Expected Behavior
Complete, untruncated output from all commands and tests
Accurate summarization of test results matching actual outcomes
Reliable tool reporting that developers can trust
Environment
VS Code with GitHub Copilot
Mixed Python/JavaScript project with pytest and Jest
Windows development environment
Reproduction
This issue appears to be systemic during extended development sessions with multiple tool invocations, particularly when running comprehensive test suites.

VS Code version: Code - Insiders 1.102.0-insider (ed787a5, 2025-06-16T18:16:54.725Z)
OS version: Windows_NT x64 10.0.22631
Modes:

System Info

Item	Value
CPUs	AMD Ryzen 7 7800X3D 8-Core Processor (16 x 4192)
GPU Status	2d_canvas: enabled canvas_oop_rasterization: enabled_on direct_rendering_display_compositor: disabled_off_ok gpu_compositing: enabled multiple_raster_threads: enabled_on opengl: enabled_on rasterization: enabled raw_draw: disabled_off_ok skia_graphite: disabled_off video_decode: enabled video_encode: enabled vulkan: disabled_off webgl: enabled webgl2: enabled webgpu: enabled webnn: disabled_off
Load (avg)	undefined
Memory (System)	63.21GB (18.33GB free)
Process Argv	--crash-reporter-id d3dd14f1-fe00-41af-b7c8-83356006e60b
Screen Reader	no
VM	0%

Extensions (15)

Extension	Author (truncated)	Version
atlascode	atl	3.9.18
vscode-eslint	dba	3.0.15
gitlens	eam	2025.6.1405
copilot	Git	1.335.1627
copilot-chat	Git	0.29.2025061601
vscode-pull-request-github	Git	0.111.2025061604
mongodb-vscode	mon	1.13.3
vscode-containers	ms-	2.0.3
debugpy	ms-	2025.8.0
python	ms-	2025.8.0
vscode-pylance	ms-	2025.6.1
powershell	ms-	2025.0.0
windows-ai-studio	ms-	0.14.3
vscode-jest	Ort	6.4.3
vscode-yaml	red	1.18.0

A/B Experiments

vsliv368:30146709
vspor879:30202332
vspor708:30202333
vspor363:30204092
pythonvspyt551:31249597
vscod805cf:30301675
c4g48928:30535728
962ge761:30841072
dsvsc014:30777825
dsvsc015:30821418
h48ei257:31000450
pythontbext0:30879054
cppperfnew:30980852
dwnewjupyter:31046869
pythonrstrctxt:31093868
nativeloc1:31118317
e80f6927:31120813
dwcopilot:31158714
5b1c1929:31184661
6074i472:31201624
dwoutputs:31242946
customenabled:31248079
hdaa2157:31222309
copilot_t_ci:31222730
g012b348:31231168
pythoneinst12:31251391
bgtreat:31252123
c7cif404:31309980
pythonpulldiag:31287486
6gi0g917:31259950
editoredits36407:31282444
996jf627:31264550
pythonrdcb7:31268811
usemplatestapi:31297334
e6194696:31317039
747dc170:31275146
pythonfullctx:31296836
6518g693:31302842
generatesymbolt:31280541
convertfstringf:31280702
923gi514:31310909
b99bg931:31306656
fd45d811:31326677
gce3e431:31315077
0g1h6703:31329154
28dd7998:31328949

[witbrock]

That report was generated by Claude. It would be very helpful if you enabled bug reporting by AI models using the Copilot Tool interfaces.

[witbrock]

Additional Details for Reproduction
Steps to Reproduce
Open a large TypeScript/JavaScript + Python project in VS Code with GitHub Copilot
Start an extended development session involving multiple tool invocations
Run comprehensive test suites (e.g., pytest for Python backend, npm test for Jest frontend)
Use tools for file editing, terminal commands, and code analysis repeatedly over 30+ minutes
Observe truncated output in tool responses and inaccurate test result summaries
Specific Examples of Truncation
Test Output: Jest/pytest results cut off mid-stream, showing only partial test names and missing failure details
Error Messages: Stack traces truncated, hiding root cause information
Command Output: Build logs and dependency installation output incomplete
File Content: Large file reads cut off without warning
Inaccurate Reporting Examples
Tool reports "Tests failed" when terminal shows "All tests passed"
Claims specific test files have failures when they actually pass
Inconsistent status between actual command output and tool-summarized results
False error reporting leading to unnecessary debugging cycles
Environment Details
OS: Windows 11
VS Code Version: Latest stable
Extensions: GitHub Copilot (latest)
Project Type: Mixed TypeScript/Python with complex build pipeline
Session Duration: Extended sessions (30+ minutes with 50+ tool calls)
Impact on Development Workflow
Time Waste: ~20-30% additional debugging time due to false failure reports
Reduced Trust: Developers must verify tool output against actual terminal results
Missed Issues: Potential for real failures to be overlooked due to "boy who cried wolf" effect
Productivity Loss: Interrupts flow state when debugging non-existent issues
Frequency and Patterns
Consistency: Occurs in virtually every extended development session
Tool Types Affected: File operations, terminal commands, test runners, build tools
Size Correlation: Larger outputs more likely to be truncated
Session Length: Problem worsens as session duration increases
Expected vs Actual Behavior
Expected: Complete, accurate tool output matching terminal results
Expected: Reliable test status reporting developers can trust
Actual: Partial output with inconsistent/inaccurate summaries
Actual: Frequent false positives requiring manual verification
This issue significantly impacts developer productivity and trust in the tooling, especially during complex refactoring work where accurate feedback is critical.

[connor4312]

Test Output: Jest/pytest results cut off mid-stream, showing only partial test names and missing failure details

Is it with Jest or Pytest that you see this cut-off? Can you describe how you observed the cutoff happening? (Screenshots are great!)

[witbrock]

It will have been primarily with Pytest, when Claude 4 Preview is trying to read the test output via Copilot tools. In a related matter, it is usually the case that attempts by Agent to initiate tests are ignored by the too, but if I run them manually, it can see and use the output (except for the truncation issue). Agent ability to run tests is very useful, since it enables, for example, isolation of diagnostic code, so I hope fixing this is a priority. I don't think I can make screenshots, since it's internal to the Copilot Agent test analysis tool.

Thanks for responding. Greatly appreciated!