[WIP] Split file writing step for JavaScript sources

[Copilot]

Implementation Plan: Split Large JavaScript Source File Writing

Problem

The compiler writes multiple JavaScript source files in a single step using heredoc commands (cat > file << 'EOF'). When the total size of all these heredoc commands exceeds 21000 characters, it violates GitHub Actions' step size limits, causing workflow compilation failures.

Solution

Implemented automatic chunking of JavaScript file writes to split large steps into multiple smaller steps that stay under the 21KB limit.

Implementation Checklist

• Analyze the problem and identify affected code locations
• Create a helper function to split JavaScript file writes into multiple steps
• Apply the splitting logic to "Write Safe Outputs JavaScript Files" step
• Apply the splitting logic to "Setup Safe Inputs JavaScript and Config" step
• Create test cases to validate the splitting behavior
• Run unit tests to ensure no regressions
• Manually test with a large workflow to verify splitting works
• Run full validation - unit tests pass, integration test failure pre-existing
• Code review and final validation

Changes Made

1. Added JavaScriptFileWrite struct (mcp_servers.go)

• Represents a JavaScript file to be written with its metadata
• Fields: Filename, Content, EOFMarker, TargetPath

2. Created writeJavaScriptFilesInChunks() helper function (mcp_servers.go)

• Calculates the size of each file write operation including:
  • Header: cat > <path> << '<EOF>'
  • Content: formatted JavaScript with indentation
  • Footer: EOF marker
• Groups files into chunks that stay under 20,900 bytes (21KB - 100 byte buffer)
• Generates separate steps for each chunk with numbered names (e.g., "part 1", "part 2")
• Preserves file order across chunks

3. Refactored safe-outputs JavaScript writing (mcp_servers.go)

• Converted from inline heredoc writing to using writeJavaScriptFilesInChunks()
• Collects all JavaScript files into a list before writing
• Separated chmod command into its own step

4. Refactored safe-inputs JavaScript writing (mcp_servers.go)

• Converted to use writeJavaScriptFilesInChunks()
• Separated directory creation, JS files, config, and chmod into distinct steps
• Improved organization and readability

5. Added comprehensive tests (mcp_servers_test.go)

• TestJavaScriptFileChunking: Tests single/multiple files and automatic chunking
• TestJavaScriptFileChunkingPreservesOrder: Validates file order preservation
• Tests with various file sizes to ensure proper chunking behavior

Manual Testing Results

Compiled safe-output-health.md:

• Safe-outputs JavaScript files split into 4 parts ✓
• Total size: 343.6 KB
• Steps: "Write Safe Outputs JavaScript Files (part 1-4)"

Compiled test-python-safe-input.md:

• Safe-inputs JavaScript files split into 4 parts ✓
• Total size: 381.5 KB
• Steps: "Setup Safe Inputs JavaScript (part 1-4)"

Test Results

Unit Tests: ✅ PASS - All unit tests pass without regressions

Integration Tests: One pre-existing failure in TestSafeOutputsMCPBundlerIntegration

• This test was already failing on the main branch before my changes
• The test expects merged child_process imports from bundling
• The codebase writes unbundled JavaScript files, so the test expectations don't match
• My changes did not introduce this failure

Technical Details

Size Calculation:

• Header size: "cat > <path> << '<EOF>'\n"
• Content size: Sum of all formatted lines (includes indentation)
• Footer size: "<EOF>\n"
• Buffer: 100 bytes safety margin below 21KB limit

Chunking Algorithm:

1. Calculate size for each file write operation
2. Group files sequentially until adding next file would exceed limit
3. Start new chunk when limit would be exceeded
4. Generate numbered step names when multiple chunks exist

Benefits

✅ Prevents compilation failures for workflows with large JavaScript dependencies
✅ Maintains file order and dependencies
✅ Automatic and transparent - no changes needed to workflow definitions
✅ Follows existing pattern used for prompt content splitting
✅ Well-tested with comprehensive unit tests

Original prompt

The compiler should split the step that write multiple JavaScript source file into multiple steps to stay under the 21000 character limit (there are already helpers to detect too large payload).

---

💡 You can make Copilot smarter by setting up custom instructions, customizing its development environment and configuring Model Context Protocol (MCP) servers. Learn more Copilot coding agent tips in the docs.