Feat/fix enametoolong devin provider

[djscheuf]

Summary

Extends on #9 , with clean-up and more complete integration, plus testing on Linux.

On Windows, passing large prompts (>32k characters) as CLI arguments to the Devin CLI causes an ENAMETOOLONG error due to the operating system's command-line length limit. This PR adds a new file-based provider (devinProvider.js) that writes prompts to a temporary file and passes them via the --prompt-file flag, bypassing the command-line length limitation entirely.

• Now includes clean-up of the old provider, and additional configuration options for the new provider. Updated docs to show intended use

Changes

devinProvider.js (new file)

• Implements a promptfoo custom provider class (DevinProvider) that exports the standard id() and callApi() interface
• Writes the full prompt to a temporary file in the OS temp directory, then invokes Devin CLI with --prompt-file instead of passing the prompt as a positional CLI argument
• Uses async spawn (instead of spawnSync) for non-blocking execution
• Parses token usage from Devin's --export JSON output, aggregating input/output/cache tokens from agent steps
• Supports both provider mode and grader mode (auto-detected from prompt format, same as devin.js)
• Cleans up temp files in a finally block to avoid leftover artifacts

promptfooconfig.yaml (modified)

• Adds a commented-out provider entry referencing file://devinProvider.js with the label Devin SWE-1.6 (file provider)
• Left commented out so users can enable it as needed without affecting the default configuration

Design Decisions

• Original devin.js provider left in place: The existing exec-based devin.js provider remains active and unchanged. This ensures ongoing testing is not disrupted. The new file-based provider is available as an opt-in alternative.
• Provider entry commented out in config: The new provider is validated and ready to use but is not enabled by default. Users can uncomment the entry in promptfooconfig.yaml to switch to the file-based provider.

Validation

• Ran npx promptfoo eval --no-cache with both the exec-based and file-based providers enabled
• 14/14 tests passed (100%) — no errors, no failures
• Both providers produced correct output across all test cases (deterministic, rubric-based, and script-based evaluations)
• Token usage tracking confirmed working via --export parsing
• confirmed with both demo config and trust-ai config.

Next Steps

• Cross-platform testing: Additional testing on macOS and Linux(now confirmed) is needed before unifying providers. The temp file approach should work cross-platform, but this has only been validated on Windows so far.
• Provider unification: Once cross-platform validation is complete, the plan is to update other providers (gh_copilot.js, claude.js, agy.js, kiro.js) with the same file-based pattern to prevent ENAMETOOLONG across all providers.
• Enable by default: After validation, consider replacing the exec-based devin.js provider with the file-based approach as the default. (added some logic to either force file-based, or dynamically use file-based depending on prompt length)