🧪 Test gap analysis — 5 gaps found in runtime extension validate() paths and feed-url integration coverage

[github-actions]

Test Gap Analysis

Test suite snapshot: 1,337 unit tests, 85 integration tests (tests/compiler_tests.rs), 3 init tests, 8 MCP HTTP tests — 1,439 total. All pass. ✅

Previous issue #392 was closed as completed — the upload dispatch and MCP tools-list gaps were fixed. #376 remains open. The gaps below are new: all four runtime extension files (python, node, dotnet, lean) have validate() implementations with multiple error/warning paths and zero unit tests. Only the happy-path compilation is covered end-to-end.

---

Priority Gaps

Module	Function/Path	Why It Matters	Suggested Test
runtimes/python/extension.rs	validate() — 5 branches (bash disabled, config+feed-url bail!, config-only warning, invalid feed URL, version injection), 0 tests	The bail! for mutually-exclusive config+feed-url is a user-facing error path that is completely untested — a regression could silently accept an invalid config	Unit test each branch with a synthetic CompileContext
runtimes/node/extension.rs	validate() — same 5-branch structure as Python, 0 tests	Identical risk: the config+feed-url mutual exclusivity bail! and version injection rejection are untested	Mirror python extension tests
runtimes/dotnet/extension.rs	validate() — 7 branches including global.json conflict detection and GLOBAL_JSON_SENTINEL skip logic, 0 tests	The global.json conflict path checks the filesystem (ctx.compile_dir.join("global.json").exists()) and bail!s — this is the most complex validation in any runtime extension and has no coverage at all	Create a temp dir with/without global.json and assert correct behaviour
tests/compiler_tests.rs	No test_python_runtime_with_feed_url_compiled_output	PythonExtension::prepare_steps() has a conditional branch: PipAuthenticate is only emitted when feed-url is set. The PIP_INDEX_URL and UV_DEFAULT_INDEX env vars injected via agent_env_vars() are never tested. dotnet has a test_dotnet_runtime_with_feed_url_compiled_output — python should have an equivalent.	Add integration test compiling a python agent with runtimes.python.feed-url set
tests/compiler_tests.rs	No test_node_runtime_with_feed_url_compiled_output	NodeExtension::prepare_steps() emits generate_ensure_npmrc + npmAuthenticate@0 when feed-url or config is set. The NPM_CONFIG_REGISTRY env var from agent_env_vars() is never tested.	Add integration test compiling a node agent with runtimes.node.feed-url set

---

Suggested Test Cases

1. python/extension.rs — validate() error paths

// In src/runtimes/python/extension.rs tests block
#[cfg(test)]
mod tests {
    use super::*;
    use crate::compile::extensions::CompileContext;
    use crate::compile::types::FrontMatter;

    fn make_ctx(agent_name: &str) -> CompileContext<'_> {
        CompileContext {
            agent_name: agent_name.to_string(),
            front_matter: FrontMatter::default(),
            compile_dir: None,
        }
    }

    #[test]
    fn test_validate_config_and_feed_url_are_mutually_exclusive() {
        let config = PythonRuntimeConfig::WithOptions(PythonOptions {
            version: None,
            feed_url: Some("(pkgs.dev.azure.com/redacted),
            config: Some("pip.conf".into()),
        });
        let ext = PythonExtension::new(config);
        let result = ext.validate(&make_ctx("test-agent"));
        assert!(result.is_err(), "config + feed-url must be mutually exclusive");
        assert!(result.unwrap_err().to_string().contains("mutually exclusive"));
    }

    #[test]
    fn test_validate_bash_disabled_emits_warning() {
        use crate::compile::types::{ToolsConfig, BashConfig};
        let config = PythonRuntimeConfig::Enabled(true);
        let ext = PythonExtension::new(config);
        let mut fm = FrontMatter::default();
        fm.tools = Some(ToolsConfig { bash: Some(vec![]), ..Default::default() });
        let ctx = CompileContext { agent_name: "agent".into(), front_matter: fm, compile_dir: None };
        let warnings = ext.validate(&ctx).unwrap();
        assert!(!warnings.is_empty(), "should warn when bash is disabled with python runtime");
        assert!(warnings[0].contains("tools.bash is empty"));
    }

    #[test]
    fn test_validate_invalid_feed_url_returns_err() {
        let config = PythonRuntimeConfig::WithOptions(PythonOptions {
            feed_url: Some("not-a-url".into()),
            ..Default::default()
        });
        let ext = PythonExtension::new(config);
        assert!(ext.validate(&make_ctx("agent")).is_err());
    }

    #[test]
    fn test_validate_injection_in_version_returns_err() {
        let config = PythonRuntimeConfig::WithOptions(PythonOptions {
            version: Some("3.12\n##vso[task.setvariable]x=y".into()),
            ..Default::default()
        });
        let ext = PythonExtension::new(config);
        assert!(ext.validate(&make_ctx("agent")).is_err());
    }

    #[test]
    fn test_prepare_steps_with_feed_url_includes_pip_authenticate() {
        let config = PythonRuntimeConfig::WithOptions(PythonOptions {
            feed_url: Some("(pkgs.dev.azure.com/redacted),
            ..Default::default()
        });
        let ext = PythonExtension::new(config);
        let steps = ext.prepare_steps();
        assert_eq!(steps.len(), 2, "should emit install + PipAuthenticate when feed-url is set");
        assert!(steps[1].contains("PipAuthenticate"), "second step should be PipAuthenticate");
    }

    #[test]
    fn test_agent_env_vars_with_feed_url() {
        let url = "(pkgs.dev.azure.com/redacted)
        let config = PythonRuntimeConfig::WithOptions(PythonOptions {
            feed_url: Some(url.into()),
            ..Default::default()
        });
        let ext = PythonExtension::new(config);
        let vars = ext.agent_env_vars();
        assert!(vars.iter().any(|(k, _)| k == "PIP_INDEX_URL"), "should set PIP_INDEX_URL");
        assert!(vars.iter().any(|(k, _)| k == "UV_DEFAULT_INDEX"), "should set UV_DEFAULT_INDEX");
    }
}

2. dotnet/extension.rs — global.json conflict detection

#[test]
fn test_validate_global_json_conflict_bails_when_version_and_file_both_present() {
    use std::fs;
    let dir = tempfile::tempdir().unwrap();
    let global_json = dir.path().join("global.json");
    fs::write(&global_json, r#"{"sdk":{"version":"8.0.100"}}"#).unwrap();

    let config = DotnetRuntimeConfig::WithOptions(DotnetOptions {
        version: Some("9.0".into()),
        ..Default::default()
    });
    let ext = DotnetExtension::new(config);
    let fm = FrontMatter::default();
    let ctx = CompileContext {
        agent_name: "agent".into(),
        front_matter: fm,
        compile_dir: Some(dir.path()),
    };
    let result = ext.validate(&ctx);
    assert!(result.is_err(), "should bail when global.json exists and explicit version is set");
    assert!(result.unwrap_err().to_string().contains("global.json"));
}

#[test]
fn test_validate_version_global_json_sentinel_is_accepted_with_file_present() {
    let dir = tempfile::tempdir().unwrap();
    fs::write(dir.path().join("global.json"), r#"{"sdk":{"version":"8.0.100"}}"#).unwrap();

    let config = DotnetRuntimeConfig::WithOptions(DotnetOptions {
        version: Some("global.json".into()),
        ..Default::default()
    });
    let ext = DotnetExtension::new(config);
    let fm = FrontMatter::default();
    let ctx = CompileContext {
        agent_name: "agent".into(),
        front_matter: fm,
        compile_dir: Some(dir.path()),
    };
    assert!(ext.validate(&ctx).is_ok(), "global.json sentinel should not conflict");
}

3. Integration test — python with feed-url

#[test]
fn test_python_runtime_with_feed_url_compiled_output() {
    let temp_dir = std::env::temp_dir().join(format!("agentic-pipeline-python-feed-{}", std::process::id()));
    fs::create_dir_all(&temp_dir).unwrap();
    let input = r#"---
name: "Python Feed Agent"
description: "Python agent with internal feed"
runtimes:
  python:
    feed-url: "(pkgs.dev.azure.com/redacted)
safe-outputs:
  noop: {}
---
## Python Feed Agent
"#;
    let input_path = temp_dir.join("python-feed-agent.md");
    let output_path = temp_dir.join("python-feed-agent.yml");
    fs::write(&input_path, input).unwrap();
    let output = std::process::Command::new(PathBuf::from(env!("CARGO_BIN_EXE_ado-aw")))
        .args(["compile", input_path.to_str().unwrap(), "-o", output_path.to_str().unwrap()])
        .output().unwrap();
    assert!(output.status.success(), "{}", String::from_utf8_lossy(&output.stderr));
    let compiled = fs::read_to_string(&output_path).unwrap();
    assert!(compiled.contains("PipAuthenticate@1"), "should include PipAuthenticate step");
    assert!(compiled.contains("PIP_INDEX_URL"), "should inject PIP_INDEX_URL env var");
    assert!(compiled.contains("UV_DEFAULT_INDEX"), "should inject UV_DEFAULT_INDEX env var");
    let _ = fs::remove_dir_all(&temp_dir);
}

---

Coverage Summary

Module	Public Fns	Unit Tests	Integration Tests	Coverage Gap
runtimes/python/extension.rs	validate (5 branches), prepare_steps (conditional), agent_env_vars	0	happy-path only	Error/conditional branches untested
runtimes/node/extension.rs	validate (5 branches), prepare_steps (conditional), agent_env_vars	0	happy-path only	Error/conditional branches untested
runtimes/dotnet/extension.rs	validate (7 branches incl. global.json check), prepare_steps (3 paths)	0	happy-path + feed-url	global.json conflict, version sentinel, injection checks all untested
runtimes/lean/extension.rs	validate (1 warning path)	0	happy-path only	Bash-disabled warning path untested
Integration (python feed-url)	prepare_steps conditional, agent_env_vars	—	0	PipAuthenticate, PIP_INDEX_URL, UV_DEFAULT_INDEX never checked
Integration (node feed-url)	prepare_steps conditional, agent_env_vars	—	0	npmAuthenticate, NPM_CONFIG_REGISTRY never checked

---

This issue was created by the automated test gap finder. Previous run: 2026-05-08 (#392, now closed ✅; #376 still open). Modules audited this cycle: runtimes/{python,node,dotnet,lean}/extension.rs. Total tests found: 1,439 (up from 1,317).

Generated by Test Gap Finder · ● 1.3M · ◷