Testing

Testing Strategy

The Numan project employs a multi-tiered testing strategy designed to ensure the reliability of a cross-platform package manager that interacts deeply with the Nushell environment. The strategy balances fast, isolated unit tests with comprehensive integration tests and real-world acceptance tests that invoke actual Nushell binaries. A "test-first" approach is mandated, where developers are expected to write tests, verify failures, implement the feature, and finally verify passing status.

The scope of testing covers core domain logic (registry, resolution, integrity), state management (lockfiles, journals), and external integrations (Nushell autoloads, nupm compatibility). Key architectural invariants, such as ensuring that the install command remains inert and never touches the Nu environment, are enforced through these automated gates.

Testing Tiers and Methodology

Numan categorizes its tests into three distinct levels to manage execution time and dependency requirements.

Unit Tests

Unit tests are located inline within source modules. They focus on pure domain logic, such as platform detection, version resolution, and metadata parsing. To maintain speed and isolation, unit tests must never spawn a real Nushell process; instead, they use injectable seams.

Integration Tests

Integration tests reside in the tests/ directory. These tests exercise multi-component workflows, such as the full install transaction or module activation sequences. They utilize tempfile::TempDir to simulate isolated Numan roots, preventing side effects on the developer's actual environment.

Real-Nu Acceptance Tests

These tests are marked with the #[ignore] attribute and are only executed when a real nu binary is present on the system PATH. They verify that Numan's generated artifacts (like use statements) are syntactically valid and functional within the actual Nushell interpreter.

Test Category	Location	Execution Command	Mandatory	Dependency
Unit	src/**/*.rs	cargo test <module>	Yes	None
Integration	tests/*.rs	cargo test	Yes	None (uses mocks)
Acceptance	tests/*.rs	cargo test -- --ignored	Yes (CI/Pre-release)	Real nu binary

Mocking and Test Seams

A critical part of the Numan testing strategy is the use of injectable traits to avoid side effects during testing, particularly regarding Nushell interaction and filesystem mutations.

Candidate Runner Abstraction

Numan uses the CandidateRunner trait to abstract the execution of Nushell for candidate validation. In production, NuCandidateRunner spawns the actual Nu binary. In tests, FakeCandidateRunner is injected to simulate success or failure without a Nu dependency.

classDiagram
    class CandidateRunner {
        <<interface>>
        +run(candidate: Path) Result
    }
    class NuCandidateRunner {
        +nu_executable: String
        +run(candidate: Path) Result
    }
    class FakeCandidateRunner {
        +should_succeed: bool
        +error_msg: String
        +run(candidate: Path) Result
    }
    CandidateRunner <|-- NuCandidateRunner
    CandidateRunner <|-- FakeCandidateRunner

Loading

The diagram shows the abstraction layer allowing tests to bypass real process spawning.

Filesystem Isolation

Tests use tempfile::TempDir to create a ephemeral NUMAN_ROOT. This allows tests to verify that files like lockfile, autoload-state.json, and journals are written correctly without affecting the host system.

Activation and Autoload Validation

Testing the activate and deactivate commands involves verifying the state machine transitions between the lockfile, the autoload journal, and the managed external file (numan.nu).

flowchart TD
    Start[Test Start] --> MockPaths[Mock NuPaths & Environment]
    MockPaths --> MockPayload[Create Dummy Module Payload]
    MockPayload --> RunAct[Invoke execute_with_candidate_runner]
    RunAct --> CheckFile[Verify numan.nu Ownership Marker]
    CheckFile --> CheckState[Verify autoload-state.json Projection]
    CheckState --> CheckLock[Verify Lockfile module_activation]
    CheckLock --> CheckJournal[Verify Journal is Cleared]

Loading

This flow illustrates the verification steps for a successful module activation integration test.

Key Validation Areas:

• Path Escaping: Tests verify that paths containing spaces, Unicode, backslashes (Windows), and quotes are correctly escaped in the generated Nu use statements.
• Deterministic Ordering: Verification that numan.nu content is sorted alphabetically by scoped ID regardless of installation order.
• Ownership Safety: Tests ensure Numan refuses to overwrite or delete files that do not contain the OWNERSHIP_MARKER.

Compatibility Testing (nupm)

The compatibility testing strategy for nupm involves a "fixture-driven" approach. A corpus of supported and rejected nupm package layouts is maintained to test the classifier and metadata parser.

Fixture Corpus

Fixtures include:

• Supported: Minimal modules with valid nupm.nuon.
• Rejected: Packages with build.nu (unsupported custom builds), external dependencies, or scripts.
• Unsafe: Packages containing symlinks that escape the package root.

Drift and Provenance Verification

Integration tests for nupm import verify that:

1. The source directory remains byte-for-byte unchanged after import.
2. The imported payload is copied to an immutable path in the Numan root.
3. numan nupm diff correctly identifies when a source nupm package has changed relative to the Numan import.

CI and Quality Gates

The testing strategy is enforced via a CI pipeline that runs on Linux, macOS, and Windows.

1. Format and Lint: CI enforces cargo fmt --check and cargo clippy -- -D warnings.
2. Automated Test Suite: Runs all non-ignored unit and integration tests (approx. 376 tests).
3. MSRV Job: Ensures the code compiles and tests pass on the Minimum Supported Rust Version (1.88).
4. Real-Nu Job: A specialized CI job that installs Nushell 0.113+ and runs cargo test -- --ignored to verify environment-sensitive activation logic.

Summary

Numan's testing strategy ensures system stability by isolating pure logic from environment-dependent interactions. By using mock runners for standard tests and reserved "ignored" tests for real-world validation, the project maintains high velocity without sacrificing reliability across its supported platforms. The extensive use of a fixture corpus for nupm compatibility further ensures that external package formats are handled predictably and safely.