Skip to content
Merged
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
60 changes: 56 additions & 4 deletions src/ParallelTestRunner.jl
Original file line number Diff line number Diff line change
Expand Up @@ -200,7 +200,59 @@ function runtest(::Type{TestRecord}, f, name)
return res
end

function runtests(ARGS, testfilter = _ -> true, RecordType = TestRecord)
"""
runtests(ARGS, testfilter = Returns(true), RecordType = TestRecord)

Run Julia tests in parallel across multiple worker processes.

## Arguments

- `ARGS`: Command line arguments array, typically from `Base.ARGS`. When you run the tests with `Pkg.test`, this can be changed with the `test_args` keyword argument
- `testfilter`: Optional function to filter which tests to run (default: run all tests)
- `RecordType`: Type of test record to use for tracking test results (default: `TestRecord`)

## Command Line Options

- `--help`: Show usage information and exit
- `--list`: List all available test files and exit
- `--verbose`: Print more detailed information during test execution
- `--quickfail`: Stop the entire test run as soon as any test fails
- `--jobs=N`: Use N worker processes (default: based on CPU threads and available memory)
- `TESTS...`: Filter tests by name, matched using `startswith`

## Behavior

- Automatically discovers all `.jl` files in the test directory (excluding `setup.jl` and `runtests.jl`)
- Sorts tests by file size (largest first) for load balancing
- Launches worker processes with appropriate Julia flags for testing
- Monitors memory usage and recycles workers that exceed memory limits
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We should use the environment flag from Julia Base here

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Uhm, this is a suggestion about the implementation, not the docstring, right?

- Provides real-time progress output with timing and memory statistics
- Handles interruptions gracefully (Ctrl+C)
- Returns nothing, but throws `Test.FallbackTestSetException` if any tests fail

## Examples

```julia
# Run all tests with default settings
runtests(ARGS)

# Run only tests matching "integration"
runtests(["integration"])

# Run with custom filter function
runtests(ARGS, test -> occursin("unit", test))

# Use custom test record type
runtests(ARGS, Returns(true), MyCustomTestRecord)
```

## Memory Management

Workers are automatically recycled when they exceed memory limits to prevent
out-of-memory issues during long test runs. The memory limit is set based on
system architecture.
"""
function runtests(ARGS, testfilter = Returns(true), RecordType = TestRecord)
do_help, _ = extract_flag!(ARGS, "--help")
if do_help
println(
Expand Down Expand Up @@ -260,10 +312,10 @@ function runtests(ARGS, testfilter = _ -> true, RecordType = TestRecord)

append!(tests, files)
for file in files
test_runners[file] = joinpath("$WORKDIR", "$file.jl")
test_runners[file] = joinpath(WORKDIR, file * ".jl")
end
end
sort!(tests; by = (file) -> stat(joinpath("$WORKDIR", "$file.jl")).size, rev = true)
sort!(tests; by = (file) -> stat(joinpath(WORKDIR, file * ".jl")).size, rev = true)
## finalize
unique!(tests)

Expand Down Expand Up @@ -292,7 +344,7 @@ function runtests(ARGS, testfilter = _ -> true, RecordType = TestRecord)
memory_jobs = Int64(Sys.free_memory()) ÷ (2 * 2^30)
jobs = max(1, min(jobs, memory_jobs))
end
@info "Running $jobs tests in parallel. If this is too many, specify the `--jobs` argument to the tests, or set the `JULIA_CPU_THREADS` environment variable."
@info "Running $jobs tests in parallel. If this is too many, specify the `--jobs=N` argument to the tests, or set the `JULIA_CPU_THREADS` environment variable."

# add workers
test_exeflags = Base.julia_cmd()
Expand Down