Skip to content

perf: parallelize rule checks using ThreadPoolExecutor #95

Description

@YasserYG8

Description:

Describe the Feature

Currently, the evaluation engine (becwright.engine.evaluate) processes rules sequentially in a for loop:

results = []
for rule in rules:
    results.append(_run_check(rule, stdin, root))

If a repository has multiple rules-especially those calling external linters or scanners (like ruff, eslint, or semgrep) the total execution time is the sum of all check runtimes. This makes git commits slower as the ruleset grows.

Since rule checks are independent of each other and spend most of their time waiting on subprocess.run (which releases Python's GIL), they can be run in parallel.


Proposed Design

We should refactor the evaluate() function in src/becwright/engine.py to use Python's built-in concurrent.futures.ThreadPoolExecutor to run check subprocesses concurrently.

Expected Benefits:

  • Significant commit speedup: The overall check time will be bounded by the single slowest check, rather than the sum of all checks.
  • Efficient resource usage: Multi-core CPUs can execute the spawned processes concurrently.

Implementation Plan

  1. Gather all check tasks (mapping rule to its matched files).
  2. Execute the tasks concurrently using ThreadPoolExecutor.
  3. Gather and return the results in order.

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions