Skip to content

Tests4Py 1.0.0

Latest
Latest

Choose a tag to compare

View all tags
@smythi93 smythi93 released this 23 Jul 20:12
v1.0.0
3e2328f
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.

First stable release. Every reproducible bug in the benchmark is now fully supported — each ships an oracle, a unit-test generator, a system-test generator, a harness, a valid grammar, and a curated diversity set of 10 passing + 10 failing tests. Every failing test is fault-distinguishing: it fails on the buggy build and passes on the fixed build.

Coverage

  • 328 / 328 reproducible bugs fully supported across 20 projects (ansible, black, calculator, cookiecutter, expression, fastapi, httpie, keras, luigi, markup, matplotlib, middle, pysnooper, sanic, scrapy, spacy, thefuck, tornado, tqdm, youtubedl).
  • 328 reproducible = 338 registered bugs − 10 non-evaluatable (test_status_buggy == PASSING).
  • Includes 18 bugs that stock Tests4Py could not distinguish because the fixed build failed its own canonical test (test_status_fixed == FAILING): 14 unlocked by the merge-commit checkout fix and 4 via bespoke oracles.
  • The 25 keras bugs whose fault requires TensorFlow (unavailable on Apple Silicon) were built and validated on x86_64 Linux.

Added

  • Optional Docker backend (t4p docker {env,project,instance,run,mode}): a three-layer image model (environment → project → instance) so the toolchain, interpreter, and dependency layers are built once and reused via Docker's layer cache. Toggle with mode = pyenv | docker (default pyenv). Fully additive; the pyenv backend is untouched.
  • Per-subject acceptance suite (tests/test_subjects.py, subjects marker): grammar validity, diversity behaviour, and generation per bug. Excluded from CI; run locally with pytest -m subjects.
  • Oracles, generators, harnesses, grammars, and 10+10 diversity resources for every supported bug.

Fixed

  • Merge-commit checkout. When a fix commit is a merge, git show --name-only lists only the merge's combined-diff files, so the source fix was never overlaid onto the fixed build — leaving it byte-identical to buggy. Checkout now detects a merge fix commit and locates the changed files via git diff <buggy>..<fixed>. Non-merge commits are unaffected.

Changed

  • Version bumped to 1.0.0.

See CHANGELOG.md for details.

Assets 4
Loading