Repository navigation
Releases
vibeaudit 0.11.1
Bug fixes from an external review.
Fixed
vibeaudit test-exploitno longer gives an authentication verdict ("401 or 403", "2xx without
authentication") for rules it has no automatic oracle for, such as BE006; it says to check the response by hand.vibeaudit fixreads a Python file's indentation from its code, not from text inside strings, so a docstring or
prompt indented by two spaces no longer makes it insert two-space blocks into a four-space file.- Python taint:
mapping.get(key)takes its taint from the mapping and the default, not the key, so a user-chosen
key into a server-side dict (FILES.get(request.args["name"])) is no longer a path traversal or injection source. - BE001 no longer reports
f"... FROM {self.table}"when the class checkedtablewithre.fullmatchagainst a
pattern that only accepts identifiers before storing it. - The per-rule table in
benchmark/RESULTS.mdshows a stable rule's precision on the findings that gate, leaving out
its findings in preview languages, as the headline numbers already did. externalin.vibeaudit.tomlis read by the Python API's default options, the MCP server andvibeaudit watch.
An inline comment had swallowed the key, so they always ran the external tools inautomode;vibeaudit scan
was not affected.
vibeaudit 0.11.0
Lexer-based scanners with labelled corpora for Rust, C# and PHP, which completes Phase 5 as preview; benchmark
numbers counted by each finding's own preview flag; and five rules back in preview after new held-out repositories.
Added
- Rust (Actix, Axum): a lexer; attribute macros,
.routebuilders,web::scopeand AxumRouter::route/
.nestroutes, with auth middleware (.wrap,.layer,.route_layer), guard attributes and auth extractors,
including a project's ownFromRequestPartsextractor that rejects with 401/403; request data followed into SQL
(sqlx, diesel, rusqlite), commands, outbound HTTP (reqwest), file paths and redirects;Cargo.tomldependencies
locked fromCargo.lockand checked against OSV, offline bundle included. - C# (ASP.NET Core): a lexer (verbatim, interpolated and raw strings); controller and minimal-API routes with
[Authorize]/[AllowAnonymous],RequireAuthorization(), route groups, fallback policies and a project's own
auth filters; request data followed into SQL (EF Core raw SQL, ADO.NET, Dapper), commands, outbound HTTP, file
paths and redirects; NuGet dependencies from.csprojandDirectory.Packages.propschecked against OSV. Three
preview rules: CSHARP001 (no authorization anywhere in the project), CSHARP002 (deserialization that lets the
payload pick the type: TypeNameHandling without a binder, BinaryFormatter), CSHARP003 (developer error page
outside Development). - PHP (Laravel, Symfony): a lexer (inline HTML, heredoc and nowdoc); Laravel routes with groups, resources,
middleware groups and controller middleware, and Symfony routes with#[IsGranted]andaccess_control; request
data followed into SQL, commands, eval, unserialize, outbound HTTP, file paths, redirects and echoed HTML,
including plain PHP pages; Blade{!! !!}output; Composer dependencies checked against OSV. - Benchmark: 11 Rust, 11 C# and 10 PHP repositories, every finding labelled, held-out repositories chosen and
pinned before scanning (106 repositories in all).
Changed
- Benchmark accounting: a finding counts as stable or preview by its own preview flag, not its rule's. Findings of
a stable rule in Go, Java/Kotlin, C#, Rust or PHP code are preview and do not gate, so "stable" in the README and
docs/BENCHMARK.md means exactly what--fail-onacts on. Stable precision is 94.4% tuned and 94.1% held-out. - Back to preview after new held-out repositories (STABILITY.md has the measurements): CFG002 (Unpinned container
base image, 5 of 6), QA007 (Package imported but not declared as a dependency, 6 of 7), DATA012 (Foreign key
without an index, 19 of 28), UI009 (Data loaded without loading or error state, 4 of 5) and QA001 (No automated
tests, 6 of 8). None is a severe class. QA001's cause is fixed (tests in every supported language now count:
_test.go,src/test, Rust test modules and files, .NET*.Testsprojects, PHP); it returns to stable once
measured on fresh held-out data. - AUTH002 and AUTH017 do not fire on an admin area of a project with no role concept anywhere, where every
signed-in user is the admin. - Project-wide evidence (rate limiting, roles) reads Go and Rust code.
- Fewer false positives, each from the tuned split: SEC003 on recorded API responses, translation folders, column
names and validation strings; SEC011 on validation messages; FE002 and FE007 on placeholder URLs and server-render
fallbacks; QA006 onexport const enum; DATA001 on Rustimpl ... for,#[cfg(test)]modules and PHP builder
calls; CFG009 on env files that set no value. - C# HTTP clients are no longer reported for a missing timeout: .NET's HttpClient times out after 100 seconds.
vibeaudit 0.10.0
Five more languages in preview, with lexer-based scanners and labelled corpora for Go and Java/Kotlin, and the
first release published to PyPI from the release workflow.
Added
- Go: a lexer; routes for net/http (including Go 1.22
"GET /path"patterns), Gin, Echo, Fiber and Chi, with
groups,Route/Mountprefixes and auth middleware, feeding the shared route checks (AUTH001-AUTH003, AUTH010,
LLM004); request data followed into SQL (database/sql, sqlx, GORM), commands, outbound HTTP, file paths and
redirects;go.moddependencies checked against OSV, offline bundle included. Four preview rules: GO001
(goroutine started by a handler that never stops), GO002 (response body never closed), GO003 (map written
concurrently without a lock), GO004 (SQL built with formatting). - Java and Kotlin (Spring Boot, Quarkus): a lexer; Spring
@*Mappingand JAX-RS routes with the Spring Security
chain (path and HTTP-method scoped matchers), Quarkus permissions and hand-written auth interceptors; request
data followed into SQL, commands, deserialization, outbound HTTP, file paths and redirects; Maven and Gradle
dependencies checked against OSV. Four preview rules: JAVA001 (controllers with no security configuration),
JAVA002 (Actuator endpoints exposed), JAVA003 (security chain permits every request), JAVA004 (polymorphic
deserialization without an allow-list). - C#, Rust and PHP: route and auth discovery, SQL and command injection, template XSS, N+1 queries in loops and
HTTP clients without a timeout. vibeaudit test-exploit: a proof-of-concept request for a finding, without destructive payloads. Dry run by default;--send
goes to a loopback address only (checked after DNS resolution) and never follows redirects.vibeaudit.runtime: FastAPI and Express middleware for staging, off unlessVIBEAUDIT_RUNTIME=1and refused in
production.openrouteras an LLM review provider. Keys are read from the environment only, never from a.envin the
scanned project.- Benchmark: 10 Go and 10 Java/Kotlin repositories, every finding labelled, held-out repositories chosen and
pinned before scanning.
Changed
- Findings in Go, Java/Kotlin, C#, Rust and PHP are preview until measured on enough held-out findings
(STABILITY.md); SQL injection, command injection and XSS still gate. Held-out so far: Java 10 of 12 true; Go
GO004 2 of 2 and AUTH010 2 of 2. - QA006 falls back to packages when a tsconfig maps the catch-all
"*"path (409 false positives on one Java
project's front end). - AUTH010 reads rate limiting in Go code; BE018 recognises Go validation tags and libraries;
_test.gofiles
count as tests. - Installation:
pip install vibeauditfrom PyPI. Releases are published there by trusted publishing from the
same attested wheel and sdist as the GitHub release.
vibeaudit 0.9.3
Full route paths behind router prefixes, UI checks for Vue and Svelte, and AUTH001 stable again.
Added
- Router prefixes are resolved for FastAPI, Flask blueprints and Express routers, so
vibeaudit routes, the route
checks and the frontend-backend contract checks see full paths. The contract checks now also run in projects
with prefixes, once every route's full path is known. - Vue and Svelte components get the same UI checks as React: keyboard access, labels, dead controls, index keys,
unconfirmed destructive actions, cleanup of subscriptions and loading/error states.
Changed
- AUTH001 (State-changing endpoint without authentication) is stable again: a route or server action that calls a
dev-only guard that throws (assertDevelopment(),devOnly(), aNODE_ENVcheck that throws) is not reported.
Held-out precision is 14 of 14 (it was 14 of 16 when it went back to preview in 0.9.2). - UI014 (Image without width and height) treats an
<img>with a CSS-module class as sized by its stylesheet. It
stays in preview: its held-out false positive has another cause.
vibeaudit 0.9.2
Cross-function taint for JavaScript and TypeScript, and two rules back in preview after a held-out measurement.
Added
- JavaScript and TypeScript taint is followed into functions of the same file and named imports from project files
(3 calls deep). A redirect to a URL an API client returned (such as a Stripe checkout session) is no longer
reported as an open redirect. - Python taint is followed into the functions a route, agent tool or model call passes data to (same file,
self
methods and imported project modules, up to 3 calls deep), and findings say which call it came through. (This
shipped in the 0.9.1 code; its note was missing from the 0.9.1 entry.)
Changed
- Two rules go back to preview after a fresh held-out repository lowered their held-out precision below 90%
(STABILITY.md):- UI014 (Image without width and height): 6 of 7 (85.7%); the false positive is an
<img>sized by CSS classes. - AUTH001 (State-changing endpoint without authentication): 14 of 16 (87.5%); two server actions are guarded by
assertDevelopment(). It keeps gating (gates_in_preview), like the other severe security rules: a missed
unauthenticated write costs more than a false alarm.
- UI014 (Image without width and height): 6 of 7 (85.7%); the false positive is an
vibeaudit 0.9.1
Credibility: measured recall and F1 next to precision, an honest list of misses, a comparison with Semgrep and
Bandit, and the means for others to check the labels.
Added
- docs/BENCHMARK.md (
python tools/benchmark.py report): precision, recall and F1 per category and per framework.
Recall comes from 17 known vulnerable locations in the repositories pinned before a published advisory
(benchmark/recall.json) and from the seeded cases; every advisory location vibeaudit misses is listed with the
reason. Advisory recall is 35% (6 of 17). - A head-to-head with Semgrep (
--config p/default, no login) and Bandit on the same pinned corpus
(tools/compare.py, run by.github/workflows/compare.ymlon Linux, manual and weekly): advisory recall,
finding counts, overlap, and precision estimated from a labelled random sample of each tool's findings.
First run: all three tools find the same 6 of the 17 advisory locations; precision is 88% for vibeaudit (every
finding labelled), about 34% for Bandit and about 22% for Semgrep (50-finding samples, vibeaudit's rubric). - A sample report:
.github/workflows/pages.ymlpublishes the HTML report and the six compliance reports for the
deliberately vulnerable Next.js + Supabase fixture to GitHub Pages: https://ashin-06.github.io/vibeaudit/ - A second-labeller kit (
tools/labelling_kit.py, docs/LABELLING.md): a blind stratified sample of 150 labelled
findings, labelling instructions, and Cohen's kappa against the maintainers' labels. .github/workflows/publish.yml: trusted publishing of the attested wheel and sdist to PyPI, off until the
one-time setup in docs/PUBLISHING.md is done.
Changed
- The README's accuracy numbers are generated from the benchmark results by
tools/benchmark.py baseline, and a
test fails when they differ. They now say which numbers are stable rules only and which include preview rules
(the held-out precision of stable rules is 95.7%; the README said 94%).
vibeaudit 0.9.0
Phases 0 to 4 of the roadmap are complete: a measured benchmark, the new checks, vibeaudit fix, the
integrations, and compliance mapping with the threat feed. The release passed the completion gate: a full audit
(docs/AUDIT-0.9.0.md) with every high and medium finding fixed, packaging checked in a
fresh environment on Python 3.9 to 3.13 on Linux, macOS and Windows, and a green benchmark gate.
Security
- A project's
llmsetting is read only byvibeaudit scan: the LSP server, watch, studio, the MCP tools and the
Python API never send code to a model provider because a repository's config asks for it. vibeaudit studiobinds its port exclusively, so on Windows no other local process can bind the same port and
answer in its place.
Fixed
- Files and messages nested too deeply to parse (package.json, tsconfig, pyproject, config, feedback and baseline
files, LSP and MCP messages, bundles, the threat feed, cache files) are treated as malformed instead of crashing
the scan or hanging the LSP server. - A damaged or forged bundle, a report path that cannot be written and an oversized studio query now fail with a
clear error instead of a traceback. - The VS Code extension's README installs the language server from the repository at the matching version;
vibeaudit is not on PyPI.
Changed
- Five rules leave preview. Each has at least 5 held-out findings at 90% or more and has been released in
preview at least once (STABILITY.md), and now counts toward--fail-on, the score and the
grade:- CFG010 Third-party GitHub Action not pinned to a commit (held-out 24/24, tuned 65/65)
- CFG016 Unencrypted database or disk storage (held-out 11/11, tuned 11/11)
- DATA012 Foreign key without an index (held-out 9/9, tuned 70/70)
- QA011 Same function copied into several files (held-out 8/8, tuned 14/14)
- UI014 Image without width and height (held-out 6/6, tuned 16/17)
- DATA013 (Redis key without an expiry) meets the held-out bar (6/6) but is 6 of 8 over the whole corpus
(75%), so it stays in preview with that reason.
Added
- Every rule is mapped to the OWASP Top 10 2021, the OWASP Top 10 for LLM Applications 2025, EU AI Act
themes, SOC 2 and, where they apply, HIPAA and GDPR (data/compliance_map.json).
--report compliance:FRAMEWORKwrites a self-contained, printable HTML page with each control's status and
links to the evidence. This is an aid for reviewers, not a certification or legal advice. vibeaudit updatedownloads a signed threat feed: slopsquatted package names (DEP001, also offline), a model
sunset calendar with retirement dates and replacements (LLM012), and prompt-injection patterns for agent
instruction files (new preview rule LLM025). The same data ships with vibeaudit; a feed is used only when its
signature verifies and it has not expired. See docs/THREAT_FEED.md.
vibeaudit 0.8.8
Phase 1 is complete: the last data, operations and CI checks.
Added
- Four preview rules:
- DATA014: personal data (email, phone, name, address, date of birth) stored in vector store metadata.
- DATA015: a cache-aside read that recomputes on a miss with no lock or single-flight (cache stampede).
- DATA016: a queue or scheduled job that charges, emails or inserts with nothing that makes a retry safe
(no idempotency key, processed-job record, upsert or unique constraint). - CFG018: a
pull_request_targetworkflow that checks out the pull request's code while secrets are
available.
Fixed
- Files with thousands of unclosed brackets no longer slow the JavaScript analysis down quadratically:
bracket matching and statement ends now use one pairing pass per file (a hostile file took 30 to 90
seconds, now a few). Results on well-formed code are unchanged, which tests check against the old scan.
vibeaudit 0.8.7
Phase 1 adds the remaining access-control checks.
Added
- Four preview rules:
- AUTH015: a password change or reset that keeps the other sessions and refresh tokens valid. Projects
that revoke sessions or tokens somewhere, and frameworks that do it themselves (Django, Firebase,
Clerk), are skipped. - AUTH016: a signed-in route that queries a table with a tenant column (org_id, team_id, workspace_id)
without filtering by tenant. Supabase tables under row level security are skipped. - AUTH017: an admin, feature-flag or maintenance endpoint that any signed-in user can call (no role check).
Endpoints with no authentication at all remain AUTH001. - FE008: a server action that returns a server secret, or an inline server action that closes over one.
- AUTH015: a password change or reset that keeps the other sessions and refresh tokens valid. Projects
vibeaudit 0.8.6
Phase 1 adds the remaining LLM checks; Phases 2 and 3 are complete with vibeaudit studio and the Redis TTL codemod.
Added
- Six preview rules for LLM features:
- LLM019: model output passed as an argument of a program run without a shell (
git checkout <answer>),
where a value such as--upload-pack=...becomes an option. Arguments after--are not reported. - LLM020: an agent loop that calls the model for as long as it asks for tools, with no step limit.
- LLM021: user text placed in a prompt without delimiters, so instructions inside it read like yours.
- LLM022: web pages or files loaded into a vector store with no cleaning step (chunk poisoning).
- LLM023: model calls for signed-in users with nothing tracking tokens or cost per user (apps where users
bring their own API key are skipped). - LLM024: a project with tests, none of which touches its LLM code, and no eval setup.
- LLM019: model output passed as an argument of a program run without a shell (
- Tests that pin down two Phase 1 items existing rules already cover: model output used as SQL, a shell
command or a file path (LLM002), and "fetch this URL" features and redirects fed by request data (BE005,
BE008). vibeaudit studio: a local web UI (127.0.0.1 only, token-protected) with routes, data flows and a score
drill-down.vibeaudit fixadds an expiry to Redis keys (DATA013):ex=3600/EXonset, or an
expire(key, 3600)after hash, list, set and sorted-set writes.
Changed
- The fuzz test's time budget uses the median of three calibration runs, so slow shared CI runners no longer
fail it; it still catches super-linear paths.