Maintainers don’t need more AI bug reports. They need fixes. #213390
Unanswered
queenofcorgis
asked this question in
Discover: GitHub Best Practices
Replies: 0 comments
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
Originally published on the GitHub blog.
AI has made finding potential vulnerabilities fast and cheap. Fixing them still takes human judgment, and in open source, that work falls to maintainers. Many are already flooded with AI-generated security findings, including duplicates and reports that aren’t vulnerabilities at all. Another raw scan doesn’t help them. Anyone pointing a new model at a project should ask its maintainers first and come with fixes, not just findings.
Maintainers are under intense pressure from this flood of reports. As Christian Grobmeier, a maintainer of Log4j, has put it, the AI defending open source has to be better than the AI attacking it.
AI shouldn’t add to maintainers’ workload. So, with maintainers, GitHub and Microsoft ran a pilot together: could AI help maintainers ship fixes instead of handing them more findings?
GitHub and Microsoft are using Codename MDASH, Microsoft’s multi-model agentic scanning harness, to find and fix security vulnerabilities in widely used open source software, strengthening the foundation that much of the world’s code is built on.
The process
Our pilot had two goals: send only findings that matter, and never send a maintainer an unverified finding. Here’s how it worked:
Reports and fixes stayed on GitHub, where maintainers already work.
The result: Codename MDASH, using Microsoft’s MAI Cyber model, flagged thousands of possible issues across the six projects. After AI triage and hand review by seven Microsoft FORGE Lab researchers, maintainers received 99 private reports. As of November 15, they had fixed 55, and 14 of those fixes were already in a release.
Image: Logos for AutoGPT, Caddy, Express, FastMCP, ImageMagick, and Netty.
Participating projects
Six projects took part, all alumni of the GitHub Secure Open Source Fund. Their maintainers agreed to the scan, received every report privately, decided what to fix, and shared what worked:
Infographic: “Codename MDASH Pilot: The Result.” It shows 9,791 possible issues flagged across six open-source projects, leading to 99 private reports sent to maintainers after AI triage and human review—about one report for every 100 raw findings.
What we learned
These lessons come from what the six projects fixed and what their maintainers chose to share. Six projects is a small sample, so read them as early lessons.
1. Fixes matter more than findings
Fixing is where maintainers spend their time. So every report arrived already checked, with a suggested fix to review, adapt, and merge. Maintainers started from a well-scoped patch, not a raw finding.
The first fixes shipped about a week after the initial scan. Caddy’s maintainers fixed all 22 of the reports they received in a week or less. Maintainers used the suggested fixes, some as written and some with changes.
AutoGPT’s maintainers had recently shipped a new check, and it was broken. The report landed three days later and pointed at the right line. They took the fix further than the suggested patch, wrote it within a day, and had it in production less than a week later.
2. Codename MDASH, with researcher review, gave maintainers high-quality findings with fixes
Maintainers got a short list, not a flood. Express maintainers said the reports cut through the noise. Caddy maintainers said they surfaced obscure bugs and filled gaps that earlier reports had missed. For FastMCP, the scan went beyond what AI scanners had given them before. ImageMagick, maintained by a two-person team, received 22 reports.
AI did the finding, researchers did the checking, and maintainers stayed in charge.
One ImageMagick report looked like a kind of issue the maintainers had seen before. Its summary showed it wasn’t and pointed to the exact lines of code. They were able to push a fix in 15 minutes.
3. Open source still has gaps to close
The pilot worked because people did careful work at every step, and that work doesn’t scale yet. Seven researchers spent about two days per project checking findings by hand. Meanwhile, low-quality reports still reach maintainers every day.
For open source, fixes need to be delivered via private vulnerability reporting (PVR). Maintainers need to run CI/CD tests on private forks to verify those fixes. And to close the gap between defenders and attackers, maintainers need access to cyber-capable models to verify a proof of concept, before and after the fix.
Why it matters
If you’re a maintainer, this approach changes what a security report costs you. A checked, private report that comes with a suggested fix takes less of your time. You shouldn’t have to triage a scanner’s output. That’s the sender’s job.
If you use AI to find vulnerabilities in open source, maintainers’ feedback points to this standard:
What you can do today
[gh-secure](https://gh.io/gh-secure).What happens next
Everyone downstream benefits from this system. When a maintainer publishes a security advisory on GitHub and it’s reviewed for the GitHub Advisory Database, Dependabot can alert the repositories that depend on that package. That’s the idea behind the GitHub Secure Open Source Fund: 216 projects so far, with more than 300 maintainers fixing more than 4,000 vulnerabilities in software downloaded more than 20 billion times a month.
Thank you
To the maintainers of AutoGPT, Caddy, Express, FastMCP, ImageMagick, and Netty: thank you for your time, your trust, and your feedback. Thank you also to the Codename MDASH team at Microsoft Security, the MAI Cyber team, the researchers of Microsoft FORGE Lab, and GitHub Security Lab.
Maintainers do the fixing. Our job is to make sure they never do it alone.
About the GitHub Secure Open Source Fund
All six projects in this pilot are alumni of the GitHub Secure Open Source Fund, which gives each project $10,000 through GitHub Sponsors, plus security training and a supportive alumni community.
The GitHub Secure Open Source Fund, powered by GitHub Sponsors, funds open source projects that underpin the digital supply chain, catalyze innovation, and are critical to the AI stack. Want your company to help secure the open source it depends on? Become a funding partner through the Fund page. Learn more about the Fund or apply.
🧠 What did you think ? Let us know your thoughts and how you're keeping your project secure!
All reactions