LF Health Score v2 - adding lifecycle & impact to Insights

[jonathimer]

We're rethinking how LFX Insights evaluates open source projects, and we'd love community input before we ship it. This post lays out the full methodology so you can give informed feedback on the details, not just the direction.

What's changing

Today's single 0–100 health score has structural problems:

• Equal weighting ignores blast radius. A widely-depended-on library and a niche utility shouldn't be scored the same way.
• No lifecycle state. The current score can't tell you a project is "abandoned" vs "stable and done" — it just hands you a low number.
• It penalizes "done." Mature libraries with no open issues, no CVEs, and no need for active commits score poorly for the wrong reasons.
• Popularity is conflated with health. Being downloaded a lot doesn't mean a project is healthy.

Failure modes today

Scenario	v1 score	Reality
lodash — massively popular, single maintainer, no releases in 5 years	~65 (Healthy)	Should flag maintainer risk + staleness
left-pad — tiny utility, "done", no bugs, no releases needed	~25 (Critical)	Penalized for being stable and small
XZ Utils — low stars, few contributors, critical infrastructure	~30 (Unsteady)	Should score high on dependency risk, low on maintainer health
Abandoned project — no commits in 3 years, open CVEs	~20 (Critical)	Correctly low, but doesn't say why or say "abandoned"

Design principles

1. Lifecycle state first, score second. A numeric score without lifecycle context is misleading. "Abandoned with 60 points" is worse than "Active with 45 points."
2. Separate "how healthy is it?" from "how much does it matter?" Health and impact are different axes. Don't bake popularity into the health score — surface it alongside.
3. Don't penalize "done." Lack of commits is only a problem when combined with other signals (open CVEs, unanswered issues, maintainer absence).
4. Maintainer health is the leading indicator. Everything else is trailing. Responsive maintainers will eventually fix CVEs, merge PRs, ship releases. Absent ones won't.

What we're proposing

Replace the single score with three independent assessments, shown together:

Assessment	Question it answers
Lifecycle State	What state is the project in? (Active / Stable / Declining / Abandoned / Archived)
Health Score (0–100)	How healthy is it?
Impact Score (0–100)	How much does it matter if it breaks?

---

1. Lifecycle State

The lifecycle state is the headline classification. It's derived from signals — not set manually, and not computed from the composite score.

State	Primary signal	Confirming signals
Archived	Repository archived flag = true, or registry status = deprecated/yanked	Authoritative — no confirmation needed
Abandoned	No maintainer activity (commits, issue responses, releases) in 18+ months	Open issues unanswered 6+ months, open CVEs unaddressed, no release in 24+ months
Declining	Maintainer activity dropped >50% (trailing 6mo vs prior 6mo)	Issue response time increasing, release cadence slowing, contributor count shrinking
Stable	Low activity but maintainer responded to issues/PRs within 90 days	No open critical CVEs, last release <24 months (or no changes needed), few or no open issues
Active	Regular commits in trailing 6 months + maintainer responsive	Active release cadence, issue triage happening, healthy PR merge rate

The "Stable" state is the key innovation. Today we can't distinguish a healthy mature library (no commits because it's done) from an abandoned one (no commits because nobody's home). The difference is maintainer responsiveness: if someone filed a critical CVE tomorrow, would the maintainer respond?

---

2. Health Score (0–100)

Health Score = Maintainer Health (0–40)
             + Security & Supply Chain (0–35)
             + Development Activity (0–25)

Rating bands

Score	Label	Meaning
85–100	Excellent	Well-maintained, secure, active or stable by design
70–84	Healthy	Good shape, minor gaps
50–69	Fair	Some concerning signals, may need attention
30–49	Concerning	Multiple risk signals
0–29	Critical	Serious gaps, likely needs intervention

Maintainer Health (0–40 pts)

The single most important category. Measures whether someone is home and whether they'd respond to a crisis.

Signal	Max pts	How measured
Maintainer responsiveness	15	Median time for a maintainer to respond to issues/PRs (any response, not merge). 15 = <7 days, 10 = <30 days, 5 = <90 days, 0 = >90 days.
Bus factor	10	Active maintainers (committed in last 12 months with merge/release permissions). 10 = 3+, 7 = 2, 3 = 1, 0 = 0 active.
Maintainer continuity	8	Has the maintainer set been stable? 8 = stable, 4 = some turnover, 0 = complete turnover or no data.
Org diversity	7	Are maintainers from multiple organizations? 7 = 3+ orgs, 4 = 2 orgs, 2 = 1 org, 0 = unknown.

Org diversity uses LFX's contributor-to-organization affiliation mapping — more precise than GitHub org membership, which is a weak proxy.

Security & Supply Chain (0–35 pts)

Signal	Max pts	How measured
Open vulnerabilities	10	Unresolved advisories from OSV/GHSA. 10 = none, 5 = medium/low only, 0 = any critical/high open. −5 modifier for critical CVE open >90 days.
Security practices	8	SECURITY.md (2), vulnerability disclosure process documented (2), branch protection enabled (2), signed releases (2).
OpenSSF Scorecard	7	Normalized from Scorecard's 0–10 to 0–7. Already computed by deps.dev for covered packages.
Dependency health	5	% of direct dependencies with known vulnerabilities or abandoned status. 5 = all clean, 3 = <10% problematic, 0 = >25% problematic.
Supply chain integrity	5	Build provenance attestation (2), repo-to-artifact consistency (2), 2FA on publish accounts (1).

Development Activity (0–25 pts)

This category adjusts based on lifecycle state. A "Stable" project gets a floor of 15/25 if it has no open issues and no unresolved CVEs — it's healthy precisely because it doesn't need active development.

Signal	Max pts	How measured
Release cadence	8	8 = release in last 6 months, 5 = last 12 months, 2 = last 24 months, 0 = >24 months. Stable projects: 8 if last release <24mo and no pending changes needed.
Issue resolution	7	Ratio of issues closed/triaged vs opened in trailing 6 months. 7 = closing faster than opening, 4 = roughly even, 0 = backlog growing >2x.
Commit activity	5	Commits in trailing 6 months. 5 = active, 3 = some, 0 = none. Stable projects: floor of 3 if no changes needed.
PR merge health	5	External PRs merged vs ignored in trailing 12 months. 5 = >50% merged or responded to, 2 = <50%, 0 = ignored.

---

3. Impact Score (0–100)

The Impact Score answers "how bad is it if this breaks?" — designed to surface load-bearing upstream packages, including those with few direct dependents but large transitive reach (the left-pad / XZ class of risk).

Methodology

1. Log-transform heavy-tailed inputs: log(downloads + 1), log(direct_dependents + 1), log(transitive_dependents + 1) to compress power-law distributions.
2. Normalize 0–1 within each ecosystem (percentile-rank or min-max) so packages are comparable across ecosystems despite different raw magnitudes.
3. Weighted blend of signals, scaled to 0–100.

Signals & weights

Signal	Weight	What it captures
Transitive dependents	Primary	Packages that depend on the target directly or indirectly. The core blast-radius signal.
Graph centrality	Primary	PageRank-style score over the dependency graph. A package depended on by Kubernetes/React outranks one depended on by 50 abandoned projects. Runs as a periodic batch job.
Downloads	Secondary	Per-registry download counts. Proxy for real-world usage but noisy (CI inflation, bots).
Direct dependents	Secondary	First-degree dependent count. Under-ranks load-bearing upstream packages so weighted below transitive + centrality.

Cross-ecosystem normalization

• Per-ecosystem quotas so npm's volume doesn't crowd out smaller ecosystems (crates, Go modules).
• Rank and cut at tier budget rather than defending hard thresholds.

Manual override: Spotlight list

Known-critical primitives (compression, crypto, TLS, common parsers) are included regardless of computed score. A deliberate safety net for the methodology's structural blind spot: low-popularity-but-load-bearing packages no download/dependent signal will surface.

Known limitations

Even transitive + centrality will miss packages that are critical but rarely depended on in the observable graph — vendored code, build-time-only tools, dependencies pulled outside the registry. This is why maintainer-responsiveness and human curation remain essential.

---

Worked examples

lodash — Declining · Health 52 (Fair) · Impact 94

• 52M downloads/mo, 142k dependents, massive transitive reach. Single active maintainer, last release 2021. No SECURITY.md. 4 historical CVEs (resolved).
• Maintainer: 15/40 · Security: 22/35 · Development: 15/25 · Impact: 94

express — Active · Health 78 (Healthy) · Impact 88

• 31M downloads/mo, 72k dependents. Multiple maintainers across orgs. Regular releases. SECURITY.md present. Scorecard 7.2/10.
• Maintainer: 34/40 · Security: 27/35 · Development: 17/25 · Impact: 88

is-odd — Stable · Health 72 (Healthy) · Impact 23

• 3M downloads/mo. One-function utility. No open issues, no CVEs, no changes needed. Maintainer responded to a filed issue 4 months ago.
• Maintainer: 20/40 · Security: 30/35 · Development: 22/25 · Impact: 23

event-stream — Abandoned · Health 11 (Critical) · Impact 41

• Hijacked in 2018, now abandoned. Known supply chain compromise. npm deprecated. Last commit 2019.
• Maintainer: 0/40 · Security: 3/35 · Development: 8/25 · Impact: 41

---

What this changes for you

• Scores will shift. Projects high on popularity but low on maintainer responsiveness will drop. Stable, mature libraries will rise.
• Rating band language changes: Excellent / Healthy / Fair / Concerning / Critical (replacing Excellent / Healthy / Stable / Unsteady / Critical — "Stable" is now a lifecycle state, not a band).

Where we'd love your feedback

1. Lifecycle thresholds. Is "18 months of no maintainer activity = Abandoned" right?
2. Detecting "Stable." Maintainer responsiveness is our signal — but how do we evaluate it when no recent issues have been filed to test it?
3. Supply chain data. Provenance attestation and 2FA status aren't in our pipeline yet. Start the category at 0 and grow it, or omit until data is mature?
4. Ecosystem-specific weights. Should "active release cadence" mean something different in npm vs Maven vs crates?
5. Signals we're missing. What would you look at to judge whether a project is healthy?
6. Historical scoring. Worth backfilling so you can see trends ("declining for 8 months"), or only compute current state?
7. Transparency. We plan to publish the full methodology so projects understand exactly how they're scored and how to improve. Anything specific you'd want documented?

If you maintain a project, work with this data, or build tooling on top of it — we want to hear from you. Drop a comment below, or open a discussion thread on any single piece you have strong opinions about.

[mrbobbytables]

Overall I really like the way this is shaping up.

My one concern is regarding who is included as a "maintainer". For single repo, or smaller projects using the detected maintainers would work - but my concern is for medium and larger projects where often who is "officially" designated as a maintainer can be a much smaller group which generally maps to project leadership and doesn't necessarily capture who's a maintainer in the classic definition.

Project	# Repos	# Org Members	# Maintainers
Dapr	47	182	24
Kubernetes (note:multiple orgs)	415	1460	102
Open Feature	66	164	16
Open Telemetry	127	834	19

NOTE: I used CNCF projects because those are the ones I'm most familiar with and had immediate access to the data; however this pattern does extend beyond CNCF projects.

From these projects - between 2-13% of the org members are officially designated as the "maintainers", but are not necessarily representative of what would classically be thought of as maintainers.

In these instances it may be better to use org members as a basis for the health score; or alternatively fall-back to a generated list (similar to what is done now).

[funnelfiasco]

Overall, I like the general direction. I think it's right to separate popularity from health, especially because a package may be relatively unimportant to the overall ecosystem, but critically important to my specific project.

A few specific notes:

Lifecycle thresholds. Is "18 months of no maintainer activity = Abandoned" right?

Given how you've defined "maintainer activity", I'd shorten this window, perhaps as low as 3 months, but probably 6. I'd also accept 12. This is all hunch-driven, of course. If you can generate a list of projects that would qualify as abandoned under different thresholds, we could look to see where the false positive rate becomes unacceptable. Also, maybe something like "inert" or the like would be a better term; it could avoid hurt feelings when there's a false positive while still signalling that the project may not respond to issues.

Detecting "Stable." Maintainer responsiveness is our signal — but how do we evaluate it when no recent issues have been filed to test it?

If a tree falls in a forest... 😄 I think if there are no recent issues/commits/releases/PRs/discussions (are GitHub discussions included in the measure, when available? If not, I think they should be), then it's reasonable to call the project "inert" until there is activity. Most "stable" projects will probably have at least some level of activity.

Supply chain data. Provenance attestation and 2FA status aren't in our pipeline yet. Start the category at 0 and grow it, or omit until data is mature?

I'd omit it until it can be measured. Otherwise, the score "punishes" projects that may actually be providing it.

Signals we're missing. What would you look at to judge whether a project is healthy?

This might be in "I wish I had a pony" territory, but looking for user activity (i.e. are people helping each other in the project's official channels, etc) could be a good signal for activity. This gets complicated quickly, since projects may have several official or semi official channels, not all of which are public.

Historical scoring. Worth backfilling so you can see trends ("declining for 8 months"), or only compute current state?

I think it's worth backfilling. A project's score trend is arguably a better signal of future health than a point-in-time snapshot.

Transparency. We plan to publish the full methodology so projects understand exactly how they're scored and how to improve. Anything specific you'd want documented?

I'd want to see the distribution of projects along various components. e.g. What's the distribution of project lifecycle activity, what's the distribution of the different supply chain scores, etc. Bonus points if there's a CSV file or Jupyter notebook that folks can use for additional analysis to make v2.1 suggestions 😄

[evankanderson]

Sorry for sitting on the late reply for a bit. I've been down this road more than once, and the nuance you're covering here makes me cautiously optimistic.

Lifecycle thresholds. Is "18 months of no maintainer activity = Abandoned" right?
Detecting "Stable." Maintainer responsiveness is our signal — but how do we evaluate it when no recent issues have been filed to test it?

I @funnelfiasco is on the right track with replacing "stable / abandoned" with "inert" for repos with no interaction. One example I think of in this case is https://github.com/jtacoma/uritemplates -- a zero-dependency solution for a well-specified RFC, with high test coverage. It's also 10-15 years old, with one question asked (and answered) in the last 6 years. Is this "abandoned"? "stable"? I suspect there are similar examples in the NPM ecosystem, though the addition of a package registry there makes things more interesting in comparison with Go's module system.

I think an "abandoned" declaration can be conclusively made if there are open unresponded issues (or PRs) older than a certain age. I'm willing to say a 90 day threshold there is sufficiently generous -- if you haven't done anything with the project in 90 days and the last activity on the project is someone asking a question, then probably no one is looking. I'm not sure how to treat "I don't respond to issues, but I do commit to the repo / merge PRs" -- given the current state of the world, I'd probably call that "active" with a low health score.

Supply chain data. Provenance attestation and 2FA status aren't in our pipeline yet. Start the category at 0 and grow it, or omit until data is mature?

I think there's at least some value in measuring how well projects are tracking and adopting supply chain improvements (for example, are they using some variation of trusted publishing for NPM, or are releases being pushed by individual maintainers / long-lived keys).

Unfortunately for your points system, some ecosystems have made more security progress (or started in a better place) than others, so you may simply have to say "this category is about X points", and then dish out the points differently for e.g. go libraries vs npm libraries.

Stuff like https://github.com/std-uritemplate/std-uritemplate may end up making you mad as well. ¯_(ツ)_/¯

Ecosystem-specific weights. Should "active release cadence" mean something different in npm vs Maven vs crates?

Also, how do you measure release cadence? Let me introduce you to https://github.com/babel/babel, which publishes approximately 1000 npm packages from a single repository. My recollection is that they only publish specific packages when there's a change in that package, but all the packages follow the same versioning scheme.

The short answer here is that you'll need to do the standard data science thing and propose a measure, then actually implement the measurement, check the distribution to see if it makes sense / matches what you want, manually investigate outliers (hello Babel.js!), and then either accept the measurement or tweak it and repeat the process. I promise you that the world is weirder than you'd expect.

Signals we're missing. What would you look at to judge whether a project is healthy?

I think you probably want to do some data science on the current measures. Right now, they look like they were constructed in a vaccuum, and it's probably harder to get "excellent" than you intend.

On the plus side, I think you're on a good track in the impact measurement, though I'm not sure that adding the log-value measurements of downloads + transitive dependencies + direct dependencies before normalizing by ecosystem is going to be the right mix. (In particular, I suspect that one or more of the components will end up dominating the other two before the normalization has occurred.) You might also want to look at using max() or min() functions for some of the aggregation, rather than direct sums -- for example, you're spending 25 "points" (excellent --> bottom of healthy) on three different measures of maintainer endurance -- bus factor, org diversity (which seems to duplicate bus factor) and turnover. You may also want to look at maintainer additions and removals separately -- a project with 6 maintainers which has added 3 in the last year would be lose 4 points, whereas a project with just 2 maintainers who have stayed the same for the whole year would have the full 8 points. If both projects have maintainers from 2 orgs, we'd end up with 10+4+4 = 18/25 points for the 6-maintainer project, and 7+8+4 = 19/25 points for the 2-maintainer project.

Looking at distributions of points and correlations or anti-correlations across your measures will help you with your specific scoring mechanics (pick a sample of e.g. 5k projects with 1k "important" to do more in-depth measurements on and test different ways of combining measurements).

Historical scoring. Worth backfilling so you can see trends ("declining for 8 months"), or only compute current state?

I'm not sure that I'd prioritize backfilling, but I'd want to keep the following information once you start:

1. At-least-monthly snapshots of the measured data, for as long as possible. You might be able to switch to yearly after a few years, but I'd probably try to keep the individual measurements alongside the computed score, so that you can test hypotheses like "decline in measure X precedes decline in measure Y" as you get more data.
2. Record which version of the measurement / algorithm you used for the measurements. I expect you'll learn and (for example) adjust the scaling curves for ecosystems over time. Knowing that the 1-year-ago impact assessment was on a different scale than the current impact assessment will help avoid jumping to conclusions over methodology errors.

Transparency. We plan to publish the full methodology so projects understand exactly how they're scored and how to improve. Anything specific you'd want documented?

I think it makes sense to publish the methodology. If I were in charge, I'd want to also make the datasets available to (at least) academic researchers in the same way that e.g. scorecard scores have been used. (random example research using the OpenSSF maintained score: https://arxiv.org/html/2601.18344v1)

[jberkus]

The short answer here is that you'll need to do the standard data science thing and propose a measure, then actually implement the measurement, check the distribution to see if it makes sense / matches what you want, manually investigate outliers (hello Babel.js!), and then either accept the measurement or tweak it and repeat the process. I promise you that the world is weirder than you'd expect.

Very well said.

[MoralCode]

Replace the single score with three independent assessments, shown together:

I generally like the idea of splitting the single score into three, even if only because it makes things a little less suceptible to goodharts law (like what happened to openssf scorecard "Some projects have been observed making superficial changes (adding a SECURITY.md with minimal content, enabling branch protection without enforcement) to improve scores without improving actual security." https://safeguard.sh/resources/blog/openssf-scorecard-v5-release)

In skimming that blog post - it sounds like there's a decent amount of overlap between its other advice and some of the goals of this proposal too.

The "Stable" state is the key innovation. Today we can't distinguish a healthy mature library (no commits because it's done) from an abandoned one (no commits because nobody's home). The difference is maintainer responsiveness: if someone filed a critical CVE tomorrow, would the maintainer respond?

Given the above, i worry a bit about making maintainer responsiveness a single determining factor in classifying a project as stable. Having read the book "Working in Public", i also am curious if there may even be different interpretations of the lifecycle states depending on the type of project (to use the authors terms: Toy, Stadium, Club, Federation). The Mozilla open source archetypes could be another source of inspiration on this.

Lifecycle thresholds. Is "18 months of no maintainer activity = Abandoned" right?

I suspect the threshold may be different across projects/ecosystems or even by the purpose for which these metrics are being used as well. Im not sure which categories you could make to allow

Detecting "Stable."
I suspect this conversation has happened several times already in the CHAOSS Data Science working group - maybe there are folks there (or in other communities that work on open source metrics) that have already been down this path before.

If you maintain a project, work with this data, or build tooling on top of it — we want to hear from you. Drop a comment below, or open a discussion thread on any single piece you have strong opinions about.

Oh yeah I should probably introduce myself haha, I'm currently the main active maintainer of CollectOSS and I rely a lot on the metrics work that has been happening within CHAOSS and have recently been contributing to the development of some newer metrics surrounding AI (which makes me wonder whether these new project health scores may also need to be able to account for the types of ways that particular metrics have shifted in their meaning and usefulness with the rise of generative AI).

Happy to collaborate and introduce you to the CHAOSS community if that is helpful as well!

[MoralCode]

A CHAOSS community member seems to have already been down this path before too (the part of the readme about "Dead" repos being a deliberately high bar made me think of this discussion)

https://github.com/andrew/weekend-at-bernies

[jberkus]

I like that you're headed for more sophistication around health metrics. Understand that project health is an unsolved problem which nobody yet has come up with project-agnostic, reliable metrics before, partly because it's a multifactor system and partly because a lot of the data is very hard to collect. But what you have above is definitely an iterative improvement on the prior LFX metrics.

That said, I think it's important, including through the UI, to emphasize to LF members and others that project health is still an area of experimentation across our ecosystem, and that they need to compare the rating given there with what they actually know about the project.

There is one foundational problem with this reformulation, though:

"Maintainer health is the leading indicator": For larger projects like Kubernetes, Istio, the Linux Kernel, etc, there is a large pool of senior contributors who do most of the actual work. Discarding data from these contributors will make all large projects look deceptively understaffed. Contrawise, if a project is growing (in terms of activity and codebase) but the contributor pool (aside from maintainers) is not growing, or even shrinking, then the project is actually in trouble regardless of how active the maintainers are. Also, the pool of regular contributors is your recruitment pool for new maintainers, so the overall health of the non-maintainer contributor pool is a critical factor for evaluating project health.

Overall, making project health all about maintainers will result in mis-evaluations as bad as the current system.

Some additional nits:

1. "Maintainer responsiveness": a) maintainers-only, again, and b) this needs to take into account the volume of PRs. Projects may, for example get blasted with many Agent-authored PRs; that's going to generate an increase in time-to-respond, but doesn't indicate that the project is in failing health. Probably this one should be taken out of the maintainer health category, because, again, it's not maintainers job to be the first responders.
2. "Maintainer continuity": setting 0 for "no data" is a problem here; projects shouldn't be penalized because the tool can't figure out their maintainer history. Also, "some maintainer turnover" is a good thing and should get projects an 8; a project with zero maintainer turnover should be penalized, because it's headed for maintainer burnout.
3. "Bus factor": This should measure review and merge/close activity, not commit activity. The primary job of a maintainer is to review and merge (or reject) PRs, not author new ones. Most do, but it's the review/merge activity that is critical for project health.
4. "Open vulnerabilities": this is good, but we'll need to try it and validate data here. A lot of "open vulns" ratings in the past have been invalidated by the inclusion of fake vulnerabilities and/or vulns whose criticality rating was inflated (there's some bad incentives in the security researcher ecosystem).
5. "Release cadence": this includes patch releases, correct? If so, we should probably shorten the period, given today's pace of vulnerabilities. Make it 8 for < 3 months, 5 for 6mo, 2 for 1 year, and 1 for 2 years. Even for very stable projects, we're all getting at least 2-3 reports a month. Problem: there is a vast difference here between large projects with a lot of surface vs. tiny libraries. You may need to evaluate this based on project size rather than "status".
6. "Issue resolution": because of AI agents opening slop issues, I don't think it's fair to use this currently as a stat. Once we work out a good way to deal with auto-closing slop issues, it can come back to the board.
7. "PR merge health": it would be a better indicator to base this on 80percentile time-to-close rather than volume.
8. "Graph centrality": doesn't this require scanning all of Github? It sounds like it does. If you have the resources to do so, I would love to have access to your data.
9. "Downloads": this is meaningless in modern ecosystems, and you're better off not using it. For example, most CNCF projects are pulled as container images from any of 5 different registries, not downloaded as tarballs.

Again, thanks for doing this! It shows some real serious thinking in how to measure project health, and is a great iterative improvement. As a member of WG-Data-Science in CHAOSS foundation, we'd love to have you show up for a meeting to discuss this model.

[jberkus]

Some additional metrics and adjustments:

• some kind of ratio of (codebase size + volume of changes) :: active contributors. Large projects should have lots of contributors, or they're in trouble; tiny projects can be stable with only a handful.
• comparison between the activity levels of contributors and their official status. A project where "active contributors" has a poor overlap with "org members", for example, has a governance problem.
• differentiating bot activity from human contributor activity, but not discarding the bot activity. Automation, and AI codegen, are important for modern projects but can't be measured the same as humans.
• levels of forum activity, which is an indicator of involved user activity. The hard thing here is that each project has its own forums on different platforms.
• large projects will have many repositories with different levels of support. For example, Backstage has dozens of plugins, some of which are flagged as "experimental" and as a result are not expected to be secure, up to date, or well-staffed. Projects shouldn't be penalized for those, or you're discouraging them from providing places for contributors to experiment. This means that each project needs to have a set of repos which is "production" vs. others which aren't measured, or are measured against different benchmarks. This set of repos will change over time so there needs to be an update mechanism.

One thing to think about in general: is it even possible to come up with a metric battery that works equally well for large projects as small ones? It may be that it's not. You do need to include some large projects in your test pool.

[jonathimer]

Thanks everyone — this is exactly the kind of feedback we hoped for when we posted the full methodology instead of just the direction. We’ve gone through every comment, and here’s what we’re changing as a result.

Changes we’re making to the methodology

1. Adding an “inert” resolution to the lifecycle tree (@funnelfiasco, @evankanderson, @MoralCode)

You convinced us that silence alone isn’t abandonment (ignoring people is). A finished, zero-dependency library with no open issues (the uritemplates (https://github.com/jtacoma/uritemplates) example) shouldn’t get the same label as a project with unanswered issues piling up. We’re splitting the rule:

 - Open, unanswered issues/PRs beyond a threshold (~90–180 days, exact value set empirically — see validation below) + no maintainer activity → Abandoned
 - No commits in 18+ months but also no interaction at all to judge responsiveness by → not abandoned; it gets a distinct “quiet” state rather than an accusation

2. Bus factor will measure who actually reviews and merges, not just who’s on the roster (@mrbobbytables, @jberkus)
   The Kubernetes numbers (102 official maintainers vs 1,460 org members) made this concrete. Where the officially-designated maintainer list undercounts reality, we’ll fall back to the set of people observably performing review/merge actions in the trailing 12 months. This also addresses @jberkus’s point that the maintainer’s primary job is reviewing and merging, not authoring commits.

3. Filtering bot and agent activity from responsiveness and issue metrics (@jberkus)
   Agent-authored issue/PR floods shouldn’t inflate a project’s time-to-first-response or tank its issue-resolution ratio. Our platform already identifies bots via contributor enrichment, so we’ll ensure to keep excluding bot-authored issues, PRs, and comments from the windowed metrics (we did so in the past already). We’re deliberately not discarding bot activity from the platform overall — as you noted, automation is part of modern project health — just from the signals it distorts.

4. Versioning the methodology and snapshotting raw inputs from day one (@evankanderson)
   Every emitted score will carry a methodology version, and we’ll keep at-least-monthly snapshots of the underlying measured inputs (not just the computed scores). We won’t backfill history, but we’ll start recording immediately so trend analysis and “did the score change or did the methodology change?” questions are answerable later.

5. A mechanism for experimental repos (@jberkus)
   Multi-repo projects shouldn’t be penalized for providing sandboxes (Backstage’s experimental plugins being the perfect example). We already support excluding repos from a project’s scoring; we’ll document this as an explicit mechanism and give projects a path to flag experimental/non-production repos. We’re also evaluating weighting the cross-repo aggregation by activity share so a dormant experimental repo can’t drag down a flagship.

Changes to the validation phase (blocking rollout)

Several of you said, in @evankanderson’s words: propose a measure, implement it, check the distribution, investigate outliers, then accept or tweak — “the world is weirder than you’d expect.” Agreed. Before the new scores replace the current ones, we’re running a data-analysis phase that now includes:

• Abandoned-threshold sweep (@funnelfiasco): generate the project lists at 6/12/18-month thresholds and manually review false-positive rates, rather than defending a number chosen by hunch
• Band attainability: check the actual score distributions, including whether “Excellent” is realistically reachable (@evankanderson’s suspicion that it’s harder than intended)
• Large projects in the test pool (@jberkus): Kubernetes-scale, multi-repo, multi-org projects explicitly included in manual review — your “does one metric battery work for both large and small projects?” question gets an empirical answer, not a design argument
• Signal-redundancy analysis: correlations across the maintainer sub-signals (bus factor vs org diversity overlap)
• Outlier investigation: monorepos publishing many packages (hello, Babel), vendored dependencies, and similar weirdness
• CVE data sanity checks (@jberkus): validating open-vulnerability counts against the known problem of inflated/fake advisories
• Publishing the analysis (@funnelfiasco, @evankanderson): we’ll publish the distributions along with the methodology, including data in an analyzable format, so the community can propose refinements against evidence. We’re also looking at what researcher access to the underlying datasets could look like.

Deferred, but on the roadmap

• Contributor-pool health (@jberkus): that maintainer-only signals miss the senior-contributor pool that does most of the work on large projects, and that the contributor pool is the recruitment pipeline for future maintainers — is fair. We’re not adding it to this iteration because we want to ship, validate, and keep parity with our package-level scoring, but contributor-pool trend signals (growth/shrinkage of active contributors, contributor-to-maintainer conversion) are on the follow-up list, and the validation phase will tell us how badly the current model mis-scores large projects in practice.
• Forum/discussion activity (@funnelfiasco): GitHub Discussions as a “someone is home” signal is legitimate; we’re scoping whether it lands in this iteration or the next.
• Percentile-based PR merge health (@jberkus): moving from merge ratio to time-to-close percentiles is queued for evaluation.
• Supply-chain adoption tracking (@evankanderson): measuring trusted-publishing adoption per ecosystem, with ecosystem-aware point allocation, is where we want to take the supply-chain category once the data exists.

Things already addressed since the original post

A few concerns were resolved in revisions since this was posted, so worth stating explicitly:

• Maintainer continuity is gone. The “0 points for no data” problem and the “zero turnover is actually bad” problem (@jberkus) plus the redundancy with bus factor (@evankanderson) killed it. Maintainer Health is now responsiveness + bus factor + org diversity.
• Supply-chain integrity is omitted, not zeroed (@funnelfiasco). Sub-signals we can’t measure are excluded and their weight redistributes to what we can measure — projects aren’t punished for our pipeline gaps. The original “start the category at 0" framing was wrong and we’ve dropped it.
• Responsiveness counts any non-author response, not maintainer-only responses (@jberkus’s first nit) — a community member answering an issue counts as “someone is home.”
• Security practices check enforcement, not file presence (@MoralCode’s Goodhart concern): branch protection scoring includes required reviews, status checks, and force-push settings — a minimal SECURITY.md and unenforced branch protection won’t score the way a real setup does. Gaming pressure never fully goes away, but we’re scoring configurations, not checkbox files.
• UI framing (@jberkus): agreed that project health measurement is an unsolved problem ecosystem-wide. The UI will surface which signal categories actually fed each project’s score (and which were unavailable), and we’ll frame the score as something to weigh against your own knowledge of a project, not a verdict.

Keep the feedback coming!

[jberkus]

This looks good enough that I feel like I want to see you build it so that we can see what it looks like, before additional feedback.