feat: add project-level impact score breakdown rollup with percentile bands (IN-1212)

[gaspergrom]

Summary

Rolls up the 4 Impact sub-metrics (transitive dependents, graph centrality, downloads, direct dependents) from package level to project level, and computes a global percentile rank-band for each (e.g. "Top 1%"), for the new project overview page's Impact breakdown section (IN-1212).

Deployed to production and verified — including a performance fix found after the initial deploy (see below). GitHub PR left unmerged pending review, per this ticket's convention.

Join path fix: packageRepos, not naive URL matching

Investigation confirmed naive string/URL matching between a project's repos and ossPackages_enriched_ds (e.g. LIKE on purl/repositoryUrl) returns false-positive matches — mirrors, forks, and unrelated same-named repos. The correct join is the one health_score_v2_impact.pipe already uses for its per-repo impactScore = MAX(packages.impact):

repositories (project's repos, enabled=true, excluded=false)
  -> repos            (matched on url — NOT repositories.id, a different id-space than repos.id)
  -> packageRepos      (verified package<->repo provenance mapping, join key repos.id = packageRepos.repoId)
  -> ossPackages_enriched_ds  (join key packageRepos.packageId = ossPackages_enriched_ds.id)

Verified live against Kubernetes: 326 enabled/non-excluded repos, all resolved through repos, yielding 897 linked packages via packageRepos — this exact chain, mirrored from the existing precedent.

The full chain is LEFT JOINed (not INNER JOINed) from insightsProjects onward so every project appears in the output, including ones with zero linked packages — this was caught during validation (see below).

Aggregation per metric

Chosen per-metric rather than one rule for all 4, each verified against real data:

• Direct dependents (dependentCount): MAX — mirrors the impactScore = MAX(packages.impact) precedent; a project's most-depended-upon package is the signal.
• Transitive dependents (transitiveDependentCount): MAX, not SUM. Verified live against Kubernetes: dozens of its own packages carry the same ~677,129 transitive-dependent count (these counts are graph-wide per ecosystem tooling, not package-unique), so summing would have inflated a single real signal by ~300x into a meaningless number.
• Downloads (downloadsLast30d): SUM, not MAX. Verified live against Kubernetes: its linked packages (pypi kubernetes, npm @kubernetes/client-node, nuget KubernetesClient, etc.) are independently-tracked registry packages with no shared/duplicated download counts, so summing safely reflects total adoption footprint across ecosystems.
• Graph centrality (centralityScore): MAX — a PageRank-style per-package score, not additive.

Percentile methodology

Projects are ranked globally against all other projects with a non-NULL value for that metric — deliberately not partitioned per-ecosystem. Verified live that only 43% of projects with linked packages (3,397 of 7,906) are single-ecosystem; the rest span 2-8 ecosystems (e.g. opentelemetry spans all 8 tracked ecosystems), so there's no single well-defined "project's ecosystem" to partition a project-grain percentile by. A project-level rollup value (e.g. total downloads summed across a project's npm + PyPI + Go packages) is compared fairly against other projects' equally cross-ecosystem-summed values.

rank() OVER (ORDER BY metric DESC) divided by the count of ranked (non-NULL) projects gives each project's position, bucketed into Top 1% / Top 10% / Top 25% / Top 50% / Bottom 50%. Projects with a NULL source metric are excluded from the ranking population entirely (not ranked as 0), and surface NULL percentile + NULL band rather than a fabricated low percentile.

Validation results (read-only, via Tinybird MCP against production)

• tb check pipes/project_insights_impact_breakdown.pipe — clean.
• Full end-to-end query run against production for Kubernetes, React, Express, TensorFlow, OpenTelemetry, Django, Envoy, Terraform, Apache Kafka, Prometheus, gRPC, Rust — all sane values, no negatives, percentiles in range:
  • React: direct dependents Top 1% (174,281, rank 2/5662), downloads Top 1%.
  • Kubernetes: direct dependents Top 10% (31,058, 1.01%), transitive dependents Top 1%, downloads Top 25%.
  • TensorFlow: mixed — Bottom 50% direct dependents, Top 50% transitive dependents.
• NULL-handling verified: pygithub and devlight-navigationtabbar (confirmed zero linked packages) correctly return NULL across all 4 metrics + all percentile/band fields — not a fabricated 0 or last-place band. This required fixing an initial INNER JOIN version of the pipe that silently dropped such projects entirely; the shipped version uses LEFT JOIN throughout so every one of the 13,548 insightsProjects rows is present (5,662 with directDependents data, 7,886 NULL).
• Row-count sanity: base rollup returns exactly 13,548 rows, matching count(DISTINCT id) on insightsProjects directly.

Data-quality caveat for reviewers

centralityScore is confirmed 100% NULL/unpopulated in production today — 0 of ~4.2M npm packages (and 0 across every other tracked ecosystem: go, maven, nuget, pypi, packagist, cargo, rubygems) have a usable value. The rollup and percentile logic for this metric is implemented and will activate automatically once the centrality scoring worker backfills real values, but until then every project will show "no data" for the Graph centrality row in the UI. This is a pre-existing data gap, not introduced by this pipe.

downloadsLast30d and other source fields also have partial NULL coverage by ecosystem (e.g. only 34/895 of Kubernetes's linked packages have a download count) — this is expected and handled gracefully (SUM over non-NULL values), not a bug.

Performance issue found post-deploy, fixed

After the initial deploy, the live endpoint was measured at 3.3–3.5 seconds per single-project request (consistent across repeated calls, not a cold-cache artifact), scanning ~37M rows / ~2.2GB every time. Root cause: the original design computed the rollup and global percentile rank live, on every request — rank() OVER (ORDER BY metric DESC) and countIf(...) OVER () require the full joined population in scope before any WHERE slug = ... filter can apply, so ClickHouse couldn't push the filter down. Every single-project lookup paid the full-scan cost.

Fix: moved the rollup + ranking computation into a new scheduled TYPE COPY pipe (project_insights_impact_breakdown_copy.pipe, COPY_SCHEDULE 40 2 * * * — 10 minutes after ossPackages_enriched_ds's own 02:30 UTC refresh, the only real staleness dependency since insightsProjects/repositories/repos/packageRepos are all near-live Sequin CDC-replicated tables) that materializes into a new datasource, project_insights_impact_breakdown_ds. The public endpoint pipe (project_insights_impact_breakdown.pipe) is now a simple filtered SELECT against that pre-ranked table — same pattern project_insights.pipe already uses against project_insights_copy_ds. Logic is otherwise byte-identical (same join, same aggregation choices, same percentile methodology) — this is a performance refactor, not a behavior change.

Result: confirmed via direct measurement, same-project values unchanged before/after:

	Before (live computation)	After (materialized lookup)
Latency (k8s)	3,318ms	6.7ms
Latency (react)	~3,300ms	10ms
Latency (opentelemetry)	~3,300ms	5.6ms
Rows scanned per call	~37,000,000	~5,400–8,200

~500x faster. Data values confirmed identical: Kubernetes directDependents=31058/Top 10%, transitiveDependents=677129/Top 1%, downloads=179350062/Top 25%, centrality=null — matches pre-fix exactly. pygithub (zero-package project) still correctly returns all-NULL fields rather than being dropped. Full-table row count still 13,548, matching insightsProjects exactly.

Deploy status

Deployed to production, including the performance fix. Not yet deployed to staging (see below) — this pipe's join chain could not be pushed to staging at all: lfx_insights_stg is missing the repos datasource entirely (not sparse — completely absent), a pre-existing staging environment gap unrelated to this PR. Flagging for separate follow-up, not attempting to fix staging here.

Test plan

• Review join logic against health_score_v2_impact.pipe precedent
• Confirm aggregation choices (MAX vs SUM per metric) — verified against real data, documented above
• Confirm percentile band thresholds (1% / 10% / 25% / 50%) — matches the "Top 1%" style badges the overview page design uses
• Deploy to staging — blocked by the missing repos datasource gap noted above
• Coordinate with the Insights PR that consumes this endpoint — already built and live end-to-end against this exact contract (insights PR Simplify onboarding flow in integrations #2054)
• Performance validated post-deploy — see table above

[cursor]

PR Summary

Medium Risk
New analytics surface with non-trivial join/aggregation and global ranking logic; behavior is documented as parity with the prior live pipe, but incorrect rollup would mislabel project impact bands until the next copy run.

Overview
Adds project-level Impact breakdown for the overview UI: four metrics (direct/transitive dependents, downloads, centrality) rolled up from linked packages, each with a global percentile and display band (Top 1% through Bottom 50%).

Rollup uses the same repositories → repos → packageRepos → ossPackages_enriched_ds chain as health_score_v2_impact.pipe, with LEFT JOINs from insightsProjects so projects without packages stay in the table with NULL metrics (not dropped). Aggregation is per metric (MAX for dependents/centrality, SUM for downloads).

Performance: the expensive global rank() window over the full joined population is moved into a nightly TYPE COPY pipe (40 2 * * *) into project_insights_impact_breakdown_ds. The public project_insights_impact_breakdown.pipe is now a filtered read from that table (same pattern as project_insights.pipe / project_insights_copy_ds), avoiding ~3s full scans per slug lookup.

^{Reviewed by Cursor Bugbot for commit e24e5f3. Bugbot is set up for automated code reviews on this repo. Configure here.}

[Copilot]

Pull request overview

Adds a Tinybird endpoint for project-level Impact metrics and global percentile bands.

Changes:

• Rolls package metrics up to projects.
• Computes global percentile bands with NULL handling.
• Supports filtering by project slug or ID.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[cursor]

Downloads sum inflated by join fan-out

Medium Severity

sum(pk.downloadsLast30d) aggregates over the joined rowset, so a package linked to multiple project repos via packageRepos, or duplicate unmerged repos / packageRepos rows (both ReplacingMergeTree, joined without FINAL), is counted more than once. That inflates project downloads and skews download percentiles/bands. The MAX metrics are resilient to this fan-out; SUM is not. Elsewhere, ossPackages_enriched.pipe uses FINAL and collapses packageRepos to one repo per package before aggregating.

 

^{Reviewed by Cursor Bugbot for commit 49e40be. Configure here.}

[cursor]

COPY schedule collides in dense window

High Severity

COPY_SCHEDULE 40 2 * * * lands in the dense 1–3 AM UTC window and collides minute-for-minute with leaderboards_project_active_organizations.pipe. The account’s 12-concurrent COPY quota is already under pressure there; jobs that exceed it can stick in queued permanently and empty downstream data.

 

^{Triggered by learned rule: Tinybird COPY pipe schedules must not collide — account concurrent quota is 12}

^{Reviewed by Cursor Bugbot for commit 8b2a09e. Configure here.}

[Copilot]

Pull request overview

Copilot reviewed 3 out of 3 changed files in this pull request and generated no new comments.

Suppressed comments (4)

services/libs/tinybird/pipes/project_insights_impact_breakdown_copy.pipe:85

• downloads is summed at the joined-row grain, but packageRepos explicitly permits one package to link to multiple repos. If two of those repos belong to the same project, the same package's 30-day downloads are counted more than once, inflating both the rollup and its percentile. Deduplicate by (projectId, packageId) before summing (do not use sumDistinct(downloadsLast30d), since different packages can legitimately have equal counts).

        sum(pk.downloadsLast30d) AS downloads,

services/libs/tinybird/pipes/project_insights_impact_breakdown_copy.pipe:94

• Both repos and packageRepos are ReplacingMergeTree datasources, but these joins omit FINAL. Until background merges complete, multiple physical versions can duplicate joined rows; that directly overcounts downloads and can select obsolete links. The canonical enrichment path reads both with FINAL (ossPackages_enriched.pipe:17,22).

    LEFT JOIN repos r ON r.url = rep.url
    LEFT JOIN packageRepos pr ON pr.repoId = r.id

services/libs/tinybird/pipes/project_insights_impact_breakdown_copy.pipe:96

• The global ranking population currently includes soft-deleted and disabled projects. insightsProjects.enabled defines whether a project is active in analytics, and established project queries exclude deletedAt; retaining inactive projects can shift every active project's percentile. Filter the base project population before grouping.

    FROM insightsProjects ip FINAL

services/libs/tinybird/pipes/project_insights_impact_breakdown_copy.pipe:92

• Soft-deleted repositories are still eligible when their enabled/excluded flags retain their prior values, so removed repos can continue contributing package metrics. Other active-repository rollups explicitly require isNull(deletedAt) (for example, repositories_populated_copy.pipe:8); apply the same filter here.

        AND rep.enabled = true
        AND rep.excluded = false

[cursor]

Cursor Bugbot has reviewed your changes and found 2 potential issues.

There are 4 total unresolved issues (including 2 from previous reviews).

^{Reviewed by Cursor Bugbot for commit e24e5f3. Configure here.}

[cursor]

Deleted projects skew percentiles

Medium Severity

project_impact_raw reads all insightsProjects rows without isNull(deletedAt) or enabled = 1. Soft-deleted and disabled projects still enter the global rank population when they have package metrics, which displaces active projects' percentile bands. Comparable pipes already exclude them.

 

^{Reviewed by Cursor Bugbot for commit e24e5f3. Configure here.}

[cursor]

Downloads sum can double-count

Medium Severity

downloads uses sum(pk.downloadsLast30d) over the repositories → repos → packageRepos join without deduplicating by packageId. packageRepos allows one package to link to multiple repos, so the same package can appear multiple times for one project and inflate downloads and its percentile.

 

^{Reviewed by Cursor Bugbot for commit e24e5f3. Configure here.}

[Copilot]

Pull request overview

Copilot reviewed 3 out of 3 changed files in this pull request and generated 1 comment.

Suppressed comments (1)

services/libs/tinybird/pipes/project_insights_impact_breakdown_copy.pipe:90

• This population includes soft-deleted/disabled projects and soft-deleted repositories. FINAL only selects the latest row version; it does not filter deletedAt. As a result, the endpoint can expose deleted projects and attribute packages from deleted repos, unlike the active-project population used by insights_projects_populated_copy.pipe:277 and health_score_security.pipe:8-12.

    FROM insightsProjects ip FINAL
    LEFT JOIN
        repositories rep FINAL
        ON rep.insightsProjectId = ip.id