Skip to content

feat: add star-count metric (WatchEvents) #3

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
guenhter merged 3 commits into from
Jul 19, 2026
Merged

feat: add star-count metric (WatchEvents) #3

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
23071b6
feat: add star-count metric (WatchEvents)
guenhter Jul 19, 2026
81a8dd0
refactor: replace "six" with "multiple" in docs/comments; stable sort…
guenhter Jul 19, 2026
78c15d6
chore: update data
guenhter Jul 19, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
The table of contents is too big for display.
Diff view
Diff view
  •  
  •  
  •  
4 changes: 2 additions & 2 deletions AGENTS.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@

**githubstats** is a data pipeline and interactive visualization for GitHub programming language popularity, derived from real developer activity on [GH Archive](https://www.gharchive.org/).

The pipeline downloads every public GitHub event for a given month, filters out bots and automated noise, resolves language compositions per repository, and computes five weighted language activity ratings. The results are consumed by a single-page web UI (`index.html`) that renders an interactive trend chart (via Apache ECharts) and a ranked table with sparklines.
The pipeline downloads every public GitHub event for a given month, filters out bots and automated noise, resolves language compositions per repository, and computes multiple weighted language activity ratings. The results are consumed by a single-page web UI (`index.html`) that renders an interactive trend chart (via Apache ECharts) and a ranked table with sparklines.

The project covers January 2015 through the present (~11 years of monthly data). A key data discontinuity exists: GitHub removed language data from event payloads in October 2025. Before that date, language attribution uses the single primary language field; from October 2025 onward, the GraphQL API is required for full multi-language breakdown.

Expand Down Expand Up @@ -55,7 +55,7 @@ githubstats/
| Per-month rating (committed) | `data/language-ratings-YYYY-MM-<type>.jsonl` |
| Combined all-months rating (committed) | `data/language-ratings-all-<type>.jsonl` |

**Valid `<type>` values:** `pr-count`, `issue-count`, `push-count`, `developer-activity`, `active-repos`
**Valid `<type>` values:** `pr-count`, `issue-count`, `push-count`, `developer-activity`, `active-repos`, `star-count`

### Build and Run Commands

Expand Down
8 changes: 5 additions & 3 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -81,14 +81,14 @@ awk -F',' 'NR>1 && $3=="PushEvent" {print $2}' \
--workers 10 \
> "data/languages-${YEAR}-${MONTH}.jsonl"

# Step 4 — compute weighted per-language ratings (five output files)
# Step 4 — compute weighted per-language ratings (one output file per statistic type)
cargo run --release --bin produce_statistics -- \
--archive "data/archive-${YEAR}${MONTH}-filtered.csv" \
--languages "data/languages-${YEAR}-${MONTH}.jsonl" \
--output-dir data/

# Step 5 — pack all monthly ratings into combined files (run once after all months are produced)
for TYPE in pr-count issue-count push-count developer-activity active-repos; do
for TYPE in pr-count issue-count push-count developer-activity active-repos star-count; do
cargo run --release --bin pack_statistics -- \
--type "$TYPE" \
--input-dir data/ \
Expand All @@ -98,7 +98,7 @@ done

### Rating files

`produce_statistics` writes five JSONL files, all sorted descending by rating:
`produce_statistics` writes multiple JSONL files, all sorted descending by rating:

| File | Signal | Formula |
|---|---|---|
Expand All @@ -107,6 +107,7 @@ done
| `language-ratings-YYYY-MM-push-count.jsonl` | Push volume | `rating[L] += push_count × (size_L / total_size)` |
| `language-ratings-YYYY-MM-developer-activity.jsonl` | Distinct contributors (PR + push) | `rating[L] += distinct_contributors × (size_L / total_size)` |
| `language-ratings-YYYY-MM-active-repos.jsonl` | Active repository breadth | `rating[L] += 1 × (size_L / total_size)` per active repo |
| `language-ratings-YYYY-MM-star-count.jsonl` | Stars (WatchEvents) | `rating[L] += star_count × (size_L / total_size)` |

Each record:
```json
Expand All @@ -122,6 +123,7 @@ Each record:
| `language-ratings-all-push-count.jsonl` | All months, push-count, sorted chronologically |
| `language-ratings-all-developer-activity.jsonl` | All months, developer-activity, sorted chronologically |
| `language-ratings-all-active-repos.jsonl` | All months, active-repos, sorted chronologically |
| `language-ratings-all-star-count.jsonl` | All months, star-count, sorted chronologically |

Each record has a `month` field prepended:
```json
Expand Down
120 changes: 60 additions & 60 deletions data/language-ratings-2015-01-active-repos.jsonl
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -41,10 +41,10 @@
{"language":"D","percentage":0.09,"rating":78.0}
{"language":"DM","percentage":0.09,"rating":74.0}
{"language":"Visual Basic","percentage":0.08,"rating":68.0}
{"language":"Scheme","percentage":0.07,"rating":57.0}
{"language":"ActionScript","percentage":0.07,"rating":57.0}
{"language":"Pascal","percentage":0.06,"rating":54.0}
{"language":"Scheme","percentage":0.07,"rating":57.0}
{"language":"Assembly","percentage":0.06,"rating":54.0}
{"language":"Pascal","percentage":0.06,"rating":54.0}
{"language":"FORTRAN","percentage":0.05,"rating":47.0}
{"language":"SQF","percentage":0.05,"rating":46.0}
{"language":"PureScript","percentage":0.05,"rating":41.0}
Expand All @@ -54,110 +54,110 @@
{"language":"Vala","percentage":0.03,"rating":25.0}
{"language":"Objective-C++","percentage":0.03,"rating":23.0}
{"language":"Smalltalk","percentage":0.03,"rating":23.0}
{"language":"Prolog","percentage":0.03,"rating":22.0}
{"language":"Logos","percentage":0.03,"rating":22.0}
{"language":"Elm","percentage":0.03,"rating":22.0}
{"language":"Logos","percentage":0.03,"rating":22.0}
{"language":"Prolog","percentage":0.03,"rating":22.0}
{"language":"Processing","percentage":0.02,"rating":21.0}
{"language":"ASP","percentage":0.02,"rating":20.0}
{"language":"Apex","percentage":0.02,"rating":19.0}
{"language":"VHDL","percentage":0.02,"rating":18.0}
{"language":"XQuery","percentage":0.02,"rating":18.0}
{"language":"Mathematica","percentage":0.02,"rating":17.0}
{"language":"AGS Script","percentage":0.02,"rating":17.0}
{"language":"Mathematica","percentage":0.02,"rating":17.0}
{"language":"Tcl","percentage":0.02,"rating":17.0}
{"language":"SourcePawn","percentage":0.02,"rating":16.0}
{"language":"Nimrod","percentage":0.02,"rating":16.0}
{"language":"Nix","percentage":0.02,"rating":16.0}
{"language":"SourcePawn","percentage":0.02,"rating":16.0}
{"language":"Cuda","percentage":0.02,"rating":15.0}
{"language":"OpenSCAD","percentage":0.02,"rating":15.0}
{"language":"Verilog","percentage":0.02,"rating":15.0}
{"language":"Game Maker Language","percentage":0.01,"rating":11.0}
{"language":"LabVIEW","percentage":0.01,"rating":11.0}
{"language":"BitBake","percentage":0.01,"rating":10.0}
{"language":"AutoHotkey","percentage":0.01,"rating":10.0}
{"language":"BitBake","percentage":0.01,"rating":10.0}
{"language":"VCL","percentage":0.01,"rating":9.0}
{"language":"Coq","percentage":0.01,"rating":8.0}
{"language":"Kotlin","percentage":0.01,"rating":8.0}
{"language":"M","percentage":0.01,"rating":8.0}
{"language":"PAWN","percentage":0.01,"rating":7.0}
{"language":"Idris","percentage":0.01,"rating":7.0}
{"language":"LiveScript","percentage":0.01,"rating":7.0}
{"language":"OpenEdge ABL","percentage":0.01,"rating":7.0}
{"language":"PAWN","percentage":0.01,"rating":7.0}
{"language":"Propeller Spin","percentage":0.01,"rating":7.0}
{"language":"Idris","percentage":0.01,"rating":7.0}
{"language":"Xtend","percentage":0.01,"rating":6.0}
{"language":"CartoCSS","percentage":0.01,"rating":6.0}
{"language":"nesC","percentage":0.01,"rating":6.0}
{"language":"Stata","percentage":0.01,"rating":6.0}
{"language":"Xtend","percentage":0.01,"rating":6.0}
{"language":"nesC","percentage":0.01,"rating":6.0}
{"language":"Gosu","percentage":0.01,"rating":5.0}
{"language":"NetLogo","percentage":0.01,"rating":5.0}
{"language":"IDL","percentage":0.01,"rating":5.0}
{"language":"NetLogo","percentage":0.01,"rating":5.0}
{"language":"SuperCollider","percentage":0.01,"rating":5.0}
{"language":"Agda","percentage":0.0,"rating":4.0}
{"language":"SAS","percentage":0.0,"rating":4.0}
{"language":"ooc","percentage":0.0,"rating":4.0}
{"language":"Zephir","percentage":0.0,"rating":4.0}
{"language":"BlitzBasic","percentage":0.0,"rating":4.0}
{"language":"Standard ML","percentage":0.0,"rating":4.0}
{"language":"Hy","percentage":0.0,"rating":4.0}
{"language":"Dylan","percentage":0.0,"rating":3.0}
{"language":"SAS","percentage":0.0,"rating":4.0}
{"language":"Standard ML","percentage":0.0,"rating":4.0}
{"language":"Zephir","percentage":0.0,"rating":4.0}
{"language":"ooc","percentage":0.0,"rating":4.0}
{"language":"ANTLR","percentage":0.0,"rating":3.0}
{"language":"ATS","percentage":0.0,"rating":3.0}
{"language":"Ada","percentage":0.0,"rating":3.0}
{"language":"AspectJ","percentage":0.0,"rating":3.0}
{"language":"Eiffel","percentage":0.0,"rating":3.0}
{"language":"Hack","percentage":0.0,"rating":3.0}
{"language":"Squirrel","percentage":0.0,"rating":3.0}
{"language":"Augeas","percentage":0.0,"rating":3.0}
{"language":"Mercury","percentage":0.0,"rating":3.0}
{"language":"ANTLR","percentage":0.0,"rating":3.0}
{"language":"Crystal","percentage":0.0,"rating":3.0}
{"language":"Awk","percentage":0.0,"rating":3.0}
{"language":"Max","percentage":0.0,"rating":3.0}
{"language":"Brightscript","percentage":0.0,"rating":3.0}
{"language":"ATS","percentage":0.0,"rating":3.0}
{"language":"Inform 7","percentage":0.0,"rating":3.0}
{"language":"Chapel","percentage":0.0,"rating":3.0}
{"language":"Ada","percentage":0.0,"rating":3.0}
{"language":"Crystal","percentage":0.0,"rating":3.0}
{"language":"Dylan","percentage":0.0,"rating":3.0}
{"language":"Eiffel","percentage":0.0,"rating":3.0}
{"language":"Hack","percentage":0.0,"rating":3.0}
{"language":"Inform 7","percentage":0.0,"rating":3.0}
{"language":"Max","percentage":0.0,"rating":3.0}
{"language":"Mercury","percentage":0.0,"rating":3.0}
{"language":"Squirrel","percentage":0.0,"rating":3.0}
{"language":"xBase","percentage":0.0,"rating":3.0}
{"language":"REALbasic","percentage":0.0,"rating":2.0}
{"language":"MoonScript","percentage":0.0,"rating":2.0}
{"language":"APL","percentage":0.0,"rating":2.0}
{"language":"Bison","percentage":0.0,"rating":2.0}
{"language":"BlitzMax","percentage":0.0,"rating":2.0}
{"language":"Component Pascal","percentage":0.0,"rating":2.0}
{"language":"DOT","percentage":0.0,"rating":2.0}
{"language":"Ecl","percentage":0.0,"rating":2.0}
{"language":"EmberScript","percentage":0.0,"rating":2.0}
{"language":"Component Pascal","percentage":0.0,"rating":2.0}
{"language":"Xojo","percentage":0.0,"rating":2.0}
{"language":"Factor","percentage":0.0,"rating":2.0}
{"language":"GAP","percentage":0.0,"rating":2.0}
{"language":"Pike","percentage":0.0,"rating":2.0}
{"language":"DOT","percentage":0.0,"rating":2.0}
{"language":"Gnuplot","percentage":0.0,"rating":2.0}
{"language":"LSL","percentage":0.0,"rating":2.0}
{"language":"Lasso","percentage":0.0,"rating":2.0}
{"language":"MoonScript","percentage":0.0,"rating":2.0}
{"language":"Nemerle","percentage":0.0,"rating":2.0}
{"language":"Pure Data","percentage":0.0,"rating":2.0}
{"language":"Objective-J","percentage":0.0,"rating":2.0}
{"language":"Parrot","percentage":0.0,"rating":2.0}
{"language":"Bison","percentage":0.0,"rating":2.0}
{"language":"Pike","percentage":0.0,"rating":2.0}
{"language":"Pure Data","percentage":0.0,"rating":2.0}
{"language":"REALbasic","percentage":0.0,"rating":2.0}
{"language":"XML","percentage":0.0,"rating":2.0}
{"language":"Lasso","percentage":0.0,"rating":2.0}
{"language":"Objective-J","percentage":0.0,"rating":2.0}
{"language":"APL","percentage":0.0,"rating":2.0}
{"language":"Gnuplot","percentage":0.0,"rating":2.0}
{"language":"LSL","percentage":0.0,"rating":2.0}
{"language":"Factor","percentage":0.0,"rating":2.0}
{"language":"BlitzMax","percentage":0.0,"rating":2.0}
{"language":"Grammatical Framework","percentage":0.0,"rating":1.0}
{"language":"Ceylon","percentage":0.0,"rating":1.0}
{"language":"UnrealScript","percentage":0.0,"rating":1.0}
{"language":"Pan","percentage":0.0,"rating":1.0}
{"language":"Red","percentage":0.0,"rating":1.0}
{"language":"Bro","percentage":0.0,"rating":1.0}
{"language":"Ragel in Ruby Host","percentage":0.0,"rating":1.0}
{"language":"HaXe","percentage":0.0,"rating":1.0}
{"language":"Xojo","percentage":0.0,"rating":2.0}
{"language":"ABAP","percentage":0.0,"rating":1.0}
{"language":"AppleScript","percentage":0.0,"rating":1.0}
{"language":"AutoIt","percentage":0.0,"rating":1.0}
{"language":"XC","percentage":0.0,"rating":1.0}
{"language":"ABAP","percentage":0.0,"rating":1.0}
{"language":"Io","percentage":0.0,"rating":1.0}
{"language":"Slash","percentage":0.0,"rating":1.0}
{"language":"Frege","percentage":0.0,"rating":1.0}
{"language":"Bluespec","percentage":0.0,"rating":1.0}
{"language":"Bro","percentage":0.0,"rating":1.0}
{"language":"Ceylon","percentage":0.0,"rating":1.0}
{"language":"Dogescript","percentage":0.0,"rating":1.0}
{"language":"Golo","percentage":0.0,"rating":1.0}
{"language":"Shen","percentage":0.0,"rating":1.0}
{"language":"XProc","percentage":0.0,"rating":1.0}
{"language":"Monkey","percentage":0.0,"rating":1.0}
{"language":"Frege","percentage":0.0,"rating":1.0}
{"language":"GDScript","percentage":0.0,"rating":1.0}
{"language":"Golo","percentage":0.0,"rating":1.0}
{"language":"Grammatical Framework","percentage":0.0,"rating":1.0}
{"language":"HaXe","percentage":0.0,"rating":1.0}
{"language":"Io","percentage":0.0,"rating":1.0}
{"language":"JSONiq","percentage":0.0,"rating":1.0}
{"language":"Monkey","percentage":0.0,"rating":1.0}
{"language":"Pan","percentage":0.0,"rating":1.0}
{"language":"Ragel in Ruby Host","percentage":0.0,"rating":1.0}
{"language":"Red","percentage":0.0,"rating":1.0}
{"language":"Shen","percentage":0.0,"rating":1.0}
{"language":"Slash","percentage":0.0,"rating":1.0}
{"language":"Thrift","percentage":0.0,"rating":1.0}
{"language":"Bluespec","percentage":0.0,"rating":1.0}
{"language":"UnrealScript","percentage":0.0,"rating":1.0}
{"language":"XC","percentage":0.0,"rating":1.0}
{"language":"XProc","percentage":0.0,"rating":1.0}
Loading
Loading