Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
20 commits
Select commit Hold shift + click to select a range
2b33d47
Add more benchmarks
Pringled Apr 16, 2026
2c6e4c2
Add more benchmarks
Pringled Apr 16, 2026
20a3328
Add more benchmarks
Pringled Apr 16, 2026
9b33742
Merge remote-tracking branch 'origin/main' into expand-benchmark
Pringled Apr 16, 2026
03cedba
fix: Remove 15 broken annotations and fix 5 wrong paths
Pringled Apr 16, 2026
67ff077
feat: Tune NL ranking parameters for improved NDCG@10
Pringled Apr 16, 2026
7e6c345
chore: Backfill categories for all annotations and top up express
Pringled Apr 16, 2026
45396aa
fix: Fix cobra annotation - help.go merged into command.go
Pringled Apr 17, 2026
1cb01e7
fix: Correct 6 wrong annotation targets found in content audit
Pringled Apr 17, 2026
8c80270
chore: Top up 8 thin repos to 20 tasks each
Pringled Apr 17, 2026
6c14e10
Add 8 new repos to benchmark (C, Bash, Haskell, Dart, Lua, Zig)
Pringled Apr 17, 2026
569434f
expand benchmark to 66 repos across 20 languages; fix aggregation to …
Pringled Apr 17, 2026
970ee37
add libuv as 3rd C repo; all 20 languages now have 3+ repos
Pringled Apr 17, 2026
0e626b1
benchmark quality improvements: n_relevant fix, per-category metrics,…
Pringled Apr 17, 2026
3f7f018
fix ktor/rxswift annotation bugs; ktor server query replaced with cli…
Pringled Apr 17, 2026
19352d3
Fix annotation bugs across sinatra, circe, cats, rxswift, exposed
Pringled Apr 17, 2026
d684944
Refresh benchmark results (drop stale, add current run)
Pringled Apr 17, 2026
a6fc524
Drop nvim-lspconfig; fix non-standard annotation categories
Pringled Apr 17, 2026
077d940
Remove stale benchmark result file
Pringled Apr 17, 2026
3b03cac
Regenerate benchmark results after fixing api/keyword categories
Pringled Apr 17, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
122 changes: 122 additions & 0 deletions benchmarks/annotations/abseil-cpp.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,122 @@
[
{
"query": "absl::flat_hash_map and flat_hash_set hash table containers",
"relevant": ["absl/container/flat_hash_map.h"],
"secondary": ["absl/container/flat_hash_set.h"],
"category": "semantic"
},
{
"query": "raw_hash_set open addressing SIMD group-based slot probing implementation",
"relevant": ["absl/container/internal/raw_hash_set.h"],
"secondary": [],
"category": "architecture"
},
{
"query": "absl::string_view for non-owning string references",
"relevant": ["absl/strings/string_view.h"],
"secondary": [],
"category": "semantic"
},
{
"query": "absl::StrCat and StrAppend for efficient string concatenation",
"relevant": ["absl/strings/str_cat.h"],
"secondary": [],
"category": "semantic"
},
{
"query": "absl::StrSplit and StrJoin for string splitting and joining",
"relevant": ["absl/strings/str_split.h"],
"secondary": [],
"category": "semantic"
},
{
"query": "absl::Status and StatusOr for error propagation",
"relevant": ["absl/status/status.h"],
"secondary": ["absl/status/statusor.h"],
"category": "semantic"
},
{
"query": "absl::Time and absl::Duration for time representation",
"relevant": ["absl/time/time.h"],
"secondary": [],
"category": "semantic"
},
{
"query": "absl::optional and absl::variant for vocabulary types",
"relevant": ["absl/types/optional.h"],
"secondary": ["absl/types/variant.h"],
"category": "semantic"
},
{
"query": "absl::Mutex and MutexLock for synchronization",
"relevant": ["absl/synchronization/mutex.h"],
"secondary": [],
"category": "semantic"
},
{
"query": "absl logging macros ABSL_LOG and ABSL_CHECK",
"relevant": ["absl/log/absl_log.h"],
"secondary": [],
"category": "semantic"
},
{
"query": "absl::Span for non-owning sequence view",
"relevant": ["absl/types/span.h"],
"secondary": [],
"category": "semantic"
},
{
"query": "absl::node_hash_map for pointer-stable hash map",
"relevant": ["absl/container/node_hash_map.h"],
"secondary": [],
"category": "semantic"
},
{
"query": "absl::Hash and the Abseil hashing framework",
"relevant": ["absl/hash/hash.h"],
"secondary": [],
"category": "architecture"
},
{
"query": "absl::Notification for one-time event signaling between threads",
"relevant": ["absl/synchronization/notification.h"],
"secondary": [],
"category": "semantic"
},
{
"query": "absl::AnyInvocable for move-only callable wrapper",
"relevant": ["absl/functional/any_invocable.h"],
"secondary": [],
"category": "semantic"
},
{
"query": "absl::btree_map and btree_set sorted containers",
"relevant": ["absl/container/btree_map.h"],
"secondary": [],
"category": "semantic"
},
{
"query": "absl::StrFormat and string formatting API",
"relevant": ["absl/strings/str_format.h"],
"secondary": [],
"category": "semantic"
},
{
"query": "flat_hash_map",
"relevant": ["absl/container/flat_hash_map.h"],
"secondary": [],
"category": "symbol"
},
{
"query": "Status",
"relevant": ["absl/status/status.h"],
"secondary": [],
"category": "symbol"
},
{
"query": "Mutex",
"relevant": ["absl/synchronization/mutex.h"],
"secondary": [],
"category": "symbol"
}
]
122 changes: 122 additions & 0 deletions benchmarks/annotations/aeson.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,122 @@
[
{
"query": "parsing a JSON object field with the (.:) operator",
"relevant": ["src/Data/Aeson/Types/FromJSON.hs"],
"secondary": [],
"category": "semantic"
},
{
"query": "how withObject expects a JSON object and runs a parser on it",
"relevant": ["src/Data/Aeson/Types/FromJSON.hs"],
"secondary": [],
"category": "semantic"
},
{
"query": "how typeMismatch produces an error message for wrong JSON types",
"relevant": ["src/Data/Aeson/Types/FromJSON.hs"],
"secondary": [],
"category": "semantic"
},
{
"query": "how parseEither runs a parser and returns Either String",
"relevant": ["src/Data/Aeson/Types/Internal.hs"],
"secondary": [],
"category": "semantic"
},
{
"query": "how parseMaybe runs a parser and returns Nothing on failure",
"relevant": ["src/Data/Aeson/Types/Internal.hs"],
"secondary": [],
"category": "semantic"
},
{
"query": "how withText parses a JSON string value",
"relevant": ["src/Data/Aeson/Types/FromJSON.hs"],
"secondary": [],
"category": "semantic"
},
{
"query": "how withArray parses a JSON array value",
"relevant": ["src/Data/Aeson/Types/FromJSON.hs"],
"secondary": [],
"category": "semantic"
},
{
"query": "how Template Haskell derives FromJSON and ToJSON instances",
"relevant": ["src/Data/Aeson/TH.hs"],
"secondary": [],
"category": "architecture"
},
{
"query": "how genericToJSON uses GHC generics to encode a value",
"relevant": ["src/Data/Aeson/Types/ToJSON.hs"],
"secondary": [],
"category": "architecture"
},
{
"query": "how camelCase field names are converted to snake_case for JSON keys",
"relevant": ["src/Data/Aeson/Types/Internal.hs"],
"secondary": [],
"category": "semantic"
},
{
"query": "how modifyFailure customizes error messages in a parser",
"relevant": ["src/Data/Aeson/Types/Internal.hs"],
"secondary": [],
"category": "semantic"
},
{
"query": "how optional fields use (.:?) with a default value via (.!=)",
"relevant": ["src/Data/Aeson/Types/FromJSON.hs"],
"secondary": [],
"category": "semantic"
},
{
"query": "how JSON encoding produces Encoding without materializing a Value",
"relevant": ["src/Data/Aeson/Encoding.hs"],
"secondary": [],
"category": "architecture"
},
{
"query": "how Key type is used as object field identifier",
"relevant": ["src/Data/Aeson/KeyMap.hs"],
"secondary": ["src/Data/Aeson/Key.hs"],
"category": "semantic"
},
{
"query": "how withScientific parses a JSON number as Scientific",
"relevant": ["src/Data/Aeson/Types/FromJSON.hs"],
"secondary": [],
"category": "semantic"
},
{
"query": "how the Parser monad accumulates and reports JSON path errors",
"relevant": ["src/Data/Aeson/Types/Internal.hs"],
"secondary": [],
"category": "architecture"
},
{
"query": "how toJSONKeyText creates a ToJSONKey instance from a text function",
"relevant": ["src/Data/Aeson/Types/ToJSON.hs"],
"secondary": [],
"category": "semantic"
},
{
"query": "how emptyObject and emptyArray constants are defined",
"relevant": ["src/Data/Aeson/Types/Internal.hs"],
"secondary": [],
"category": "semantic"
},
{
"query": "how sum types are encoded with tag and contents fields",
"relevant": ["src/Data/Aeson/Types/ToJSON.hs"],
"secondary": [],
"category": "architecture"
},
{
"query": "how formatError formats a JSONPath and message into a string",
"relevant": ["src/Data/Aeson/Types/Internal.hs"],
"secondary": [],
"category": "semantic"
}
]
Loading
Loading