-
-
Notifications
You must be signed in to change notification settings - Fork 0
Interpreting findings
What OikosBot's output means, and how much weight to put on it.
Findings are emitted as SARIF 2.1.0 and ingested by GitHub code scanning. Each result carries a property bag:
| Property | Meaning |
|---|---|
eco_score, econ_score, quality_score, overall_health
|
0–100 component scores |
energy_joules, carbon_gco2e, duration_ms, memory_bytes
|
Resource estimates for the unit analysed |
confidence |
Estimated / Calibrated / Measured — read this first
|
pareto_status |
optimal (on the frontier) or dominated
|
pareto_score |
0–100; distance to the frontier in normalised space |
pareto_dominated_by |
How many peers dominate this one |
suggestion |
Advisory prose, when present |
Severity is driven by eco_score alone: < 30 → error, < 60 → warning, otherwise note. Pareto status does not influence severity.
fixes[]is deliberately absent. OikosBot's suggestions are advisory prose, not text replacements. Emitting them as SARIFfixeswithout the spec-requiredartifactChangescaused GitHub to reject the entire upload — so for a period, no OikosBot SARIF was ingestible at all. Suggestions now live in the message and inproperties.suggestion, with a regression test guarding it.
oikosbot compare <base> <head> returns one of four verdicts:
| Verdict | Means |
|---|---|
| Improvement | Head dominates base — better on at least one axis, worse on none |
| Regression | Base dominates head |
| Trade-off | Some axes improved, others worsened. Neither dominates |
| Neutral | No axis moved by more than ε |
Trade-off is not a failure state. It is the honest answer when a change buys speed with memory, or reduces carbon at the cost of complexity. OikosBot's position is that these should be named and documented, not averaged into a single number that hides them.
To document one, put a Pareto-Trade-off: trailer in the PR body, or a heading containing "pareto trade-off":
Pareto-Trade-off: +18% memory to remove an O(n²) scan; net carbon down 31%.
The comparison is ε-tolerant (1e-6): a difference smaller than ε is treated as no change, so estimate noise cannot manufacture a "strictly better" axis.
Check confidence before anything else.
-
Estimated— every per-file finding today. Resource figures derive from an AST node count, so they are directional: useful for ranking functions within a repository, or spotting an outlier. Not defensible as absolute numbers. -
Measured— estate-level wall-clock and the capability signals. These come straight from the API.
There is a deeper caveat on per-file findings, and it is documented rather than hidden: because energy, duration, carbon and memory are all derived from the same node count, they are not independent axes. A Pareto frontier over them is closer to a one-dimensional sort than a genuine multi-objective frontier. See What is owed.
Three sections, read in this order:
- The frontier — repositories at θ = 1.0. These define best practice within this estate; DEA is relative, so a frontier repo is not "good in absolute terms", it is "not dominated by any peer here".
- Off-frontier, with peers — the actionable part. Each entry names the frontier repositories it is compared against, so the question becomes "what does that one do differently?" rather than "why is my number low?"
- X-inefficiency — repositories consuming real input with zero verified output.
On the term "X-inefficiency": Leibenstein's concept (firms with identical technology and inputs achieving very different results, for reasons that are not technical) fits this pattern well. But there is no software-engineering literature applying it, so its use here is our framing, not an established result. Stated plainly so nobody cites it as one.
Findings with rule IDs like oikosbot/policy-<name> come from the Eclexia policy layer.
.ecl filename and applies hardcoded thresholds — it never reads the file's contents, and its thresholds contradict the files they claim to implement. Every such evaluation now prints a ::warning:: saying so. See What is owed.