[repository-quality] 🎯 Repository Quality Report: AIC Attribution Completeness & Token Weight Override Propagation (2026-06-23)

[github-actions[bot]]

🎯 Repository Quality Improvement Report - AIC Attribution Completeness & Token Weight Override Propagation

Analysis Date: 2026-06-23
Focus Area: AIC Attribution Completeness & Token Weight Override Propagation
Strategy Type: Custom (random=12, 0–59 tier)
Custom Area: Yes — engine.token-weights is a core correctness surface for cost trust.

Executive Summary

Analysis of the AIC attribution pipeline reveals three structural gaps producing silent cost mis-attribution. Most critically: TokenClassWeights-only configurations are silently dropped by compiler_yaml.go:953 (gate checks len(tw.Multipliers) > 0), and even when TokenWeights reaches aw_info.json, populateAIC() never reads or applies it. A third gap: 17/100 models in models.json fallback to input price for cache operations (over-estimates cost), and unrecognized models produce silent 0-AIC with no warning.

Full Analysis Report

Current State Assessment

Metric	Value	Status
Models in pkg/cli/data/models.json	100	✅
Models missing both cache price fields	17	⚠️
computeModelInferenceAIC accepts *TokenWeights	No	❌
populateAIC applies TokenWeights from aw_info.json	No	❌
Compiler emits token weights for class-weights-only configs	No	❌
Warning when model not found in catalog	None	❌

Key Findings

Gap 1 — Compiler serialization gate (compiler_yaml.go:953)

// only emits env var when Multipliers non-empty — drops TokenClassWeights-only configs
if data.EngineConfig != nil && data.EngineConfig.TokenWeights != nil && len(data.EngineConfig.TokenWeights.Multipliers) > 0 {

Gap 2 — populateAIC blind to TokenWeights (token_usage.go:1097)

func populateAIC(summary *TokenUsageSummary) {
    aic := computeModelInferenceAIC(usage.Provider, model,
        usage.InputTokens, ...) // no TokenWeights param — ignores workflow overrides

Gap 3 — Silent 0-AIC for unknown models (model_costs.go:181)

if !ok { return 0 }  // no warning, indistinguishable from zero-token run

---

🤖 Tasks for Copilot Agent

NOTE TO PLANNER AGENT: Split the following tasks into individual work items.

Improvement Tasks

Task 1: Fix compiler TokenClassWeights serialization gate

Priority: High Effort: Small
Code Region: pkg/workflow/compiler_yaml.go:950-966

Description: Change the emission condition from len(tw.Multipliers) > 0 to len(tw.Multipliers) > 0 || tw.TokenClassWeights != nil so class-weight-only workflows get GH_AW_INFO_TOKEN_WEIGHTS emitted.

Acceptance Criteria:

• Condition updated; TokenClassWeights-only configs emit the env var
• Comment updated to reflect new intent
• Compiler test asserts env var present for class-weights-only frontmatter

In `pkg/workflow/compiler_yaml.go` around line 953, change:
  if ... && len(data.EngineConfig.TokenWeights.Multipliers) > 0 {
to:
  tw := data.EngineConfig.TokenWeights
  if tw != nil && (len(tw.Multipliers) > 0 || tw.TokenClassWeights != nil) {
Update the comment above to: "Embed custom token weights when multipliers or class-weight overrides are configured."
Add a compiler test asserting GH_AW_INFO_TOKEN_WEIGHTS appears for a workflow with only token-class-weights set.

---

Task 2: Wire TokenWeights into CLI AIC computation

Priority: High Effort: Medium
Code Region: pkg/cli/model_costs.go:143-187, pkg/cli/token_usage.go:1097-1112

Description: Add weights *types.TokenWeights to computeModelInferenceCostUSD and populateAIC. Apply TokenClassWeights as per-class price overrides and Multipliers as a post-pricing scalar. Load TokenWeights from aw_info.json in analyzeTokenUsage.

Acceptance Criteria:

• computeModelInferenceCostUSD accepts and applies *types.TokenWeights
• Non-nil TokenClassWeights fields override pricing factors
• Multipliers key matching a model prefix scales the computed cost
• populateAIC reads aw_info.json and passes TokenWeights through
• nil weights gives identical results to current behavior
• Tests cover nil, multiplier-only, and class-weights-only cases

In `pkg/cli/model_costs.go`, add `weights *types.TokenWeights` param to `computeModelInferenceCostUSD`. Override each price factor from `weights.TokenClassWeights` when non-zero, then apply model multiplier from `weights.Multipliers` after computing base cost. Update `computeModelInferenceAIC` to forward the param. In `populateAIC`, accept `weights *types.TokenWeights` and pass through. In `analyzeTokenUsage`, load `aw_info.json` (use pattern from `audit_expanded.go:findAwInfoPath`) and extract `TokenWeights` to pass to `populateAIC`.

---

Task 3: Emit warning when model pricing is unavailable

Priority: Medium Effort: Small
Code Region: pkg/cli/token_usage.go:1097-1112 (populateAIC)

Description: After computing AIC, if token counts > 0 but AIC == 0, call addTokenUsageWarning naming the unknown model.

Acceptance Criteria:

• Warning added using awfTokenWarningPrefix convention when AIC=0 and tokens>0
• Models with genuinely zero tokens do not trigger warning
• Test TestPopulateAIC_WarnsForUnknownModel asserts warning contains model name

In `populateAIC`, after computing `aic`, add:
  totalTokens := usage.InputTokens + usage.OutputTokens + usage.CacheReadTokens + usage.CacheWriteTokens + usage.ReasoningTokens
  if aic == 0 && totalTokens > 0 {
    addTokenUsageWarning(summary, fmt.Sprintf("%s no pricing data for model %q; AIC shown as 0", awfTokenWarningPrefix, model))
  }
Add unit test asserting warning is present for unknown model with non-zero InputTokens.

---

Task 4: Complete cache pricing in models.json for 17 missing models

Priority: Medium Effort: Small
Code Region: pkg/cli/data/models.json

Description: 17 models lack cache_read/cache_write fields and fall back to input price. Add correct values from provider pricing pages, or explicit zeros for non-cacheable endpoints.

Acceptance Criteria:

• All 17 identified models have cache_read and cache_write fields or explicit "0" with comment
• Test asserts key models (Anthropic, OpenAI) all have cache pricing
• PR description cites pricing source URL for each changed entry

Run: python3 -c "import json; [print(f'{p}/{m}') for p,pd in json.load(open('pkg/cli/data/models.json'))['providers'].items() for m,e in pd['models'].items() if 'cache_read' not in e.get('cost',{})]"
For each result, look up current pricing from provider's public pricing page and add cache_read/cache_write in scientific notation matching existing format.
Add test in model_costs_test.go asserting standard Anthropic/OpenAI models have both cache fields.

---

Task 5: Add end-to-end TokenWeights attribution test

Priority: Low Effort: Medium
Code Region: pkg/workflow/compiler_yaml_test.go, pkg/cli/model_costs_test.go

Description: No test exercises the full round-trip: frontmatter → compiler → env var → reparsed TokenWeights → computeModelInferenceCostUSD. Add a table-driven test covering multiplier-only and class-weights-only cases.

Acceptance Criteria:

• Test tagged //go:build !integration
• Multiplier 2.0 produces exactly 2× baseline AIC
• TokenClassWeights{Output: 8.0} scales output-token cost proportionally
• Compiler test asserts env var emitted for class-weights-only frontmatter

Create pkg/cli/token_weights_e2e_test.go (build tag !integration). Test: marshal types.TokenWeights{Multipliers: {"test-model": 3.0}}, unmarshal, call computeModelInferenceCostUSD with weights, assert result is 3x baseline. Second sub-test: TokenClassWeights{Output: 8.0} scales output proportionally. Add compiler test in compiler_yaml_test.go asserting GH_AW_INFO_TOKEN_WEIGHTS appears for class-weights-only workflow frontmatter.

---

📊 Historical Context

Previous Focus Areas (last 10)

Date	Focus Area	Type	Custom
2026-06-15	Documentation	Standard	N
2026-06-16	Benchmark Infrastructure	Standard	N
2026-06-17	Engine Catalog Adoption Gap	Custom	Y
2026-06-18	Race Detector Gap	Custom	Y
2026-06-19	Compilation Idempotence Gaps	Custom	Y
2026-06-22	Context Propagation & Cancellability	Standard	N
2026-06-23	AIC Attribution Completeness	Custom	Y

---

🎯 Recommendations

Immediate Actions (This Week)

1. Fix TokenClassWeights compiler gate — one-liner change — Priority: High
2. Add 0-AIC warning in populateAIC — Priority: High

Short-term Actions (This Month)

1. Wire TokenWeights into CLI AIC computation — Priority: High
2. Complete cache pricing in models.json — Priority: Medium

Long-term Actions (This Quarter)

1. End-to-end TokenWeights attribution test — Priority: Low

---

📈 Success Metrics

• TokenClassWeights-only configs emitting env var: 0% → 100%
• Models with complete cache pricing: 83/100 → 100/100
• AIC computation applying workflow TokenWeights: 0% → 100%
• Unknown-model AIC warnings: 0 → 1 per unrecognized model with non-zero tokens

---

Next Steps

1. Review and prioritise the 5 tasks above
2. Assign Tasks 1–3 to Copilot coding agent via planner agent
3. Re-evaluate after Task 2 lands (~2026-07)

---

References:

• [§28030681606] — This workflow run

Generated by ⚡ Repository Quality Improvement Agent · 102.9 AIC · ⌖ 8.21 AIC · ⊞ 6.6K · ◷

• expires on Jun 24, 2026, 5:54 AM UTC-08:00

[github-actions[bot]]

Smoke bot tap wall. Me see latest cave post. Run 28035178551 done.

Warning

Firewall blocked 6 domains

The following domains were blocked by the firewall during workflow execution:

• accounts.google.com
• android.clients.google.com
• clients2.google.com
• contentautofill.googleapis.com
• safebrowsingohttpgateway.googleapis.com
• www.google.com

To allow these domains, add them to the network.allowed list in your workflow frontmatter:

network:
  allowed:
    - defaults
    - "accounts.google.com"
    - "android.clients.google.com"
    - "clients2.google.com"
    - "contentautofill.googleapis.com"
    - "safebrowsingohttpgateway.googleapis.com"
    - "www.google.com"

See Network Configuration for more information.

📰 BREAKING: Report filed by Smoke Copilot · 352.3 AIC · ⌖ 13.6 AIC · ⊞ 19.2K · ◷

[github-actions[bot]]

This discussion was automatically closed because it expired on 2026-06-24T13:54:43.945Z.

Closed by Workflow