Feature/3.0 - #32
Merged
Merged
Conversation
feat: add architecture decisions and tests for LLM providers and PDF …
feat: optimize sorting of findings by savings and update README with …
feat: implement pagination and sorting for findings in the dashboard
…omparison to Cloud Custodian
docs: enhance README with CloudOracle's analysis-first approach and c…
…rfaces for AWS, Azure, and GCP
…and snapshot creation
…ce resource attributes extraction
…ation and filtering
…hangeEstimate struct
- Added detailed integration tests for EC2, RDS, EBS, Lambda, NAT Gateway, and Aurora PostgreSQL cost estimation logic. - Introduced `checkpoint135` for estimating Terraform resource change costs from a Terraform plan file. - Added `probe-pricing` for diagnosing AWS Pricing API warnings and exploring resource attributes. - Integrated Terraform files for configuring AWS provider dependencies and locking versions.
Feature/terraform
`uv run python -m insights_agent.main "<question>"` (or the `insights-agent` console script) wires Settings → logging → GeminiProvider → CloudOracleClient → ReAct graph and prints either the natural-language answer or a JSON envelope (`--json`). `--verbose` streams the tool calls the model made to stderr so the operator can see which /api/v1 endpoint was actually consulted. Top-level error handling maps the realistic failure modes to distinct exit codes (130 on Ctrl-C, 2 on missing/invalid config, 1 on runtime errors) so callers in shell pipelines can branch deterministically. Includes the tiny `build_graph` signature widening `list[BaseTool] → Sequence[BaseTool]` so the CLI can pass the result of `build_tools` without an extra conversion at the call site. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
… diagram `insights-agent/README.md` was a stub. Replace it with a self-contained setup-in-under-10-minutes guide covering: prerequisites, `uv sync`, the seven env vars (required vs optional, defaults), CLI flags and exit codes, an end-to-end smoke test the operator can run by hand against a local Go server, dev workflow (pytest / ruff / mypy), and a one-page architecture pointer table mapping concerns to source files. Root README gains an "AI Insights Agent" section with a Mermaid arch diagram (User → CLI → LangGraph → Gemini → tools → Go API → Postgres) and a roadmap update marking sub-hitos 8.0 / 8.1 done with the remaining 8.2–8.7 items listed so readers can place this work in the larger plan. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Standardize the whole repo on English for comments, docstrings, and code
example queries so a reader on the project doesn't have to bounce between
languages. The change is text-only — no identifiers, behavior, or tests
move.
Translations:
- All `sub-hito 8.x` references in code → `milestone 8.x`
(pyproject.toml, cost_handlers.go, llm/__init__.py, graph/basic.py,
tools/cloudoracle.py).
- `internal/cloud/*_test.go` test docstrings and inline comments.
- `internal/report/pdf.go` section markers and field comments.
- Sample queries in `insights-agent/README.md` and the matching scripted
AIMessage / `ask(...)` query in `tests/test_graph.py` — the existing
asserts (`"$150"`, `"snapshots"`) still match the new English answer
so the test still passes.
Plus pre-existing working-tree formatting in `README.md` (single-line
badges, table padding in the v2 callout, an extra blockquote blank line,
and a Mermaid example query already updated to English) folded into the
same commit since it was already staged-adjacent and is the same kind
of language/cosmetic cleanup.
Verified after: `uv run pytest` (58/58, 91.83% coverage), `uv run ruff
check .`, `uv run mypy src/`, and `go test ./internal/cloud/... ./internal/api/...
./internal/report/...` all green.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…endpoint Milestone 8.2 (more tools): expose the rule-based analyzer findings as agent-friendly savings recommendations. Go: new authed GET /api/v1/recommendations handler that runs analyzer.Analyze over the current inventory, with optional provider/severity filters and a top cap. Totals (total_count, total_monthly_savings_usd, by_severity) describe the full filtered set before the cap. Carries data_source: "heuristic_rules" to distinguish heuristic estimates from the snapshot-derived cost endpoints. Python: CloudOracleClient.recommendations() + cloudoracle_recommendations tool with a rich docstring; system prompt updated to surface the heuristic_rules caveat. Validation errors map to ToolException so the ReAct loop can recover. Tests: 8 Go handler tests; extended Python tool tests. Both suites green (internal/api; 65 Python tests, 92% coverage, ruff + mypy clean). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Milestone 8.2 (more tools): answer "is my spend growing?" with a per-day cost time series. Go: new authed GET /api/v1/cost-trends handler over ListTrends(days). Returns the per-day series plus a precomputed first/latest/change summary (absolute_usd, percent_from_first, direction up/down/flat) so the agent phrases the trend without crunching the array. percent_from_first is null when the first day is zero. Optional provider filter recomputes each day's total from that day's per-service breakdown. days clamps to 1..365. Shares the snapshots_approximation data_source with the cost endpoints. Python: CloudOracleClient.cost_trends() + cloudoracle_cost_trends tool with a rich docstring steering trend/over-time questions here (vs cost_summary for a single period). Validation errors map to ToolException. Tests: 9 Go handler tests (delta/direction, provider recompute, days clamp, zero-first nil percent, flat, empty, auth, error); extended Python tool tests. Both suites green (internal/api; 71 Python tests, 93% coverage, ruff + mypy clean). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Milestone 8.2 complete (more tools): answer "what do I have?" with a resource inventory summary. Go: new authed GET /api/v1/inventory handler over ListResources, aggregating counts and projected monthly cost by provider and by (provider, service). Optional provider filter; top cap applies only to by_service so the totals stay accurate when the list is truncated. Because resources carry AccountID, the "functions" provider disambiguation (gcp vs azure) is exact here. Distinct data_source: live_inventory — costs are summed per-resource projected monthly rates from the latest scan, not billed spend. Python: CloudOracleClient.inventory() + cloudoracle_inventory tool with a docstring steering "what do I have?" / footprint questions here (vs cost_summary for spend over a range). Validation errors map to ToolException. Tests: 7 Go handler tests (aggregation, provider filter, top cap with accurate totals, functions disambiguation, auth, empty, error); extended Python tool tests. Both suites green (internal/api; 77 Python tests, 93% coverage, ruff + mypy clean). The agent now ships 5 tools across 5 authenticated v1 endpoints, closing milestone 8.2. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…ne 8.3) Add a sixth agent tool, finops_knowledge_search, that retrieves from a curated FinOps knowledge base for conceptual / policy / how-to questions the HTTP tools can't answer (rightsizing, commitment discounts, data-source caveats, cost allocation, glossary). Architecture (RAG kept in Python, where LangChain lives; the Go server stays a clean data API): - knowledge/: 5 packaged markdown notes, shipped in the wheel. - rag/corpus.py: load + chunk markdown to Documents (offline-testable). - rag/embeddings.py: EmbeddingsProvider ABC + Gemini impl, mirroring the llm/ provider pattern. - rag/store.py: langchain-postgres PGVector factory + store-agnostic retriever. - rag/ingest.py: ingest_corpus() core + insights-agent-ingest console script. - tools/knowledge.py: build_knowledge_tool(retriever) -> finops_knowledge_search, formatting results with [source: file — title] citations; errors map to ToolException so the ReAct loop can recover. Wiring is optional and gated on DATABASE_URL: with it unset the agent runs with just the five HTTP tools and no Postgres dependency. config.py gains database_url / embeddings_model / knowledge_collection / rag_top_k; main.py adds the knowledge tool only when a pgvector DB is configured; the system prompt steers conceptual questions to it. docker-compose switches Postgres to pgvector/pgvector:pg16 (drop-in). Tested fully offline (no DB, no embeddings API): corpus chunking, and the real retrieval + citation path via InMemoryVectorStore + DeterministicFakeEmbedding. 100 Python tests, 92% coverage, ruff + mypy clean. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…tone 8.4)
Replace create_react_agent on the production path with an explicit StateGraph:
START → supervisor → {worker} → supervisor → … → synthesize → END
- supervisor routes by tool call: bound with one routing tool per specialist
plus `finish`, the tool it calls names the next hop. Routing via tool calls
(not with_structured_output) keeps the node driveable by the scripted fake
model the suite already uses.
- three specialist workers, each a hand-rolled ReAct loop (_run_react, the
actual create_react_agent replacement) over a tool subset:
cost_analyst (cost-summary/by-service/trends/inventory),
savings_advisor (recommendations + knowledge),
concept_expert (knowledge). A worker contributes one summarizing message;
its tool churn stays local so the supervisor/synthesizer see a clean
transcript.
- synthesize composes the final answer from the findings, in the user's
language, with data-source caveats and citations.
- a hop cap bounds the supervisor loop so a model that never emits `finish`
still terminates.
main.py now builds the supervisor; graph/basic.py (create_react_agent) is
retained as the simple graph and still owns the shared AgentResult /
_stringify_content helpers the supervisor reuses.
Tests: test_supervisor.py drives it end-to-end with the scripted model —
single-worker route→tool→finish→synthesize, two-specialist routing, off-scope
finish-without-worker, hop cap, plus _run_react and _to_text units. 109 Python
tests, 93% coverage, ruff + mypy clean.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…llback (8.5) Three of milestone 8.5's production guardrails, wrapping every run through guardrails/runner.py:run_guarded (the single entry point the CLI and the upcoming HTTP surface share): - Cost/usage caps: RunLimits (max_hops, max_tool_calls, max_worker_iters) from settings, threaded into the supervisor graph and the worker ReAct loop. When a cap is hit the supervisor stops dispatching and synthesizes from what it has, so a confused or injected loop can't run up unbounded LLM/tool cost. Workers now also surface their tool observations through the graph state. - Layered answer validation (guardrails/validation.py): deterministic grounding first — every monetary figure in the answer must match a number in the tool observations, an unmatched figure is a hard fail; then an optional LLM judge for a second opinion when the answer makes numeric claims that pass the deterministic layer. - Deterministic fallback (guardrails/fallback.py): on a run exception (quota, timeout) or a failed validation, return an honest no-LLM answer rendering the raw tool data (or stating nothing was retrieved) instead of a fabricated narrative or a raw traceback. config gains MAX_HOPS / MAX_TOOL_CALLS / MAX_WORKER_ITERS / ENABLE_ANSWER_VALIDATION / ENABLE_LLM_JUDGE; main wires run_guarded and the --json output now includes fallback_used + the validation verdict. Tests cover figure extraction, grounding (pass/fail/tolerance), the judge layers, fallback rendering, and run_guarded (happy / exception / invalid / disabled). 131 Python tests, 93% coverage, ruff + mypy clean. HTTP surface follows next. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Expose the agent over HTTP, sharing one runtime with the CLI:
- runtime.py: GeminiAgentRunner assembles the model + client + tools + graph +
run limits once and exposes ask() through the guardrails. The CLI (main.py)
now uses it too, so the two entry points behave identically; the RAG
knowledge-tool builder moved here from main.
- api/app.py: FastAPI create_app with GET /health and POST /ask
({query} -> {answer, tool_calls, fallback_used, validation}). The stack is
built once in the lifespan; optional X-API-Key auth via AGENT_API_KEY (same
convention as the Go server). create_app(runner=...) injects a fake runner so
the surface is testable without Gemini / a live Go server / Postgres.
- api/serve.py: insights-agent-serve console script (uvicorn).
- config gains AGENT_HOST / AGENT_PORT / AGENT_API_KEY.
Tests drive the surface with FastAPI's TestClient and an injected fake runner:
health, ask happy path + metadata, empty-query 422, fallback passthrough,
auth enforced/open, and that an injected runner isn't closed by the app. 138
Python tests, 91% coverage, ruff + mypy clean.
Milestone 8.5 complete: cost caps + layered validation + deterministic fallback
+ HTTP surface.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…action (8.7) Replace the hard-wired snapshot cost path in the v1 endpoints with a billing.Source interface so a real billing integration can be swapped in by config, starting with AWS Cost Explorer. - internal/billing: CostRecord / Report / Source / SourceError, and CostExplorerSource — a GetCostAndUsage query grouped by SERVICE over the period (CE's exclusive end handled), summed across time buckets and pages, returning real unblended cost with data_source "billing_aws_cost_explorer". The CE client is narrowed to an injectable interface (mocked in tests), the same pattern internal/cloud uses for EC2/RDS. - internal/api: snapshotSource implements billing.Source over the existing cost_snapshots aggregation (preserves data_source "snapshots_approximation" and the snapshot_query_failed code exactly). The cost-summary / cost-by-service handlers now group normalized records and echo the report's dynamic data_source; the snapshot-specific aggregateByProvider/ByService helpers are gone. Server gains a WithBillingSource option (default snapshots). - config: CLOUDORACLE_BILLING_PROVIDER (snapshots | aws_cost_explorer). cmd builds the CE source from AWS_REGION/AWS_PROFILE when selected and falls back to snapshots (loudly) if init fails. Tests: CE source (bucket/page summation, exclusive-end TimePeriod, error wrapping, missing-metric skip) with a fake client; api handlers against an injected non-snapshot source (dynamic data_source, provider filter, error code). Existing snapshot cost tests pass unchanged. The agent's FinOps corpus documents the new real-billing data source. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The internal/diff golden tests (byte-exact Markdown/narrative fixtures) failed on Windows checkouts: core.autocrlf=true with no .gitattributes rewrote the fixtures to CRLF while the renderer emits "\n". The committed fixture content was already correct, so this adds `* text=auto eol=lf` (plus binary markers) to keep LF in the working tree on every platform. All Go tests now pass. Also adds docs/v3-guide.md — the Insights Agent guide (supervisor + RAG + guardrails architecture, the /api/v1 contract and data_source semantics, real billing via AWS Cost Explorer, CLI/HTTP usage) alongside v1/v2-guide — and links it from the README.
Rename the "AI Insights Agent" heading to "v3 — Insights Agent (current focus)" so it's unambiguously v3 and parallel to the v1/v2 sections, move the "current focus" marker from v2 to v3, and drop "(in progress)" from the v3 roadmap entry now that the v3 milestones are complete.
An end-to-end run surfaced this: the cost_analyst worker produced the full
answer ("AWS spend was 8662.07 USD ...") but the final answer was just a
trailing caveat ("Your final bill may differ."). The synthesizer was fed the
workers' contributions as prior *AIMessages*, so the model treated the answer
as already given and only appended a short follow-up, dropping the numbers.
Fix: _synthesis_input collects the user question plus the specialists' findings
into a single human turn the model answers fresh, instead of replaying worker
AIMessages. Adds offline regression tests (unit on _synthesis_input, plus a
graph-level assertion that the synthesizer's input is a human turn carrying the
finding, not a replayed assistant turn).
…nknown behavior The claudeRequest.MaxTokens field used the JSON tag "maxTokens", but the Anthropic Messages API expects snake_case "max_tokens". As written the field was silently ignored and Claude fell back to its own default instead of the configured 1024 cap. Also align README.md and docs/architecture.md with the actual parser behavior: the iac decoders do not parse an after_unknown field. Unknown-until-apply attributes arrive as JSON null and are treated as missing; a missing required attribute routes the resource to Skipped.
Adds a billing.Source that reads real net cost (list cost plus credits) from the standard GCP billing export in BigQuery, grouped by service, as the GCP counterpart to the AWS Cost Explorer source. Selectable via CLOUDORACLE_BILLING_PROVIDER=gcp_bigquery; reports data_source billing_gcp_bigquery. - internal/billing/bigquery.go: BigQuerySource with a narrow bigQueryAPI seam (returns parsed rows) so tests run offline; real client wraps *bigquery.Client via ADC. - config: gcp_bigquery enum + CLOUDORACLE_GCP_BILLING_DATASET/_TABLE, with a cross-field check requiring project+dataset+table. - main: billing wiring switch gains the GCP case with loud snapshot fallback on init failure. - docs + agent knowledge + README roadmap updated.
Extends the pr-check cost engine to GCP, starting with
google_compute_instance. GCP has no attribute-queryable pricing API like
AWS's, so compute is priced from a curated static table embedded at build
time (us-central1 base rates + per-region multipliers + PD $/GB-month),
which is deterministic and testable offline. Estimates cap at Medium
confidence with a "static price table" caveat.
- internal/iac/gcp: extractor package mirroring internal/iac/aws.
ExtractComputeInstance reads machine_type (short name or self-link URL),
zone→region, scheduling (preemptible / provisioning_model=SPOT), and
boot_disk.initialize_params size/type.
- internal/pricing/gcp_prices.{json,go}: embedded price table + lookups.
- internal/pricing/gcp_compute.go: EstimateGCPComputeInstance (compute +
boot disk). Region multiplier and preemptible/Spot drop confidence to
Low; Spot is priced at on-demand as a labeled upper bound.
- internal/pricing/change.go: estimateState routes google_* types through
the GCP path (no AWS Pricing API src). Unpriced machine types become a
Skipped change, not a hard error.
Verified end-to-end: `oracle pr-check` on a mixed GCP plan renders the
comment with correct per-resource breakdowns, region multiplier, Spot
caveat, and skips for unsupported types.
Adds standalone persistent-disk pricing to the GCP path, reusing the embedded PD $/GB-month table. Priced at the base rate (storage region variation is not modeled); Medium confidence with the static-table caveat. - iac/gcp: ExtractComputeDisk (type + size) wired into Extract / SupportedTypes. - pricing/gcp_disk.go: EstimateGCPComputeDisk; defaults type to pd-standard. Unknown type or a size the plan omits (image/snapshot-sized disks) becomes a Skipped change via errUnpricedGCPDisk, not a hard error. - change.go: dispatch the ComputeDisk arm.
…(GCP v2) Adds Cloud SQL pricing (compute + storage) from the embedded static rates. Custom tiers (db-custom-V-M) are priced per vCPU-hour + per GB-RAM-hour; shared-core (db-f1-micro, db-g1-small) are flat; legacy db-n1-* tiers parse to their vCPU/RAM ratios. REGIONAL (HA) availability doubles compute and storage. Priced at US rates (region variation not modeled), Medium confidence with the static-table caveat. SQL Server bundles licensing into its per-vCPU rate that we don't model, so SQLSERVER_* versions are Skipped with a clear reason rather than mis-priced at PostgreSQL rates. Unknown/absent tiers and unknown disk types also Skip. - iac/gcp: ExtractSQLInstance (database_version + settings.tier/disk_size/ disk_type/availability_type) wired into Extract / SupportedTypes. - pricing/gcp_sql.go: EstimateGCPSQLInstance + parseSQLTier. - gcp_prices.json: cloudsql rate block. - change.go: dispatch the SQLInstance arm. Verified end-to-end: db-custom-4-16384 REGIONAL + 100GB SSD renders as $438.71 (compute $404.71, storage $34.00) with the HA caveat; a SQL Server instance in the same plan is skipped.
- v2-guide: split Supported resources into AWS (live Pricing API) and GCP (embedded static table); new "How GCP is priced" subsection explaining the static-table decision, the drift caveat, and how --region works for GCP (zone-derived region overrides the flag). Also generalize the --region flag/input descriptions beyond AWS, and fix a stale reference (estimator.go → change.go). - README: v2 blurb and roadmap note GCP pricing (compute instance, disk, Cloud SQL) via the static table.
feat(billing): GCP BigQuery billing-export cost source
feat(pricing): GCP v2 — price google_compute_instance in Terraform plans
Switch the workflow examples from @v2.0.0 to @v2 so consumers track the v2 major line and pick up minor releases (e.g. GCP pricing in v2.1.0) without editing their workflow.
- Test badge 469→581 unit, 21→22 integration (func Test count after the GCP pricing + BigQuery billing tests). - Tech stack: AWS SDK adds Pricing + Cost Explorer; GCP SDK adds BigQuery.
main and feature/3.0 carried the same v3 work under different SHAs (v3 was merged to main via #23-#26 as fresh commits), so a plain merge conflicted on 13 files. feature/3.0 is a verified content superset of main (a -X theirs merge of feature/3.0 into main produces a tree identical to feature/3.0), so conflicts resolve in favor of feature/3.0. This merge records main as a parent so PR #32 (feature/3.0 -> main) merges without conflicts; the tree is unchanged from feature/3.0.
The e2e-test/ Terraform config was removed (72128fe) but cost-self-test.yml was left running `terraform init` in the now-missing e2e-test/ directory, so the cost-impact check failed before it ever built the Action. Replace the live-Terraform + AWS-credentials flow with a committed GCP plan fixture (e2e-test/plan.json). GCP resources price from the embedded static table, so the self-test now needs no cloud credentials and no Terraform — it just builds the Docker Action from the checkout (`uses: ./`) and runs `oracle pr-check --no-llm` against the fixture, exercising the compute / disk / Cloud SQL estimators and the skipped-unsupported-type path end-to-end (and dogfooding the new GCP pricing on this very PR).
💰 Cloud Cost ImpactNet monthly change: +$630.11 🔴 This plan adds 4 resources, with a net monthly cost increase of +$630.11. Top movers by cost impact
📋 Full breakdown (4 priced, 1 skipped)Created (4)
Skipped (1)
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.