fix(ingestion): give the enrichment repair function a caller, a reach, and monitoring - #2548
fix(ingestion): give the enrichment repair function a caller, a reach, and monitoring#2548BigSimmo wants to merge 9 commits into
Conversation
…, and monitoring #W98GR7 recorded four claims. Two hold, one holds by a different mechanism than recorded, and the central technical premise is refuted by the code on main. REFUTED: "deletes an artifact family BEFORE calling OpenAI and re-inserting, never staged-then-swapped". In all four writers in supabase/functions/indexing-v3-agent/index.ts — upsertMemoryCardsFromSections, upsertSectionIndexUnits, upsertVisualArtifacts, upsertCoreEmbeddingFields — the embeddingBatch await completes before sql.begin is entered, and the delete and re-insert share one Postgres transaction. A provider outage aborts before any delete happens, and a failed insert rolls the delete back. So no ordering change is made here; the ledger row is corrected instead. The existing test did NOT pin that, and the first version of this change claimed it did. tests/indexing-v3-agent.test.ts asserted delete-before-insert INSIDE the transaction, which would stay green if the embedding call moved inside sql.begin — precisely the arrangement the issue describes. A new assertion pins the ordering the refutation actually rests on, including that no second embeddingBatch call appears inside the transaction. CONFIRMED, and fixed: repair_strict_enrichment_gate_batch is invoked by nothing. It now has an operator caller — dry run by default, mutating only under --apply plus a confirmation, health-probed first, following scripts/cleanup-abandoned-reindex-generations.ts. Deliberately not wired to a worker loop, a scheduled workflow, or an admin route. CONFIRMED with the mechanism corrected: both terminal states are excluded from claim eligibility forever, but by two different routes. claim_indexing_v3_agent_jobs excludes needs_enrichment_artifacts by name, while 'failed' is excluded through `attempt_count < max_attempts`. NEW, and the reason wiring up a caller alone would not have worked: the repair function touches documents.metadata, document_index_quality and ingestion_jobs and never touches indexing_v3_agent_jobs — the table the claim RPC reads. It could not have unstuck a stuck document however often it ran. 20260902120000 adds that reset. Two things about that migration are worth stating, because the first version of it got both wrong. Its baseline is the body currently DEPLOYED — the one codified by 20260712171500 and mirrored at schema.sql's later copy — not the original 20260625033425 body that schema.sql still carries earlier in the file as a superseded copy. Those differ by more than whitespace: the original declares v_processing_lock_timeout and preserves a fresh-locked processing row's metadata, the deployed one does neither. Rebuilding from the wrong copy shipped that as an undeclared second behaviour change and would have moved a def_hash 20260819110500 pins for reasons unrelated to this issue. And the reset needs its own lease guard, because the deployed body has none. gate_passed is a structural fact about which artifacts exist, and request_indexing_v3_enrichment re-queues a document without clearing them, so it stays true for the whole of a new run. Without the guard the reset would match a document mid-run and clear status, locked_by and locked_at underneath it, and request_ingestion_reindex_if_agent_idle — which decides "is the agent active?" from exactly those columns — would then approve a concurrent reindex over the same artifact tables. CONFIRMED, and fixed: no monitoring. check:enrichment-health counts the three permanently-excluded states and the gate-failing documents. It reads indexing_v3_agent_jobs itself rather than the metadata mirror: a divergence between recorded state and reality is the whole subject of this issue. It reads the live database, so it is confirmation-gated and is in no CI job. The dry run now reruns the function's own candidate predicate rather than approximating it. Counting gate-failing indexed documents was a different set in both directions — a gate-passing document with disagreeing recorded state IS a candidate, a gate-failing one already recorded as pending is NOT — and a preview that says "0 failing" while apply repairs 50 is worse than no preview. It also now says what apply costs: each gate-failing document is queued for a full re-ingestion with OpenAI embedding and caption calls, and the script refuses to apply against any project ref but the expected one. Decision logic lives in src/lib/enrichment-repair.ts so it is unit-testable without a database; the scripts own only the provider I/O. The cross-route race in docs/ingestion-state-machine.md R24d — deep-memory.ts deleting the same artifact families unscoped — is a real defect on a different code path, and is queued rather than folded in here. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01N8JUuufwbFHy8PsgbztmGN
|
Important
This repository does not receive automatic reviews because it has fewer than 10 stars. ⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: CHILL Plan: Team Run ID: Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
|
Updates to Preview Branch (claude/enrichment-staging) ↗︎
Tasks are run on every commit but only new migration files are pushed.
View logs for this Workflow Run ↗︎. |
Bugbot couldn't run - usage limit reachedBugbot is counted against Cursor usage for this user or team, and this run hit a usage or spend limit. A user or team admin can review and increase usage limits in the Cursor dashboard. (requestId: serverGenReqId_39f48354-1a82-4972-895a-e318bf438bfc) |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 769cd00e1d
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| ) | ||
| .eq("document_status", "indexed") | ||
| .order("document_updated_at", { ascending: true, nullsFirst: true }) | ||
| .limit(500); |
There was a problem hiding this comment.
Filter the full corpus before limiting the preview
When the 500 oldest indexed documents are healthy but a later document is a repair candidate, this limit removes that candidate before selectStrictGateRepairCandidates runs, while the RPC filters the entire corpus before applying its limit. The confirmation can therefore report zero gate-failing candidates and then queue an unpreviewed full re-ingestion with provider spend and clinical-document transfer. Apply the candidate predicate server-side before limiting; a fixture with 500 healthy rows followed by one failing row proves the mismatch.
Useful? React with 👍 / 👎.
| from candidates c | ||
| where c.gate_passed | ||
| and j.document_id = c.document_id | ||
| and j.status in ('pending', 'processing') |
There was a problem hiding this comment.
Exclude live ingestion jobs from repair
During an atomic reindex, the old artifact generation deliberately remains live, so gate_passed can be true while an ingestion job is actively processing. This predicate admits that live job, and completed_open_jobs then marks it completed and clears its lease underneath the running worker, allowing the worker to continue mutating artifacts after it has lost ownership. Preserve fresh processing jobs or require a stale lease; a database test with a gate-passing document and a freshly locked processing job should assert that the repair leaves it unchanged.
Useful? React with 👍 / 👎.
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
Resolves the one conflict, in the generated data/repo-awareness-snapshot.json, by regenerating it with npm run snapshot:repo-awareness rather than by hand. check:repo-awareness-snapshot confirms it is in step (204 pages, 575 documents, 2664 reviews). Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01N8JUuufwbFHy8PsgbztmGN
Both found in review on PR #2548. repair_strict_enrichment_gate_batch cleared ingestion_jobs lease columns with no lease-age guard on completed_open_jobs or deferred_open_jobs. During an atomic reindex the old artifact generation stays live, so gate_passed can be true while a job is actively processing; marking it completed and clearing locked_at/locked_by let the worker keep mutating artifacts after losing ownership. The new reset_agent_jobs CTE already carried a 45-minute guard -- the same predicate now covers all three CTEs that write a lock column. The function had no caller before this branch, which is why the hazard was latent; adding one is what makes it reachable, so the fix belongs here. The operator script's dry run read the 500 oldest indexed documents and filtered locally, while apply calls the RPC, which filters the whole corpus and only then limits. A corpus whose oldest page was healthy previewed as zero candidates and then queued real re-ingestions -- OpenAI spend against live clinical documents -- that the operator never saw. The preview now pages the view and stops once the limit is met. The predicate deliberately stays in selectStrictGateRepairCandidates rather than moving into PostgREST filters: neq drops NULLs while the TS predicate reads NULL as not-completed, and that difference undercounts. Tests pin both: the paging contract and the single-predicate rule as source assertions, plus a fixture whose only candidate sits past the old 500-row window. Verified red against the pre-fix script. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01N8JUuufwbFHy8PsgbztmGN
CI triageCI failed on this PR. Automated classification of the 3 failed job(s):
Compared with main CI run #15123 (failure). That run's conclusion is an aggregate and did not exercise Classification is evidence routing, not permission to ignore a failure. Exact quarantined Playwright identities remain governed by the flake ledger. |
…ging # Conflicts: # data/outstanding-issues-snapshot.json # data/repo-awareness-snapshot.json
Bugbot couldn't run - usage limit reachedBugbot is counted against Cursor usage for this user or team, and this run hit a usage or spend limit. A user or team admin can review and increase usage limits in the Cursor dashboard. (requestId: serverGenReqId_cadfb40d-18e6-43d7-821c-e5099224ef81) |
Merge conflict resolved; one red check remains and needs a hand off this branchThe conflict is fixed. This branch went Head What will still fail: Why I have not fixed it: The fix: take the Not for merge. Labelled Generated by Claude Code |
…ging # Conflicts: # data/outstanding-issues-snapshot.json # data/repo-awareness-snapshot.json # docs/scripts-index.md
…ging # Conflicts: # data/repo-awareness-snapshot.json
Bugbot couldn't run - usage limit reachedBugbot is counted against Cursor usage for this user or team, and this run hit a usage or spend limit. A user or team admin can review and increase usage limits in the Cursor dashboard. (requestId: serverGenReqId_f8a52a3c-d2e7-470a-9845-0f2824e30307) |
…ging # Conflicts: # data/repo-awareness-snapshot.json # docs/scripts-index.md
Bugbot couldn't run - usage limit reachedBugbot is counted against Cursor usage for this user or team, and this run hit a usage or spend limit. A user or team admin can review and increase usage limits in the Cursor dashboard. (requestId: serverGenReqId_f69d846d-615a-4536-b9d4-19241724f644) |
…ging # Conflicts: # data/repo-awareness-snapshot.json # docs/scripts-index.md
Bugbot couldn't run - usage limit reachedBugbot is counted against Cursor usage for this user or team, and this run hit a usage or spend limit. A user or team admin can review and increase usage limits in the Cursor dashboard. (requestId: serverGenReqId_4f088264-b497-4edc-8cea-7418065ff957) |
…ging # Conflicts: # data/repo-awareness-snapshot.json # docs/scripts-index.md
Bugbot couldn't run - usage limit reachedBugbot is counted against Cursor usage for this user or team, and this run hit a usage or spend limit. A user or team admin can review and increase usage limits in the Cursor dashboard. (requestId: serverGenReqId_ee79e040-01bf-400b-a944-01263d17a0c4) |
Summary
#W98GR7recorded four claims about the enrichment pipeline. Two hold, one holds by a different mechanism than recorded, and the central technical premise is refuted by the code onmain. This PR fixes what is real and corrects the record on what is not.Refuted, and why no ordering change is made
The issue says
supabase/functions/indexing-v3-agent"deletes an artifact family before calling OpenAI and re-inserting, one family at a time, never staged-then-swapped", so a provider outage leaves the family permanently empty. Onmainit does the opposite. In all four writers —upsertMemoryCardsFromSections,upsertSectionIndexUnits,upsertVisualArtifacts,upsertCoreEmbeddingFields— theembeddingBatchawait completes beforesql.beginis entered, and the delete and re-insert share one Postgres transaction. A provider outage therefore throws before any delete happens, and a failed insert rolls the delete back.tests/indexing-v3-agent.test.tsdid not actually pin that ordering — it pinned delete-before-insert inside the transaction, which is a weaker and different claim — so this PR adds the assertion that does. See "Three things review corrected" below.The Edge Function is not modified here. The ledger row is corrected instead.
One caveat stated rather than buried: this checkout is shallow and the audit commits the issue cites are not reachable, so whether the file was fixed since the audit or the claim was inaccurate from the outset is not established. What is established is what the code does now.
Confirmed, and fixed
The repair function was invoked by nothing. Every repo-wide hit for
repair_strict_enrichment_gate_batchis the migration, the schema mirror, generated types, drift bookkeeping, docs, or a schema-text assertion. It now has an operator caller.And wiring up a caller alone would not have worked — this is the finding that came out of building it.
repair_strict_enrichment_gate_batchtouchesdocuments.metadata,document_index_qualityandingestion_jobs, and never touchesindexing_v3_agent_jobs, which is the tableclaim_indexing_v3_agent_jobsactually reads. It could not have unstuck a stuck document however often it ran.20260902120000adds that reset: completed where the gate passes, back to claimable with a fresh attempt budget where it does not, each repair stamping a counter so a document repaired again and again is visible rather than looping silently.Terminal states are excluded from claim eligibility forever — by two different mechanisms, not the one recorded.
claim_indexing_v3_agent_jobsexcludesneeds_enrichment_artifactsby name in itsstatus not in (...)filter, whilefailedis excluded throughattempt_count < max_attempts, becauseagentFailureDecisiononly writesfailedonce attempts are spent. A monitor that looked only at the status name would under-report.No monitoring existed.
needs_enrichment_artifactsappeared in no script and no workflow, so a stuck document reported asindexedwith an empty artifact family and nothing counted it — silent corruption, not a crash.check:enrichment-healthcounts the three permanently-excluded states plus gate-failing documents. It readsindexing_v3_agent_jobsitself rather than theindexing_v3_agent_statusmirror ondocuments.metadata: a divergence between recorded state and reality is the whole subject of this issue, so the check reads the table the claim RPC reads.Design notes
The operator script follows
scripts/cleanup-abandoned-reindex-generations.ts, the repo's established shape for a repair RPC that touches live clinical rows: dry run by default, mutating only under--applyplus a confirmation, health-probed first. It is deliberately not wired to a worker loop, a scheduled workflow, or an admin route.cli-utils.ts'sconfirm()returnsfalseon a non-TTY, so--applywithout--yescannot mutate in a piped or detached context.check:enrichment-healthis provider-backed and is deliberately absent fromverify:cheap,verify:pr-localand every CI job, so it cannot fire unattended.Decision logic lives in
src/lib/enrichment-repair.tsso it is unit-testable without a database; the scripts own only the provider I/O.Three things review corrected, recorded because the first version of this branch shipped all three
The migration was built from the wrong baseline.
schema.sqlcarriesrepair_strict_enrichment_gate_batchtwice: the original20260625033425body, and the later20260712171500"codify live ahead" body which is the one that wins on replay and is what is actually deployed. They differ by more than whitespace — the original declaresv_processing_lock_timeoutand preserves a fresh-locked processing row's metadata; the deployed one does neither. Rebuilding from the superseded copy would have shipped that as an undeclared second behaviour change and moved adef_hashthat20260819110500pins, for reasons unrelated to this issue. The migration is now a true minimal delta on the deployed body, andsupabase/schema.sqlcarries a byte-identical copy — verified programmatically, not by eye.The reset needed its own lease guard.
gate_passedis a structural fact about which artifacts exist, andrequest_indexing_v3_enrichmentre-queues a document without clearing them, so it stays true for the whole of a new run. Without a guard the reset would match a document mid-run and clearstatus,locked_byandlocked_atunderneath it — andrequest_ingestion_reindex_if_agent_idle, which decides "is the agent active?" from exactly those columns, would then approve a concurrent reindex over the same artifact tables. The deployed body has no lease-age guard anywhere, so the new CTE carries its own.The refutation's premise was not actually pinned by a test.
tests/indexing-v3-agent.test.tsasserted delete-before-insert inside the transaction — which would stay green if the embedding call were moved insidesql.begin, precisely the arrangement the issue describes. A new assertion pins the ordering the refutation rests on:embeddingBatchcompletes beforesql.beginis entered, and no second embedding call appears inside the transaction.Two further review findings applied: the dry run now reruns the function's own candidate predicate instead of counting gate-failing indexed documents (a different set in both directions — a gate-passing document with disagreeing recorded state is a candidate, and a gate-failing one already recorded as
pendingis not — so a preview reporting "0 failing" while apply repaired 50 was possible), and both the preview and the confirmation prompt now say what--applyactually costs. The script also refuses to apply against any project ref butsjrfecxgysukkwxsowpy.What
--applyactually does, stated plainlyFor a gate-failing document the repair queues a pending
ingestion_jobsrow, andworker/main.tsignores the incoming stage — so that is a full re-ingestion: download, extract or OCR, chunk, OpenAI embeddings, image captioning. Real provider spend, and a cross-border transfer of clinical document text. It takes the atomic reindex path (isAtomicReindexCandidateisstatus === "indexed", which every candidate is), so the old generation stays live until the new one commits and no document is unsearchable in between. That belongs in the operator's line of sight, and it is now in both the dry-run output and the confirmation prompt.Verification
npm run typecheck— clean.npm run lint— clean.npm run check:migration-role— passed.npm run check:function-grants—OK — all 36 SECURITY DEFINER public function(s) are revoked from PUBLIC ... and none are re-opened by a grant to PUBLIC/anon.npm run docs:check-scripts—1193 npm-run reference(s) resolve to real scripts.npm run docs:check-index—all 64 repository roots/modules/routes and all schema tables are indexed.npm run docs:update— inventory regenerated and committed (285 script files, 286 npm scripts).tests/enrichment-repair.test.tsandtests/indexing-v3-agent.test.ts—Test Files 2 passed (2)/Tests 32 passed (32).Test Files 1 failed | 947 passed (948). The single failure istests/drift-detection.test.ts, the known drift-manifest staleness described below; nothing else is red.npm run format— run, and the result is committed.ingestion-worker-reviewerandclinical-governance-reviewersubagents on this exact diff.On the honesty of the Edge Function's test coverage, since this PR relies on it for the refutation:
tests/indexing-v3-agent.test.tsis two different things. Real unit tests of the pure logic extracted intobehavior.ts—agentFailureDecision,deferralDecision,completionGateFromRowand siblings — which execute and assert real return values. And static source-text assertions againstindex.tsthat slice the raw file and do string containment and ordering checks; they do not execute it and cannot, because it is Deno-runtime code that Vitest cannot load. There are no Deno tests anywhere undersupabase/functions. So the ordering claim rests on reading the source plus a static pin, not on behavioural coverage, and it would be wrong to imply otherwise. The new assertion is the same kind — a stronger static pin, not behavioural proof.Verification not run:
npm run check:enrichment-healthandnpm run repair:enrichment-gateare the new scripts and both read or write the live clinical database — neither was run.npm run check:supabase-project,check:driftandcheck:migration-historyare provider-backed and were not run.supabase migration up --localcould not be run: the Supabase CLI is not installed here, and CI'sMigration replayjob is skipped on draft pull requests. No statement in20260902120000has been executed or parsed anywhere, because there is no Docker daemon and no Postgres in this environment.tests/drift-detection.test.tsis red on this branch for the same reason as its sibling:supabase/schema.sqlchanged andsupabase/drift-manifest.jsoncannot be regenerated without Docker. It was not hand-edited to make the check pass.pr-policyclassifies this diffclinicalRisk: true, operationalRisk: true, the latter becausepackage.jsongains two script entries. It emits the advisory note that operational-risk changes are bundled with clinical ones; splitting a two-line npm-script registration into its own PR would leave the scripts it registers unreachable, so they are kept together deliberately.Risk and rollout
20260712171500's), verified programmatically. Applying it also changes the function'sdef_hash, whichsupabase/drift-manifest.jsoncurrently pins to the old value — regenerating the manifest is what keeps the post-mergelive-driftrun clean, and is step 2 below. The function isSECURITY INVOKER, granted toservice_roleonly, and is invoked by nothing automatically — so applying the migration changes no behaviour on its own. The behaviour only changes when an operator runs the repair script deliberately.20260902120000to the live clinical database within seconds — the Supabase GitHub integration has "Deploy to production" enabled with production branchmain. Merge only inside a window the owner has approved. The two new scripts make provider calls only when an operator invokes them.On whether merging deploys Edge Functions in this repository — the question asked before this work started. Migrations deploy automatically on merge; Edge Functions appear not to, but that is not provable from the repository. No workflow contains
supabase functions deploy(all eleven Supabase-mentioning workflows checked);docs/db-maintenance.mdanddocs/disaster-recovery-runbook.mdboth give the function deploy as an explicit operator CLI command inside an approved change window;docs/process-hardening.mdrecords a specific past operator-run function deploy by version number; andAGENTS.mddocuments the migration auto-deploy toggle in detail while saying nothing about functions — which is exactly where that fact would live. The residual ambiguity: the Supabase GitHub integration has a separate function-deploy toggle, and nothing committed proves its state. A dashboard read of that setting would settle it, the same checkAGENTS.mdrecords was done for the migration toggle on 2026-08-21. It does not gate this PR, since the Edge Function is not modified — but it must be confirmed before anyone assumes a merged function change is live.Clinical Governance Preflight
Clinical KB Database(sjrfecxgysukkwxsowpy)The reasoning behind each. Nothing here changes what is cited, how sources are rendered, or how an answer is verified; the diff is ingestion repair tooling. No document workflow is introduced — the repair returns an existing stuck document to the enrichment path it was already meant to take, and no new content is ingested. No Supabase environment value is edited, so
check:supabase-projectwas deliberately not run. Both new scripts are server-only Node entry points usingcreateAdminClient(); neither is reachable from a route or the browser, and errors go throughsafeErrorLogDetails. Demo mode is untouched. Item 6 is what the change is about: a document with an incomplete artifact family currently reports as fully indexed, and the monitoring makes that conservative rather than silent. On SaMD: no clinical decision-support behaviour changes — a repaired document is re-enriched by the existing pipeline through the existing gates, and nothing bypasses the strict enrichment gate.Notes
Do not merge this PR. It is a draft and is labelled
holdanddo-not-merge. Auto-merge is not enabled and must not be.Approval still needed, and from whom — Josh, the repository owner:
20260902120000. Merging is the apply step; there is no separate one. Merge only inside a window he has approved.npm run drift:manifeston a machine with Docker, committed, before this leaves draft.check:enrichment-healthis a read;repair:enrichment-gate --applywrites to live clinical document rows and re-queues enrichment work that will make OpenAI calls.The cross-route race in
docs/ingestion-state-machine.mdR24d —deep-memory.tsdeleting the same artifact families unscoped, racing the route that writes them — is a real defect on a different code path, and is very likely the defect this issue's ordering claim was reaching for. It is queued rather than folded in here.After this lands,
npm run issues:reconcileapplies the two queued inbox records to the canonical ledger from its own serialized branch.🤖 Generated with Claude Code
https://claude.ai/code/session_01N8JUuufwbFHy8PsgbztmGN
Generated by Claude Code
Note
High Risk
Changes a live-database repair function (auto-deployed on merge) and adds operator tools that can queue full re-ingestion with OpenAI spend on clinical documents; migration alters ingestion job lease behavior when repair is invoked.
Overview
Addresses #W98GR7 by wiring up the previously unused
repair_strict_enrichment_gate_batchpath and closing gaps where documents could stay permanently unclaimable byindexing-v3-agent.Migration
20260902120000extends the repair RPC with areset_agent_jobsCTE that updatesindexing_v3_agent_jobs(not only metadata/quality/ingestion jobs), adds 45-minute lease guards on ingestion and agent job mutations so repair cannot steal active locks, and surfacesagent_job_resetin the repaired array.supabase/schema.sqlis updated to match.New
src/lib/enrichment-repair.tsholds testable candidate selection, health verdicts, and repair output formatting. Operator scriptsrepair:enrichment-gate(dry-run by default,--applywith confirmation and project ref guard) and provider-backedcheck:enrichment-health(counts stuck agent states and gate-failing indexed docs) are registered inpackage.json.Tests pin repair semantics (
tests/enrichment-repair.test.ts) and add a static assertion thatembeddingBatchruns beforesql.beginin the four indexing-v3-agent upsert paths, documenting that the issue’s “delete-before-OpenAI” premise does not match current code. Ledger/docs snapshots and outstanding-issue inbox records are updated; a separate deep-memory.ts unscoped-delete race is queued, not fixed here.Reviewed by Cursor Bugbot for commit e807b80. Configure here.