feat(api): library import onboarding — POST /libraries/:id/import (O-8)

[thewrz]

Why

The onboarding entry point for the editability program (O-8 of the onboarding &
editability design): one endpoint that takes a master document and produces a
reviewable library master. Headless-first (ADR-002) so curl/agents/integration
tests can drive the whole flow before any UI exists. Extends the surface of
WT-3's POST /templates/import (consensus style derivation) with persistence
into an ADR-015 library + editability classification.

What

New POST /libraries/:id/import — multipart upload (DOCX/SEC/TXT), async via the
existing lib/jobs.ts pattern. It orchestrates already-tested building blocks:

1. Parse (existing 5-signal pipeline, off-thread via parsePool) with
   per-paragraph source-fact capture.
2. Persist the spec into the target library (persistParsedSpec now
   accepts an explicit libraryId that overrides source-derived resolution).
3. Derive style (DOCX only) per WT-3 consensus → a style_templates row
   linked to the spec. Non-DOCX → the report flags styleSourceNeeded: true
   (assign later via O-12 / ADR-015 inherit), it does not fail.
4. Classify editability against the library's convention profile or the
   built-in Industry Default (reuses reclassifySpec).
5. Job result = onboarding report: style derivation report (WT-3) +
   editability summary (per-value counts + low-confidence review list) + parse
   warnings.

Source bytes are discarded after parse (ADR-021); only origin_meta
(sha256 + sanitized filename) is recorded. openapi.yaml gains the path and the
OnboardingJob/OnboardingReport/EditabilitySummary schemas so the CI
contract gate stays green.

Poll progress at GET /libraries/import/jobs/:jobId (a 3-segment literal route
that can never shadow the 2-segment /libraries/:id/... routes).

Out of scope (per the issue): onboarding_status finalize/reopen (O-11),
corrections/reclassify API (O-9, already merged separately), MCP parity (O-13).

Design decisions

Ambiguous calls I made and documented, so the reviewer can flag any wrong turn:

• New src/api/onboarding.ts instead of growing libraries.ts. The pipeline
  is substantial and distinct; libraries.ts stays focused and both stay under
  the 400-line cap. Mirrors parse.ts being separate from specs.ts.
• Separate onboarding job store in lib/jobs.ts (own Map + result type)
  rather than reusing ParseJobResult. The three-section onboarding report does
  not fit the parse contract, and /parse/jobs must keep its existing schema.
• reclassifySpec(specId, {}) for the classify step — it already resolves
  the library profile (or built-in default) and stores classifications, so the
  endpoint stays pure orchestration with no duplicated classify logic.
• Style derivation reuses the WT-3 trio verbatim (analyzeDocxStyles →
  deriveTemplate → createTemplateWithRules → setSpecStyleSource). A
  duplicate template name on re-import (pg 23505) is treated as non-fatal — the
  derivation report still surfaces; any other DB error fails the job loudly.
• Low-confidence threshold = 0.6 for the editability review list. A
  reasonable default; trivially adjustable (summarizeEditability takes it as a
  parameter).
• Both new ops added to the contract test RESPONSE_ALLOWLIST, mirroring how
  post /parse + get /parse/jobs/{} are handled — behavior is asserted in the
  dedicated onboarding.integration.test.ts, not the structural contract file.

Testing

• Unit tests pass (pnpm test → 1005 passing with DATABASE_URL set, as CI provides)
• Integration tests pass (pnpm test:integration — onboarding/contract/parse/libraries/specs all green)
• Manual verification: DOCX fixture → 18 paragraphs, 3 style rules derived, template linked, classifications persisted; .sec fixture → 53 paragraphs, styleSourceNeeded: true, no template; unknown library → 404; bad extension / bad library id → 400
• CI green

Note: docs.integration.test.ts fails only in the isolated worktree (the
gitignored public/scalar/standalone.js Scalar asset + sendFile CWD
resolution); it passes 3/3 in a normal clone and in CI. Confirmed unrelated to
this change.

🤖 Co-authored by Claude Opus 4.8. Closes #135.

Summary by CodeRabbit

Release Notes

• New Features
  • Added async library import workflow supporting DOCX and SEC master documents
  • Introduced job polling to track and retrieve import status and reports
  • Automatic style template derivation for Word documents
  • Content editability classification in onboarding reports

[coderabbitai]

Warning

Review limit reached

@thewrz, we couldn't start this review because you've reached your PR review rate limit.

More reviews will be available in 38 minutes and 42 seconds. Learn how PR review limits work.

Your organization has run out of usage credits. Purchase more credits in the billing tab to continue.

⌛ How to resolve this issue?

After more reviews become available, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based credits.

🚦 How do rate limits work?

CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan refill rate.

For paid Pro and Pro+ PR reviews, CodeRabbit uses rolling per-developer review limits. Reviews become available again as older review attempts age out of the rolling limit window.

Please see our Fair Usage Limits Policy for further information.

ℹ️ Review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro Plus

Run ID: 36b4d6ed-dd65-412f-b07a-160d39366f80

📥 Commits

Reviewing files that changed from the base of the PR and between c84c495 and ebcfad7.

📒 Files selected for processing (8)

• src/api/onboarding-report.ts
• src/api/onboarding.test.ts
• src/api/onboarding.ts
• src/api/parse.ts
• src/ast/index.ts
• src/lib/jobs.ts
• src/lib/parse-worker.test.ts
• src/lib/parse-worker.ts

📝 Walkthrough

Walkthrough

Implements the O-8 library import onboarding feature: a new async POST /libraries/:id/import endpoint that parses an uploaded DOCX/SEC master, persists it into a target library, optionally derives a DOCX style template, classifies editability, and returns a polling job with an OnboardingReport. Adds an in-memory onboarding job store, an editability summarizer, OpenAPI schemas, route wiring, and integration/unit tests.

Changes

Library Import Onboarding (O-8)

Layer / File(s)	Summary
Onboarding job store and types src/lib/jobs.ts, src/lib/jobs.test.ts	Adds OnboardingStage, OnboardingReport, OnboardingJob, and related types alongside a dedicated onboardingJobs in-memory Map with createOnboardingJob, updateOnboardingJob, and getOnboardingJob lifecycle functions including TTL cleanup; unit tests validate lifecycle transitions and store isolation from parse jobs.
persistParsedSpec optional libraryId override src/db/queries/specs.ts, src/db/queries/specs.integration.test.ts	Extends persistParsedSpec's result parameter with an optional libraryId that bypasses source-derived library resolution; integration test verifies the persisted row is stored in the explicitly supplied library.
Editability summarizer src/api/onboarding-report.ts, src/api/onboarding-report.test.ts	Adds LOW_CONFIDENCE_THRESHOLD, a recursive tree-walk accumulating per-value counts and low-confidence node lists, and summarizeEditability; unit tests cover aggregated counts and empty-tree cases.
Onboarding handler and processing pipeline src/api/onboarding.ts	Implements importLibraryHandler (upload validation, library lookup, job scheduling, 202 response), importJobHandler, and the full processOnboardingJob pipeline (runParseAndPersist, upsertOnboardedTemplate, deriveStyleIfDocx, classifyAndSummarize, report assembly) with deterministic SHA-256 origin metadata and error/failure handling.
OpenAPI schemas, route wiring, and contract allowlist openapi.yaml, src/api/router.ts, src/api/contract.integration.test.ts	Adds POST /libraries/{id}/import and GET /libraries/import/jobs/{jobId} paths plus all onboarding component schemas (OnboardingStage, EditabilitySummary, OnboardingReport, OnboardingJob, etc.); registers both routes behind parseRateLimit and upload.single; extends the contract test RESPONSE_ALLOWLIST.
Onboarding endpoint integration tests src/api/onboarding.integration.test.ts	Full integration suite covering DOCX success, .sec success with styleSourceNeeded, DOCX re-import idempotency and template stability, 404 for unknown library, 400 for unsupported extension, and 400 for invalid library id.
Implementation plan document docs/superpowers/plans/2026-06-23-library-import-onboarding.md	Adds the complete O-8 implementation plan (Tasks 1–7), gate checklist, and self-review checklist.

Sequence Diagram(s)

sequenceDiagram
  participant Client
  participant importLibraryHandler
  participant processOnboardingJob
  participant parsePool
  participant persistParsedSpec
  participant deriveStyleIfDocx
  participant classifyAndSummarize
  participant onboardingJobs

  Client->>importLibraryHandler: POST /libraries/:id/import (multipart file)
  importLibraryHandler->>importLibraryHandler: validateUpload (ext/MIME/zip checks)
  importLibraryHandler->>onboardingJobs: createOnboardingJob()
  importLibraryHandler-->>Client: 202 { jobId }
  importLibraryHandler-)processOnboardingJob: schedule async (fire-and-forget)

  processOnboardingJob->>parsePool: parse upload off-thread
  parsePool-->>processOnboardingJob: SpecTree + refs
  processOnboardingJob->>persistParsedSpec: persist with explicit libraryId + SHA-256 originMeta
  persistParsedSpec-->>processOnboardingJob: specId

  alt DOCX input
    processOnboardingJob->>deriveStyleIfDocx: analyze styles, upsertOnboardedTemplate, link template
    deriveStyleIfDocx-->>processOnboardingJob: { templateId, derivationReport }
  else non-DOCX input
    processOnboardingJob->>deriveStyleIfDocx: skip derivation
    deriveStyleIfDocx-->>processOnboardingJob: { templateId: null, derivationReport: null }
  end

  processOnboardingJob->>classifyAndSummarize: classify editability, summarizeEditability
  classifyAndSummarize-->>processOnboardingJob: EditabilitySummary

  processOnboardingJob->>onboardingJobs: updateOnboardingJob(complete, OnboardingJobResult)

  Client->>importLibraryHandler: GET /libraries/import/jobs/:jobId
  importLibraryHandler->>onboardingJobs: getOnboardingJob(jobId)
  onboardingJobs-->>importLibraryHandler: OnboardingJob
  importLibraryHandler-->>Client: 200 OnboardingJob

Loading

Estimated code review effort

🎯 4 (Complex) | ⏱️ ~60 minutes

Possibly related issues

• #135 (feat(api): library import onboarding — POST /libraries/:id/import orchestration): This PR directly implements the scope defined in issue #135, including the async multipart endpoint, parse/persist/style-derive/classify pipeline, onboarding report with all three sections, and the acceptance criteria for DOCX, .sec, unknown library, and malformed upload cases.

Possibly related PRs

• wrzonance/SpecR#60: Both PRs modify persistParsedSpec in src/db/queries/specs.ts—the earlier PR introduced the persistence layer, while this PR extends it with an optional explicit libraryId.
• wrzonance/SpecR#164: Both PRs touch DB spec persistence logic around library_id resolution inside persistParsedSpec.
• wrzonance/SpecR#165: Both PRs extend persistParsedSpec's payload shape—the retrieved PR adds originMeta/provenance fields while this PR adds optional libraryId.

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 33.33% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title 'feat(api): library import onboarding — POST /libraries/:id/import (O-8)' is specific and directly summarizes the main change, clearly identifying the new API endpoint for library import onboarding.
Linked Issues check	✅ Passed	The PR fully implements issue #135 requirements: multipart upload endpoint, async processing via jobs.ts, parsing, persisting to target library, DOCX style derivation (or styleSourceNeeded flag), editability classification, and three-section onboarding report. All acceptance criteria are met.
Out of Scope Changes check	✅ Passed	All changes directly support the onboarding endpoint implementation (O-8). The PR excludes onboarding_status finalization (O-11), corrections/reclassify (O-9), and MCP parity (O-13) as expected, and introduces no unrelated modifications.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch feat/issue-135

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[thewrz]

@coderabbitai review

[coderabbitai]

✅ Action performed

Review finished.

Note: CodeRabbit is an incremental review system and does not re-review already reviewed commits. This command is applicable only when automatic reviews are paused.

[coderabbitai]

Actionable comments posted: 3

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@src/api/onboarding.ts`:
- Around line 234-237: In the catch block where updateOnboardingJob is called
after an error occurs, the update only sets the status to 'failed' and includes
the error message, but does not explicitly set the progress stage and percentage
fields. To fix the inconsistent job payload, modify the updateOnboardingJob call
to also include progress stage and percentage values that indicate a failed
state, ensuring the job payload is consistent for polling clients when an error
is caught.
- Around line 133-135: The workerRaw data from the worker thread pool is cast
directly to WorkerOutput type without runtime validation, which violates the
requirement to validate external input crossing process boundaries. Replace the
type assertion with a Zod schema validation. Create a Zod schema that matches
the WorkerOutput interface structure and use it to parse and validate workerRaw
before destructuring the tree and refs properties. This ensures malformed data
from the worker thread cannot propagate into downstream persistence and
classification operations.

In `@src/lib/jobs.ts`:
- Around line 2-4: The import statement for ParseWarning bypasses the AST module
barrel by importing directly from types.js instead of through the index.js
barrel export. Update the import statement to import ParseWarning from
../ast/index.js instead of ../ast/types.js. Then verify that ParseWarning is
re-exported from src/ast/index.ts by adding the re-export from ./types.js if it
doesn't already exist. Apply this same pattern correction to the other files
mentioned (src/api/onboarding-report.ts and src/api/onboarding.ts) that have
similar direct imports of ParseWarning from internal modules.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro Plus

Run ID: 66f193da-3b38-4e2f-97d1-7b4a66246e09

📥 Commits

Reviewing files that changed from the base of the PR and between 4c00c0b and c84c495.

📒 Files selected for processing (12)

• docs/superpowers/plans/2026-06-23-library-import-onboarding.md
• openapi.yaml
• src/api/contract.integration.test.ts
• src/api/onboarding-report.test.ts
• src/api/onboarding-report.ts
• src/api/onboarding.integration.test.ts
• src/api/onboarding.ts
• src/api/router.ts
• src/db/queries/specs.integration.test.ts
• src/db/queries/specs.ts
• src/lib/jobs.test.ts
• src/lib/jobs.ts