feat: add document format import (SDD/TDD, MCP integration, review hardening) - #8
Merged
Merged
Conversation
added 6 commits
August 29, 2026 08:56
…rdening) Implements OKF v1.3 document format import driven by openspec SDD/TDD (S1-S29, 29/29 conformance): - pkg/convert: pure-Go conversion layer on downmark v0.10.0 (pdf/docx/xlsx/pptx/html/csv/txt), with input/result limits, timeout control, deterministic staging output - cmd okf add: document conversion + full archive extraction (zip/tar/ tar.gz/tar.bz2) staged before smart import; dry-run supported; per-file failure never aborts the batch - MCP: okf_import_document tool (Content-Length framed stdio already in PR #6) - openspec: proposal/design/spec/tasks/conformance + 19-dim AGENTS.md development norm - docs: v1.3.0 release notes, CLI/README updates Two-round code review fixes included: - Round 1 (explicit): removed dead code (collectMarkdownFiles, computeTargetPath); tightened dry-run & mixed-report test assertions; documented Timeout==0 semantics - Round 2 (latent): archive hardening (whole-archive <=50MB, per-member <=10MB, io.LimitedReader decompression guard against forged headers, explicit .tar.xz error); single-document conversion failure now exits non-zero (all-failed) while partial failure keeps batch semantics; added special-title YAML round-trip, tar.xz, and all-conversion-failed tests; recorded known limits (okf sync does not track doc sources, MCP overwrite semantics) in conformance.md Verified: go build/vet clean, all tests pass (56), test_mcp.py E2E green.
The project go.mod requires go 1.26 (downmark v0.10.0 toolchain); Go 1.20 cannot parse `go 1.26.0` / `toolchain` directives, which broke the build check on PR #8. Align the CI toolchain with go.mod.
- concurrency: KnowledgeBundle RWMutex (Add/RemoveConcept write, FilterConcepts/ Stats/GetConcept read, delegated filters inherit read lock); fix shared mutable cfg in stress concurrent-generation test (per-goroutine copy) - staticcheck: fix archive tempdir cleanup leak (defer bound to empty func), tar.TypeRegA deprecation, duplicated assertion in deps_test, drop 2 dead funcs - coverage: whole-repo >=60% gate wired into CI and gauntlet - property tests (testing/quick, zero new deps): equalFold vs strings.EqualFold, containsFold/indexFold consistency, sanitizeFilename safe+idempotent, WrapConcept frontmatter round-trip through parser - gauntlet: tools/gauntlet.sh (build/vet/gofmt/staticcheck/test -race/coverage/ shuffle/mutation/real-exec, fail-on-first) + tools/mutants.sh (4/4 killed) - tests: cover extractTarFull branches (gzip/plain/symlink/size/traversal), WrapConcept direct unit test - ci: run gauntlet as single gate, install staticcheck - style: gofmt over tracked files (format-only) - docs: AGENTS.md gains GAUNTLET/concurrency/property-test norms
- mark tools/*.sh executable in git (were 100644 -> 100755); CI checkout could not execute tools/mutants.sh, failing the gauntlet step with 126 - invoke mutants via `bash tools/mutants.sh` so it does not depend on the exec bit at runtime - bump actions/checkout + actions/setup-go to v5 (drop Node.js 20 deprecation warning)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Implements OKF v1.3 document format import (pdf / docx / xlsx / pptx / html / csv / txt), driven by openspec SDD/TDD — spec 29 scenarios S1–S29, 29/29 conformance (
openspec/changes/add-document-format-import/conformance.md).What's included
pkg/convert— pure-Go conversion layer ondownmark v0.10.0(Go-1.26 toolchain), with input-size (64MiB) / result-size (32MiB) limits, per-call timeout control, and deterministic staging output.okf add— document files and archives (zip/tar/tar.gz/tar.bz2) are converted/extracted into a deterministic staging dir before smart import; dry-run supported; a single conversion failure never aborts the batch (S15); all-failed exits non-zero.okf_import_documenttool (stdio already Content-Length framed from PR feat: add MCP server with stdio transport and project knowledge base #6), plus searchability verification across all 7 formats (S29).AGENTS.md(19-dimension development norm, SDD/TDD + wiring/coverage requirements).Two-round code review fixes (included in this PR)
Round 1 (explicit): removed dead code (
collectMarkdownFiles,computeTargetPath); tightened dry-run (no files at all) and mixed-report (captured stdout assertsConverted (documents): 1) test assertions; documentedOptions.Timeout == 0semantics.Round 2 (latent):
okf.MaxArchiveSize), per-member <= 10MB,io.LimitedReaderdecompression guard against forged/absent size headers, explicit.tar.xzrejection with guidance; zip-slip/symlink rejected.Failed (documents)..tar.xzclear error, all-conversion-failed exit code.Verification
go build ./.../go vet ./...cleango test ./...all pass (56 tests, incl. golden fixtures + existing suite)test_mcp.pyE2E green (incl.okf_import_document)Known limits (documented in conformance.md)
okf syncdoes not track converted-document sources (staging path is ephemeral); track changes viaokf add <doc>— no data loss (DetectSourceMissingonly flags, never deletes).okf_import_documentwrites<original>.mdat bundle root and overwrites same-name files (no merge policy, per spec S22).