feat: rebuild importers on adapters, resolve block refs - #193
Merged
Conversation
Reconcile turns key:: value continuation lines into Op::SetProp on the block node, but the reverse projection never wrote them back: build_outline hardcoded empty properties, so any op-to-md re-render (a GUI edit, a sync re-projection) silently deleted every property line from disk. The next external-edit reconcile then emitted prop-removal ops, converging the loss to every device.
Block properties now project back alpha-sorted, same policy as page properties, in render_page_md and render_block_md. The fix pushed journal.rs past the 900-line guard, so it also got split into journal/{paths,render,apply} modules with mod.rs re-exports keeping every existing path compiling.
The old import pipeline flattened every ((uid)) block ref into a whole-page link and mangled {{embed}}s into leftover syntax. For a graph that lives on refs and embeds, that was silent data loss at migration time.
New outl-import crate: each source (Roam, Logseq, Obsidian) is an adapter parsing into a typed IR, the emitter writes markdown with inert placeholders, and a resolve pass maps source uids to real ((blk-)) handles through the sidecars, rewriting via the op log so block ids never shift. Whatever can't be mapped degrades to [[Title]] links or greppable ((unresolved:uid)) markers, everything counted in the import report. Logseq gains a real outline parser (task states, collapsed, priorities, org dates), Obsidian keeps its vault semantics on the new engine, and `import auto` detects the source shape. Large imports paint a live progress line on stderr, --dry-run measures fidelity before touching anything.
|
Warning Review limit reached
Next review available in: 25 minutes Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available. How can I continue?After more reviews become available, a review can be triggered using the To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews. How do review limits work?CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability. For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window. Please refer docs for additional details. Review details⚙️ Run configurationConfiguration used: Organization UI Review profile: ASSERTIVE Plan: Pro Plus Run ID: ⛔ Files ignored due to path filters (1)
📒 Files selected for processing (53)
✨ Finishing Touches✨ Simplify code
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Contributor
There was a problem hiding this comment.
Pull request overview
This PR modernizes outl’s import story by replacing the legacy string-based importers with a new adapter-based pipeline (outl-import) that can resolve block refs/embeds into real ((blk-...)) handles via the op log, while also fixing a core projection bug that could drop block-level key:: value properties during op→md re-renders. It also extracts CLI workspace-boot logic into a reusable crate (outl-ws) and updates docs/changelog to reflect the new behavior and CLI surface.
Changes:
- Fixes convergent data-loss where block-level properties could be dropped from
.mdduring re-render (outl-actionsjournal projection now re-emits block props). - Introduces
crates/outl-import: typed IR + adapters (Roam/Logseq/Obsidian) + two-pass placeholder resolution into real handles through the op log, with rich reporting and progress events. - Rewires
outl importCLI to the new pipeline (addsauto,--dry-run,--json,--preserve-timestamps) and removes legacy importer modules; addscrates/outl-wsfor shared workspace bootstrapping.
Reviewed changes
Copilot reviewed 53 out of 54 changed files in this pull request and generated 7 comments.
Show a summary per file
| File | Description |
|---|---|
| README.md | Updates importer description to reflect ref/embed resolution + report-first workflow. |
| docs/contributing.md | Notes the importer move to outl-import while preserving the reuse-first lesson. |
| docs/cli.md | Updates outl import command shape and documents flags + resolution semantics. |
| crates/outl-ws/src/lib.rs | New shared workspace bootstrap (open) with locks/actor resolution/shard registration/slug repair. |
| crates/outl-ws/src/layout.rs | New workspace layout/config helpers extracted for reuse. |
| crates/outl-ws/Cargo.toml | Adds new outl-ws crate manifest/deps. |
| crates/outl-import/Cargo.toml | Adds new outl-import crate manifest/deps. |
| crates/outl-import/CLAUDE.md | Documents the adapter/emit/resolve architecture and invariants for import. |
| crates/outl-import/src/lib.rs | Public API (run_import, dry_run, progress sink), IR/emit modules wiring. |
| crates/outl-import/src/adapter.rs | Defines SourceAdapter trait and ImportError. |
| crates/outl-import/src/ir.rs | Defines the typed import IR (pages/blocks/inlines/embeds/tasks/etc.). |
| crates/outl-import/src/report.rs | Defines ImportReport fidelity contract and human printer. |
| crates/outl-import/src/progress.rs | Defines progress event types and sink signature. |
| crates/outl-import/src/emit/mod.rs | Orchestrates render→write→reconcile→resolve; dry-run simulation. |
| crates/outl-import/src/emit/render.rs | Single owner of output syntax + placeholder emission + DFS bookkeeping. |
| crates/outl-import/src/emit/resolve.rs | Pass B: placeholder→handle via sidecars + op-log edits + fallback sweep. |
| crates/outl-import/src/adapters/mod.rs | Exposes concrete adapters and shared scanners. |
| crates/outl-import/src/adapters/scan.rs | Shared low-level scanners (balanced, uid checks, alias link parsing). |
| crates/outl-import/src/adapters/roam/inline.rs | Roam inline dialect tokenization/translation (refs/embeds/components/etc.). |
| crates/outl-import/src/adapters/roam/tests.rs | Golden tests for Roam dialect fidelity rows. |
| crates/outl-import/src/adapters/logseq/inline.rs | Logseq inline dialect tokenization. |
| crates/outl-import/src/adapters/logseq/tests.rs | Golden tests for Logseq outline/dialect behavior. |
| crates/outl-import/src/adapters/obsidian/mod.rs | Obsidian adapter (frontmatter policy, wikilink normalization, routing). |
| crates/outl-import/src/adapters/obsidian/stems.rs | Obsidian collision discovery + deterministic stem assignment. |
| crates/outl-import/tests/common/mod.rs | Shared e2e harness using real Workspace + JsonlStorage. |
| crates/outl-import/tests/roam_e2e.rs | E2E: Roam JSON→workspace with real handles/embeds/collapsed/report. |
| crates/outl-import/tests/logseq_e2e.rs | E2E: Logseq graph→workspace with handles/props/collapsed/journals. |
| crates/outl-import/tests/obsidian_e2e.rs | E2E: Obsidian vault scenarios ported to adapter pipeline. |
| crates/outl-cli/src/main.rs | Extends outl import CLI args with auto + new flags. |
| crates/outl-cli/src/cmd/import.rs | Replaces legacy import dispatcher with glue over outl-import (+ progress). |
| crates/outl-cli/src/ws.rs | Delegates CLI workspace open to outl-ws, mapping errors to ApiError. |
| crates/outl-cli/src/workspace_layout.rs | Re-exports layout from outl-ws under legacy CLI path. |
| crates/outl-cli/CLAUDE.md | Updates CLI docs to reflect new importer crate + flags. |
| crates/outl-cli/Cargo.toml | Adds deps on outl-import and outl-ws. |
| crates/outl-cli/src/cmd/import/roam.rs | Removes legacy Roam string-based importer. |
| crates/outl-cli/src/cmd/import/logseq.rs | Removes legacy Logseq string-based importer. |
| crates/outl-cli/src/cmd/import/CLAUDE.md | Removes legacy importer pipeline doc under CLI. |
| crates/outl-actions/src/journal/mod.rs | Splits journal module into {paths,render,apply} and re-exports surface. |
| crates/outl-actions/src/journal/paths.rs | New paths/atomic write/projection removal helpers. |
| crates/outl-actions/src/journal/render.rs | Fixes projection to include block-level properties; adds block renderer. |
| crates/outl-actions/CLAUDE.md | Updates journal section to describe the split + block-prop projection fix. |
| CLAUDE.md | Adds outl-import to repo layout. |
| CHANGELOG.md | Documents the block-property projection fix and the new import pipeline. |
| Cargo.toml | Adds outl-import and outl-ws to workspace members/deps. |
| Cargo.lock | Locks new crates and their dependencies into the workspace lockfile. |
| .github/copilot-instructions.md | Mirrors the “importers moved to outl-import” note in guidance. |
Signed-off-by: Avelino <31996+avelino@users.noreply.github.com>
This was referenced Jul 29, 2026
4 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Rebuilds the importers (Roam, Logseq, Obsidian) on an adapter-based pipeline that resolves
((uid))block refs and{{embed}}s into real outl block references, and fixes a data-loss bug in the core md projection that this work surfaced.The old string-based importers flattened every block ref into a whole-page link and left
{{embed}}s as broken syntax. For a graph that lives on refs and embeds (mine has 475 refs + 216 embeds), migration meant silent semantic loss, the kind of thing you only notice weeks later when a reference points at the wrong granularity.This PR is big on purpose: the legacy pipeline and the new one can't coexist (same CLI surface, same output layout), so the swap is atomic and the revert is clean. It lands as two commits so the core fix is bisectable and backportable on its own:
fix:block-level properties were dropped on every op-to-md re-render (build_outlinehardcoded empty props while reconcile emittedOp::SetProp), so the next external-edit reconcile converged the loss to every device. Blast radius was every client, not just import. Includes thejournal.rssplit intojournal/{paths,render,apply}(the fix pushed it past the 900-line guard).feat:the newoutl-importcrate + CLI wiring.Changes
outl-importcrate, three stages with one owner each:roam,logseq,obsidian) parse the source into a typed IR. All dialect knowledge lives here: Roam__italic__vs CommonMark, colon-form vs space-form embeds, org-styleDEADLINE:stamps becoming[[date]]links (the issue feat(remind): block-levelremind::property + scheduled local notifications (mobile + desktop) #63 model).((blk-XXXXXX))handles only exist after reconcile stamps the sidecars.edit_text+ re-render), never by editing.mdbehind the workspace's back. Two guards (block-count match + content-hash match, both derived from outl's own parser) make mis-wiring a ref structurally impossible: anything unmappable degrades to a[[Title]]link or a greppable((unresolved:uid)), all counted in the report.Logseq gets a real outline parser (the old one passed lines through as strings):
id::becomes the ref key,collapsed:: truesurvives asOp::SetCollapsed,DOING/NOW/LATER/WAITING/CANCELEDmap toTODO/DONEwith astate::property,[#A]becomespriority::,:LOGBOOK:drawers are dropped and counted. Obsidian keeps its vault semantics (frontmatter policy, wiki-link collapsing, path-derived slug collisions) on the new engine via aPageBody::Rawbody shape.outl import autopicks the adapter from the source's shape,.obsidian/checked before Logseq's folder heuristic on purpose.UX:
--dry-run+--jsonanswer "what would I lose" with numbers before writing a byte. Long imports paint a live progress line on stderr (TTY-only, piped runs stay silent).Testing
91 tests in the crate (golden per fidelity-matrix row + e2e over a real
Workspace), 88 suites green across the workspace, clippy-D warningsclean.The acceptance test was my real Roam backup: 4.472 pages, 65.728 blocks. Three import rounds against it each exposed a failure class no synthetic fixture would find (Logseq residue pasted into Roam diverging the hash, parser splitting blocks the renderer didn't predict, markers hiding inside property values). Final state: 457 refs + 214 embeds resolved to real handles, 19 + 2 degraded to links, 73 greppable unresolved, zero placeholder markers on disk, which is now a tested invariant.
Known issue found along the way, out of scope here: outl-md's parser silently drops a continuation line that follows a property line (violates the crate's "no silent block loss" invariant). The importer defends against it via the fallback sweep; the parser fix deserves its own PR.
Related: opened #190 (page embeds) and #191 (alias wikilinks) for constructs the import preserves verbatim today.
Summary by CodeRabbit
--dry-run, JSON reports, timestamp preservation, progress updates, and detailed fidelity warnings.priority:: high.