Skip to content

Releases: iwe-org/iwe

v0.12.0

Choose a tag to compare

@gimalay gimalay released this 12 Jul 13:10

iwe

Added

  • refs_text markdown option — preserve (default) keeps each markdown link's text as written; normalize makes iwe normalize and document output rewrite the text to the linked document's title.
  • iwe stats similarity — list mutually near-identical page pairs across the store, each pair once and tab-separated in alphabetical order (forward matches computed once per page, concurrently).
  • iwe stats gains an Orphans section listing every page with no incoming links (index pages are exempt as intentional entry points); per-document stats (-k) gain a Similar page line (markdown) / similarPages array (JSON/YAML) of near-identical documents.
  • iwe schema validate — validate documents against the schemas bound to them by the [schemas] config section (each entry names a schema file in .iwe/schemas/ and a glob that binds it to document keys). Reports violations as -f text (default) or -f json, and accepts the universal filter flags to scope the check. Exits 1 when any document has violations, 2 on a config or schema-file error, 0 when clean. Bare iwe schema still infers the frontmatter schema.
  • iwe schema validate --schema-file <PATH> — validate the selected documents (-k or the filter flags) against a schema file directly, bypassing the [schemas] config bindings, for ad-hoc checks; violations are reported under the file's stem.
  • iwe schema validate --explain — print the binding trace (which section and block bound to which schema entry, additional for the rest) instead of validating, to see how the greedy matcher reads a document against a schema.

Changed

  • iwe normalize and document rendering keep markdown link text as written by default; set refs_text to normalize to rewrite each link's text to the linked document's title (previously always rewritten).
  • update --strict and delete --strict now reject a write that would leave a touched document violating its bound schema, aborting with exit 2 and the violation report before anything is written; previously --strict enforced only the --expect guards.
  • update --strict and delete --strict also print non-blocking stats warnings to stderr after the schema gate — orphan pages and dangling links across the store, plus (on update) near-identical pages among the changed documents. They never change the exit code or block the write.

iwes

Added

  • refs_text markdown option — set to normalize to make document formatting rewrite each markdown link's text to the linked document's title; preserve (default) keeps the text as written.

Changed

  • Document formatting keeps markdown link text as written by default (previously each link's text was rewritten to the linked document's title on format).

iwec

Added

  • iwe_create, iwe_update, and iwe_query (update / delete) now return non-blocking stats warnings alongside a successful result — orphan pages, dangling links, and (on create/update) near-identical pages — as warning: content blocks, each finding reported once per session. They never block a write; schema validation remains the only hard reject.
  • iwe_stats with a key now returns a similarPages array of documents near-identical to that page.

Changed

  • Markdown link text in rendered document content is kept as written by default; set the refs_text markdown option to normalize to rewrite each link's text to the linked document's title as before.
  • Mutating tools (iwe_create, iwe_update, iwe_delete, iwe_query, iwe_rename, iwe_extract, iwe_inline, iwe_attach) now reject a change that would leave a touched document violating its bound schema, returning a detailed error carrying the readable report and a structured violations payload; nothing is written and the in-memory graph is left untouched. iwe_normalize and iwe_squash are unaffected.

liwe

Added

  • refs_text field on MarkdownOptions and DjotOptions — a RefsText (preserve by default, or normalize) that controls whether a regular markdown link's text is rewritten to the linked document's title; read through FormatOptions::refs_text(), with InlinesContext and Graph gaining normalize_ref_text.
  • schema module — document-schema validation: compile_schema compiles a YAML/JSON schema (frontmatter JSON Schema plus an ordered section tree with header, maxTokens, maxDepth, minContains/maxContains, allSections, additionalSections) into a CompiledSchema, and CompiledSchema::validate checks a Document and returns Violations carrying a breadcrumb, hint, schema pointer, and keyword. Unknown keywords are load errors (SchemaError).
  • schema validation reaches block content: the document and every section, quote, and list item carry blocks / additionalBlocks / allBlocks — an ordered, greedy-matched array of block schemas discriminated by type (paragraph, bullet-list, ordered-list, code, quote, table, rule, or a list of these for a disjunction), each with text / lang identity, maxTokens, minContains / maxContains, list items / minItems / maxItems, and quote/item recursion; Document and Section now carry a blocks: Vec<Block> field and Crumb gains Block(usize) / Item(usize). An inclusion link is matched as a paragraph.
  • CompiledSchema::explain renders the binding trace — which section and block bound to which schema entry, additional for the rest — for a Document. A sections or blocks array with an unreachable entry (a wildcard that is not last, or an exact duplicate of an earlier entry) is a load error.

Changed

  • A regular markdown link's text is kept as written when rendering a document; set refs_text to normalize to re-derive it from the linked document's title (previously the text was always re-derived).

diwe

Added

  • config::RefsText re-export and the refs_text field it sits on (MarkdownOptions/DjotOptions) — selects whether a markdown link's text is preserved (default) or normalized to the linked document's title.
  • stats findings functions — graph_findings (whole-store orphan and dangling-link Findings, discriminated by Rule) and mutation_findings (the same plus a similar-page check for the created/updated keys), with orphan_keys and the now-public broken_links behind them.
  • stats::SimilarityIndex — a search index plus per-key token counts, built once per run (SimilarityIndex::build) and reused for similar(key) (near-identical pages for one key, as stats::SimilarPage { key, score }) and pairs() (every mutually-similar pair across the store, each once, computed concurrently). Duplicate detection uses mutual BM25 similarity with a token-size gate and a high threshold.
  • GraphStatistics.orphans — the list of orphan keys behind the existing orphaned_documents count; stats::KeyStatisticsReport pairs a KeyStatistics with its similar pages. index pages (root index or any <dir>/index) are treated as intentional entry points and excluded from both the orphan list and the count.
  • search::Bm25Index point-score API — similar_to(key, floor) (documents whose self-normalized score against key's own embedding clears a floor, self excluded), self_score(key), and score_between(query_key, doc_key); search_query::corpus_text is now public.
  • [schemas] config binding — config::SchemaBinding and config::Patterns types and the Configuration.schemas map bind document schemas to document keys by glob (a single glob or a list). The new schema module resolves and runs them: schema::SchemaBindings matches a key to its schema names, and schema::validate_documents compiles the bound schema files, validates a set of documents, and returns a schema::KeyReport per (key, schema) with violations.
  • schema::validate_pending_documents (and schema::validate_pending_documents_in, which takes an explicit schemas directory) validate a set of pending (Key, content) documents against their bound schemas by building a throwaway graph, so a change can be checked before it is written. schema::pending_from_changes collects the touched documents from a Changes set, schema::render_reports_text renders a KeyReport list as text, and config::schemas_dir_in resolves the schemas directory under a given base path.
  • schema::validate_documents_against_file — validate documents against one schema file directly, ignoring the [schemas] config bindings; reports are keyed by the file's stem.

Changed

  • search now orders tied scores deterministically (score descending, then key ascending); previously ties came back in arbitrary order.

v0.11.0

Choose a tag to compare

@gimalay gimalay released this 10 Jul 12:37

iwe

Added

  • new --key <KEY> — create a document at an explicit key, bypassing the template's key derivation. Subdirectory keys (e.g. people/ada) are allowed; omit the file extension.
  • new --if-exists fail — report an error and exit non-zero when the document already exists. It is the default when --key is given (an explicit key asserts an identity), where previously an existing key silently gained a -1 suffix.
  • retrieve --expand-includes / --expand-included-by / --expand-references / --expand-referenced-by — one flag per expansion direction, each taking an optional depth (bare flag = one level, 0 = unbounded, omitted = not followed). --expand-referenced-by (pull documents that reference a seed) and transitive --expand-references are new directions.
  • retrieve --lexical / --fuzzy — a one-shot form that searches for seed documents within the candidate set (-k / --filter / anchors) and then expands the graph around the ordered seeds.
  • retrieve --max-documents N — cap the number of documents returned after expansion, trimming periphery documents first (0 = unlimited).

Changed

  • retrieve --limit now caps the selected seed documents before expansion — top-N by relevance when searching, the first N of the selection otherwise (previously it capped the number of documents returned after expansion; use --max-documents for that).
  • retrieve no longer expands by default: with no --expand-* flag (and no deprecated flag) it returns the requested document(s) only. The previous implicit -d 1 -c 1 is now written explicitly as --expand-includes 1 --expand-included-by 1.

Deprecated

  • retrieve -d / --depth, -c / --context, -l / --links — retained as hidden aliases for --expand-includes N / --expand-included-by N / --expand-references 1 (keeping their legacy 0 = off meaning). Passing one together with its --expand-* counterpart is an error.

iwec

Added

  • iwe_create gains an optional key parameter — create a document at an explicit key instead of a title-derived slug. Derive it from stable metadata (entity name, session date); subdirectory keys (e.g. people/ada) are allowed; omit the file extension. Creation fails if a document with that key already exists.
  • iwe_query find operations accept a search clause (search: { lexical, fuzzy }) — relevance selection that restricts and orders results; a lexical query with no searchable terms returns an empty array plus a warning content block.
  • iwe_retrieve gains search / fuzzy (seed queries), expand (object over includes / includedBy / references / referencedBy → integer depths, 0 = unbounded), and max_documents (cap the documents returned after expansion). With a search query the tool finds seed documents within the candidate set (keys + selector) and expands the graph around the ordered seeds.

Changed

  • iwe_retrieve limit now caps the seed documents before expansion (top-N by relevance when searching, the first N of the selection otherwise); use max_documents for the post-expansion document cap.
  • iwe_retrieve no longer expands by default — omit expand and it returns the requested document(s) only (previously the implicit behavior was one level of children and parents).
  • iwe_delete and iwe_query deletes now also remove any parent directory left empty by a removed document, matching the CLI (previously empty directories were kept).

Deprecated

  • iwe_retrieve depth / context / links — retained as aliases for expand's includes / includedBy / references; passing expand together with any of them is an error.

liwe

Added

  • Changes::merge — fold another Changes into this one, deduplicating by key so a later update replaces an earlier one for the same document.
  • operations gains the section/reference selection helpers (sections, select_section, references, select_reference, SelectError, SectionRef, InclusionRef) and attach_reference / AttachTarget, so all operations live in liwe::operations alongside extract, inline, delete, and rename.
  • search clause on FindOp (SearchSpec { lexical, fuzzy }) — a relevance stage that restricts membership to documents matching the query and supplies the default ordering, jointly with filter; search + sort keeps membership from search while sort supplies the order. The ranking logic (query::search::ranked / matched, RRF fusion) is now a shared stage used by both DocumentFinder and the query engine.
  • RetrieveOptions expansion generalized to four edge-named depths — includes, included_by, references, referenced_by (u32, 0 = off, UNBOUNDED = no limit) — replacing the depth / context / links fields. Inbound-reference expansion (referenced_by) and transitive outbound-reference expansion (references > 1) are now expressible; retrieve::expand_depth maps an --expand value (0 = unbounded) to a depth.
  • format::DocumentFormat trait (read / write / write_skip_frontmatter) as the format boundary, with the built-in MarkdownFormat and (feature-gated) DjotFormat implementations and a format::format_for constructor.
  • query::QueryScores and query::execute_with_scores — the query engine now ranks a search clause from caller-injected per-key relevance scores instead of computing them itself, keeping the kernel free of any search index.

Changed

  • Djot support is now behind an opt-in djot cargo feature; jotdown is an optional dependency and the djot module compiles only when the feature is enabled. Markdown remains built in. Enable features = ["djot"] to read and write .dj documents.
  • The BM25 search index left the kernel: Graph no longer builds or holds one, so Graph::search, has_search_index, search_scores, and lexical_query_has_terms are removed, and Graph::from_state / Graph::import no longer take a search_language argument. The BM25 index and the search scoring (ranked / matched) moved to the diwe crate; query::SearchSpec stays as the DSL type.
  • Graph::has_search_index() reports whether the graph carries a BM25 index; running a find with a search clause against a graph without one is an execution-time error (EvalError::SearchIndexMissing).
  • RetrieveOptions.limit now caps the seed set before DocumentReader::retrieve_many expands (it was a post-expansion cap on the number of documents returned); the post-expansion cap moves to the new RetrieveOptions.max_documents field. Both are Option<usize>, None / Some(0) = unlimited.
  • EdgeRef moved from retrieve to query::edges.

Removed

  • The engine modules (find, retrieve, stats, tokens, fs, file) and the .iwe/config.toml mapping (Configuration, LibraryOptions, CompletionOptions, SearchOptions, Command, ActionDefinition and its variants, NoteTemplate, load_config) moved to the new diwe crate; liwe keeps the document kernel and the format/option types (Format, FormatOptions, MarkdownOptions, DjotOptions, FormattingOptions, LinkType, InlineType, TargetType, Operation) in liwe::model::config.
  • Graph::from_path — the filesystem-loading constructor moved to diwe; build a State from disk and call Graph::from_state.

diwe

Added

  • diwe is the IWE engine library carved out of liwe. It carries the app-facing layer: find (BM25 / fuzzy search), retrieve (document expansion with token budgeting), stats, tokens, fs (filesystem / workspace loading), graph_from_path, and the .iwe/config.toml mapping (config::Configuration, config::load_config). It depends on liwe for the document kernel and re-exports liwe's format/option types from diwe::config.
  • search (the BM25 index) and search_query (BM25 + fuzzy resolvers, RRF fusion, build_index, ranked / matched, and an execute wrapper that resolves a query's search clause into scores and injects them into the liwe engine). DocumentFinder::with_index takes a caller-built index.
  • fs::apply_changes — write a Changes set to a workspace (creates, updates, and removals), pruning any parent directories left empty by a removal.

v0.10.0

Choose a tag to compare

@gimalay gimalay released this 10 Jul 01:00

iwe

Added

  • refs_path markdown option — set it to absolute to write links as root-absolute paths (/dir/note.md) on normalize, instead of the default paths relative to the linking document.

Fixed

  • Root-absolute links (a leading /, such as /dir/note.md) and links carrying a #fragment now resolve from any directory. Previously such links were dropped from the graph unless the linking file sat at the library root, so tree, stats, retrieve, and backlinks under-reported references.

iwes

Added

  • refs_path markdown option — absolute makes document formatting and link completion write links as root-absolute paths (/dir/note.md) instead of paths relative to the linking document.

Fixed

  • Root-absolute links (a leading /) and links carrying a #fragment now resolve from any directory, so backlinks, go-to-definition, and completions see references that were previously dropped unless the linking file sat at the library root.
  • The link code action writes the new link relative to the current document and honors the refs_path setting — previously it wrote the target's full library path, producing a broken link when invoked from a document in a subdirectory.

liwe

Added

  • RefsPath enum and a refs_path field on MarkdownOptions / DjotOptions (default RefsPath::Relative), surfaced through FormatOptions::refs_path(); RefsPath::Absolute renders regular links as root-absolute paths (/dir/note.md) instead of paths relative to the linking document.
  • Key::link_url(relative_to, refs_path) builds a regular link's path for the given RefsPath, so every link-writing path shares one implementation.

Fixed

  • Key::from_rel_link_url resolves a regular link with a leading / from the library root regardless of the linking document's directory (previously a leading / only resolved from a document at the root), and strips a trailing #fragment before computing the key so note.md#section resolves to the same key as note.md.

v0.9.0

Choose a tag to compare

@gimalay gimalay released this 09 Jul 14:57

iwe

Added

  • --project / --add-fields accept block-addressed sources: { $content: PREDICATE } narrows a document's body to the selected blocks (rendered at their original depth), $blocks / { $blocks: PREDICATE } lists each selected block as type / path / text data, and { $matches: REGEX } greps matching lines with their section paths.
  • find markdown output renders $blocks and $matches entries one line each as key › section path › text, and switches to the fenced-block form with the narrowed body when a parameterized $content field is projected.
  • --project accepts a bare block predicate — --project '$header: {}' renders each document's body narrowed to the selected blocks (the headers-only form) under key and content fields, so --format json/yaml output keeps the document identity that the markdown fence already carries.
  • update gains a block-edit flag per operator — --replace, --replace-text, --insert-before, --insert-after, --append, --delete — each taking a { <selector>, payload } mapping and composing with --set / --unset into one atomic update. A validation failure (unmet expect, overlapping selections, incompatible target) prints the offending blocks and exits non-zero without writing. --replace-text accepts a from-less argument ({ $header: Goals, to: Aims }) that rewrites the block's entire own text — the clean way to rename a header or restate a line.
  • A block edit targeting a $header acts on the heading line alone: --delete '{ $header: Goals }' dissolves the section (contents re-attach to the parent and re-level) and --replace '{ $header: Goals, content: "## Aims" }' retitles it (contents kept), while --delete '{ $section: Goals }' removes the whole tree. --insert-after '{ $header: Goals, content: ... }' adds content at the top of the section, below the heading line.
  • update and delete gain --expect — a document-level guard asserting the number of matched documents (N or { min, max }); on a mismatch the command lists the matched documents as key › title and exits non-zero without writing. Both also gain --strict, which requires an expect guard on every mutating application (the document-level --expect and each block operator's expect) and aborts before writing if any is missing; --dry-run is exempt so counts can be learned.
  • find --blocks PRED adds a blocks field listing each block matching the predicate (lowers to addFields: { blocks: { $blocks: PRED } }), and find --matches PATTERN restricts results to documents whose content matches PATTERN and adds a matches grep field (lowers to a $content membership filter plus addFields: { matches: { $matches: PATTERN } }).
  • find --filter accepts the $content block-membership operator — --filter '$content: { $header: Status }' selects documents that contain at least one block satisfying the predicate.

Changed

  • update -k / --key is repeatable, matching find: one key lowers to $eq, two or more to $in (body-overwrite mode still takes exactly one). Previously it accepted a single key only.
  • update writes only documents whose rendered content actually changes and reports honestly — Updated N document(s) when every matched document changed, Matched N document(s), M changed otherwise (No documents matched when none) — so a no-op edit (e.g. expect: 0) leaves the file, and its mtime, untouched.

Fixed

  • --project / --add-fields now parse the argument as a YAML mapping whenever it contains a : or {, and report a parse error on malformed input instead of silently falling back to the comma list. Previously an unbraced multi-field mapping like --project 'a: { $content: ... }, b: { $content: ... }' failed the YAML parse, degraded to the comma list, and emitted a: null, b: null with no error. The comma list keeps the name, name=source, and bare $selector forms; write multi-field or block projections as a braced mapping.

iwec

Added

  • iwe_query tool runs an IWE query/block-selection operation document — operation is find / count / update / delete and document is the operation as a YAML string. It exposes the $content membership filter, the $content / $blocks / $matches projection sources, and the block update operators ($replace, $replaceText, $insertBefore, $insertAfter, $append, $delete). find and count read; update applies frontmatter and block edits; delete removes documents with reference cleanup. The tool is always strict: every mutating application must carry an expect guard or the operation is refused with the missing guards named. update / delete accept dry_run to preview without writing.

liwe

Added

  • query::block module with the BlockPredicate grammar for addressing blocks inside a document: text and regex predicates, $within / $contains axes, per-type operators ($section, $header, $paragraph, $item, $list, $quote, $code, $table, $ref, $hr), $references, and $and / $or / $nor composition.
  • Block-addressed projection sources: { $content: PREDICATE } renders the selected blocks, $blocks reports each selected block as type / path / text, and { $matches: REGEX } greps matching lines with their section paths — all evaluated through the new query::block_eval::BlockIndex.
  • IntoBlockPredicate trait accepting scalar shorthands wherever a block predicate is expected.
  • FindOp::add_fields sets an additive (addFields) projection; CountOp and DeleteOp implement From<FindOp>, reinterpreting a built find with its projection dropped.
  • project accepts a $-prefixed block predicate mapping (project: { $header: {} }), lowering it to a key field and a content field carrying the narrowed body.
  • Block update operators in the update document — $replace, $replaceText, $insertBefore, $insertAfter, $append, $delete — each pairing a block predicate selector with its payload and an optional expect guard, validated and applied atomically; represented by BlockUpdate, BlockUpdateOp, and Expect on Update::block_ops.
  • Unit block operators act on their target as selected: a $header node covers its heading line alone ($delete dissolves the section, a heading $replace retitles it) while a $section covers the whole tree.
  • UpdateOp and DeleteOp carry an optional expect guard asserting the number of matched documents; on violation nothing is written and the error lists each matched document.
  • $content filter operator (Filter::Content) matches documents holding at least one block satisfying a block predicate; it composes with every other filter clause.
  • query::strict_guard_violations names the mutating applications that lack an expect guard.

Changed

  • query::execute returns Result<Outcome, block_update::EvalError> (was Outcome) so a failed block update reports its validation error instead of writing; find, count, and delete always return Ok.
  • Update carries block_ops: Vec<BlockUpdate> alongside the frontmatter operators.
  • Projection carries a base: ProjectionBase (Empty / Frontmatter / Document) in place of mode: ProjectionMode, so FindOp::project is a plain Projection (was Option<Projection>); Projection::document_fields() replaces Projection::default_for_find().
  • ProjectionContext is constructed with ProjectionContext::new(graph, key) (was a public struct literal).

Fixed

  • A mis-typed project mapping now reports a parse error instead of being silently read as a comma list of frontmatter field names and yielding null.

Removed

  • query::prelude module (with its WithFilter trait) — the Rust builder functions moved into the test suite; construct Operation, FindOp, and Filter directly.
  • query::project::apply_projection_or_defaultapply_projection covers the no-projection case.

v0.8.0

Choose a tag to compare

@gimalay gimalay released this 07 Jul 14:54

iwe

Added

  • find gains explicit --fuzzy (subsequence match on document title and key) and --lexical (BM25 full-text scoring over title and body) query flags; supplying both fuses the two result sets with Reciprocal Rank Fusion. Set the stemming language for lexical search with [search] language in .iwe/config.toml.
  • find --lexical prints a warning when the query reduces to only stop words after stemming, so an empty result set is explained instead of looking like an empty index.

Fixed

  • find and retrieve truncation warning now suggests only the limits that actually apply (--limit, --max-tokens, --max-document-tokens) instead of always naming --max-tokens, which does nothing for a metadata-only index bounded by --limit.

Deprecated

  • find's bare positional query defaults to fuzzy matching and now prints a warning on stderr; it will be removed in a future release. Use --fuzzy or --lexical instead.

iwes

Changed

  • Workspace-symbol search fuses fuzzy matching with BM25 full-text relevance (over document title and body) using Reciprocal Rank Fusion, so a query term in a document's body can lift it above an equally-fuzzy result.

iwec

Changed

  • iwe_find replaces its query parameter with explicit fuzzy (match on document title and key) and lexical (BM25 full-text over title and body) parameters; supplying both fuses the results with Reciprocal Rank Fusion. Set the stemming language for lexical search with [search] language in the configuration.

liwe

Added

  • search::Bm25Index plus Graph::search and Graph::search_scores provide BM25 full-text ranking over document title and body; the index is built and kept in sync by the ingestion pipeline (insert_document / update_document / remove_document).
  • Graph::to_plain_text renders a document to plain text (markup stripped, link display text kept, code and table cells included); Node::plain_text now also covers table cells.
  • [search] configuration table with a language field (one of 17 stemming languages, default english), exposed through Configuration::search_language.
  • search::rrf_weight and search::RRF_K for Reciprocal Rank Fusion of ranked result lists.
  • Graph::lexical_query_has_terms (backed by Bm25Index::has_query_terms) reports whether a lexical query keeps any searchable terms after stop-word removal and stemming.

Changed

  • Graph::from_state and Graph::from_path take an extra Option<Language> argument that enables search indexing when set (None skips it); Graph::import is unchanged and keeps indexing off.
  • FindOptions carries separate fuzzy and lexical query fields (was a single query); when both are set the finder fuses the two rankings with Reciprocal Rank Fusion.

v0.7.0

Choose a tag to compare

@gimalay gimalay released this 04 Jul 00:21

iwe

Added

  • retrieve --limit, and --max-tokens / --max-document-tokens on retrieve and find, to bound output for context-limited callers. 0 disables a limit. A warning: line is printed to stderr when output is truncated.

Changed

  • find markdown output is now a compact index (one line per document) instead of full document blocks; a document body is rendered only when the projection includes $content (via --project / --add-fields). Use retrieve for full content.

Removed

  • retrieve --no-content — removed; retrieve always returns content. Use find for a metadata-only index.
  • retrieve --dry-run — removed.

iwec

Added

  • iwe_retrieve and iwe_find accept max_tokens / max_document_tokens (and iwe_retrieve a limit) to bound output; all are unlimited unless set. When a limit trims the output, the tool appends a second content block with a JSON truncation summary (truncated, emitted, matched, clipped, tokens, budget, hint) alongside the unchanged primary JSON.

Removed

  • iwe_retrieve tool no_content parameter — removed; the tool always returns content.

liwe

Added

  • tokens module (count_tokens, truncate_to_tokens) backed by tiktoken-rs.
  • RetrieveOptions gains limit, max_tokens, max_document_tokens; FindOptions gains max_tokens, max_document_tokens; RetrieveOutput / FindOutput gain a truncation summary.

Removed

  • RetrieveOptions::no_content field — removed; DocumentReader always populates content.

v0.6.1

Choose a tag to compare

@gimalay gimalay released this 03 Jul 18:53

iwe

Fixed

  • retrieve --backlinks can now be turned off: --backlinks false disables incoming references, while a bare --backlinks (and the default) still includes them. Previously the flag was stuck on and --backlinks false was rejected outright.
  • stats -k <key> accepts a key written with a .md/.dj extension (stats -k note.md) instead of reporting the document as not found.
  • find --format keys combined with --project now prints the matched keys instead of nothing.
  • attach reports an error and exits instead of crashing when an action's key_template or document_template is malformed; attach --list and --dry-run are affected too.
  • schema and find --filter '{$type: datetime}' no longer crash when a document holds a datetime value with multibyte characters.

iwes

Fixed

  • Renaming a wiki link ([[target]] or [[target|label]]) now selects the target for editing instead of an empty spot at the closing brackets.
  • Positions in a document that starts with an empty frontmatter (--- / ---) are no longer shifted up by two lines, so goto-definition, hover, rename, and code actions land on the right line.
  • A failed rename (for example when the target file name is already taken) is now returned as a proper LSP error response instead of an empty success, so the editor surfaces the message to the user instead of silently doing nothing.
  • Link completion no longer leaves a stray [ behind when the cursor sits after trailing spaces; the completion is inserted at the cursor instead of overwriting part of an earlier word.
  • Find references invoked on a link now reports references to the linked document (rather than the current document) when the request asks to include the declaration.
  • An unknown LSP request now returns a MethodNotFound error instead of panicking the request handler and leaving the client waiting for a response that never arrives.
  • Formatting a document that is not part of the library (a file outside the library path, or a brand-new unsaved file) no longer crashes the server; it returns no edits.
  • A transform action environment value that contains non-ASCII characters no longer crashes code action resolution.
  • A long editing session on a document that contains a table no longer grows the server's memory without bound; the table's lines are released each time the document is re-parsed.

Removed

  • The advertised workspace/executeCommand capability, which offered an unimplemented generate command that had no handler.

iwec

Fixed

  • iwe_normalize now actually reformats documents on disk: it compares each document's normalized form against the file's current contents and rewrites the ones that differ, reporting an accurate normalized count instead of always returning 0.
  • iwe_create rejects a title with no alphanumeric characters (e.g. "!!!") instead of writing an empty-named file.
  • iwe_attach and the iwe://config resource return an error instead of crashing the server when an attach action's template is malformed.
  • The file watcher maps a file named note.md.md to the key note.md instead of note, matching how the graph loads documents from disk.
  • Repeatedly saving a watched document that contains a table no longer leaks memory as the server re-parses it.

liwe

Fixed

  • DocumentInline::key_range now returns the target range for wiki links ([[target]] and [[target|label]]); it previously assumed [text](url) syntax and returned an empty range at the wrong offset.
  • The markdown reader now offsets line positions by the stripped empty frontmatter (--- / ---), so block and inline ranges point at the original document lines instead of being two lines too low.
  • Sorting a field that mixes value types (e.g. some documents with priority: 3 and others with priority: high) now produces a stable, deterministic order: values are grouped by type before being compared, so cross-type pairs no longer collapse to "equal" and scramble the result.
  • Numeric filters compare integers exactly instead of routing everything through 64-bit floats, so large whole numbers past 2^53 (like id: 9007199254740993) are no longer treated as equal to their neighbours.
  • A not-a-number filter target (.nan) no longer compares as equal to every number, so range operators like $gte: .nan match nothing instead of matching everything.
  • Datetime detection in query filters no longer crashes on a value with multibyte characters in the timezone tail (for example 2026-04-26T10:30:00+0é9).
  • Code blocks whose content contains a run of the fence token are now written with a longer fence, so the block no longer terminates early and leaks its content into the surrounding text.
  • A paragraph that starts with an escaped \* keeps its backslash when written, so it is not re-parsed as a bullet list on the next pass.
  • A document key strips only one file extension, so a file named note.md.md maps to the key note.md instead of collapsing onto note and being duplicated on write.
  • Deleting a table node now releases its header and row lines back to the arena; repeatedly re-parsing a document that contains a table no longer leaks those line slots and grows memory without bound.

v0.6.0

Choose a tag to compare

@gimalay gimalay released this 27 Jun 20:00

iwe

Added

  • preserve_newlines config option keeps each line of a paragraph on its own line during normalize instead of joining them with spaces, so source files written with one sentence per line (semantic line breaks) survive formatting (default off).

Fixed

  • normalize no longer collapses nested or multi-paragraph djot list items onto one line; the blank line that keeps them separate is preserved so the list survives repeated runs.
  • Commands no longer crash on a djot document that contains a reference link definition or a definition list.
  • normalize keeps the word boundary at a hard line break instead of running the surrounding words together.
  • normalize preserves djot task list checkboxes (- [ ] / - [x]), display math ($$), and autolinks (<url>) instead of mangling them.

iwes

Added

  • preserve_newlines config option keeps each line of a paragraph on its own line when formatting instead of joining them with spaces, so documents written with one sentence per line (semantic line breaks) survive format-on-save (default off).

Fixed

  • Formatting a djot document no longer collapses nested or multi-paragraph list items into the parent item; the blank line separating them is kept so the list structure survives format-on-save.
  • The server no longer crashes when parsing a djot document that contains a reference link definition or a definition list.
  • Formatting keeps the word boundary at a hard line break instead of running the surrounding words together.
  • Formatting preserves djot task list checkboxes (- [ ] / - [x]), display math ($$), and autolinks (<url>) instead of mangling them.

liwe

Added

  • FormattingOptions gains a preserve_newlines field (default false); when enabled, a soft line break inside a paragraph is read as DocumentInline::SoftBreak and written back as a plain newline instead of being collapsed into a space, so the source line layout survives normalization.

Changed

  • Inline::Math now carries a MathType (Inline::Math(MathType, String)), distinguishing inline from display math.

Fixed

  • The djot writer now leaves a blank line before a nested list or any second block inside a list item, so list items with sub-lists or extra paragraphs round-trip instead of collapsing into the item's first line.
  • The djot reader no longer panics on a document that contains a reference link definition or a definition list; the orphaned text is dropped instead of crashing the parser.
  • A hard line break now becomes a space when line breaks aren't preserved, instead of running the words on either side together.
  • Djot task list items (- [ ] / - [x]) round-trip instead of having the checkbox escaped and the item text split onto a separate line.
  • Djot display math ($$) is no longer written back as inline math ($).
  • Djot autolinks (<url>) round-trip instead of being expanded to a full [url](url) link.

v0.5.0

Choose a tag to compare

@gimalay gimalay released this 24 Jun 03:37

iwe

Added

  • format config option (markdown | djot, default markdown) selects the document format, with a matching [djot] options table. With format = "djot", iwe reads, normalizes, exports, and creates djot documents and works with .dj files instead of .md.

iwes

Added

  • format = "djot" in the configuration makes the server read, format, and write djot documents and map file URIs using the .dj extension (default remains markdown with .md).

Fixed

  • The server no longer leaks memory as documents are edited and saved; each update used to retain the previous version's graph data, growing memory without bound over a long session.

iwec

Added

  • --host flag sets the address the HTTP transport binds to (default 127.0.0.1); pass --host 0.0.0.0 to accept connections from other machines.
  • format = "djot" in the configuration makes the server read, write, and watch djot .dj documents (default remains markdown with .md).

Fixed

  • The server no longer leaks memory while watching a folder; repeatedly saving documents used to grow memory without bound and could exhaust RAM and swap over a long session.

liwe

Added

  • FormatOptions (Markdown(MarkdownOptions) | Djot(DjotOptions)) bundles the document format with its formatting options; it is what the graph reads and writes through. Graph::new_with_options, from_state, from_path, and import accept impl Into<FormatOptions>. DjotReader/DjotWriter parse and serialize djot documents so the graph round-trips a .dj document back to djot, and Configuration gains a top-level format selector, a djot: DjotOptions table, and Configuration::format_options().
  • Inline and DocumentInline gain Span, Mark, Insert, Delete, and Symbol variants, and an inline::Attributes type, so djot's bracketed attribute spans, highlight/insert/delete marks, and symbols round-trip losslessly through the graph.

Changed

  • NodeIter::to_text(parent, &FormatOptions) replaces the markdown-specific to_markdown, serializing to whichever format the FormatOptions carries.
  • Key::to_path and the fs discovery and write helpers (walk_md_paths, new_for_path, write_file, write_store_at_path) now take a Format so document files use the configured extension (.md or .dj).

Fixed

  • Updating or removing a document now reclaims the graph nodes, lines, and reference-index entries that belonged to its previous version, so a long-lived Graph no longer grows without bound as the same documents are edited over and over.

v0.4.0

Choose a tag to compare

@gimalay gimalay released this 22 Jun 18:11

iwe

Fixed

  • Normalization now preserves escaped Markdown literals (such as \*text\*, a leading \#, or \[label\](url)) instead of dropping the backslashes and re-interpreting the text as live markup.

iwes

Fixed

  • Documents with Windows line endings are no longer stripped of their frontmatter when edited or saved, and code action ranges no longer drift one column per line on such documents.
  • Renaming a link target in a document inside a subfolder no longer deletes the target file and replaces it with an empty one; the new file is written to the correct folder with the original content, and backlinks are updated to a valid path.
  • Formatting and code actions no longer turn escaped literal text into live Markdown; an escaped \*text\*, \#, or \[label\](url) keeps its escapes, and a list item written as \[ \] is no longer rewritten into a task checkbox.

iwec

Added

  • --transport flag selects how the server is served: stdio (default, unchanged) or http. With --transport http the server listens for Streamable HTTP connections at http://127.0.0.1:<port>/mcp, with --port setting the port (default 8000). The server speaks plain HTTP and binds to localhost only; put a reverse proxy in front of it for TLS or remote access.

liwe

Fixed

  • MarkdownReader now normalizes Windows line endings before parsing, so documents with \r\n line endings keep their frontmatter and report correct positions (previously frontmatter was dropped and positions drifted one column per line).
  • The Markdown writer now re-escapes special characters in text, so escaped literals such as \*text\*, a leading \#, \[label\](url), and 1\. survive normalization instead of turning back into emphasis, headings, links, or list markers; inline code containing a backtick is fenced with enough backticks to render intact, and a list item written as an escaped \[ \] is no longer mistaken for a task checkbox.