Explicit support for OKF and/or ICM

[rtbs-dev]

Hi there! Love iwe. Wondering if there's a way to either add support or mention in the docs how to interop with the more formalized file hierarchy/specifications being put out recently, specifically OKF from google, and/or the ICM system an independent researcher come up with previous to that.

I have a feeling iwe is uniquely situated to create these kinds of structures, so explicit support or mention of these frameworks might be really helpful!

Thanks!

[gimalay]

Hi @rtbs-dev !

Thanks for bringing this up. It's a very interesting direction to explore and it is very well aligned with the project goals and intended use cases. I'll take a look into these and see what is the best way to support OKF/ICM in iwe.

Please let me know If you have ideas/recommendations on how this kind of integration/support may look like.

Thanks!

[rtbs-dev]

Wonderful! I'll give this some serious thought, for sure. I have a few ideas, off the top:

Metadata

The first thing I can think of is first-class support for manipulating/reading/accessing document metadata. I've been involved in an excellent discussion on this topic here, though that was djot-specific. (sidenote, I think there are peculiarities of the djot design that lend itself even better to AI parsing and interaction than MD, so maybe that's a future feature...)

Obviously it's one way to do the obsidian thing and build a specific syntax and datastructure/query for metadata, but I wonder if wee could bypass that by allowing the user/ai system to make use of cli tools like yq and similar to let the user put arbitrary strucutred data in a meta block (defaulting to yaml, as in OKF spec).

Secondly, letting the generalization away from yaml-only let's iwe support literate programming blocks (like org-babel), which let's users+AI tools store relevant scripts/tools inside the iwe mdd files, as tagged code "blocks". The tooling for making these as a human already exists well for e.g. myst, jupytext, rmarkdown, qmd files.

Folder Hierarchy Refactors.

Most of the OKF/ICM systems ideas are just unix-style folders, given semantic meaning.
I think it would be useful to let the iwe refactor commands take parameters to place sections into subfolders, not just external files. If following ICM, that would also allow users to provide a numbered folder prefix, which auto-sorts the workflow position. letting users re-sort the workflow with iwe could be useful as well, e.g. if I swap two headings in an index.md, then iwes could update the number prefixes on the corresponding subfolders.

This gets to the broader issue of _routing docs, where a "progressive disclosure" system (e.g. I've built one here as an agent skill) needs to keep context low and still access stuff on-demand. ICM is very concerned with this. I think giving agents cli tooling via icm to create, edit, and navigate these hieraarchy routers more efficiently would be phenomenal.

[gimalay]

Thank you for the ideas! I'll start a few threads below to address each point.

[gimalay]

This gets to the broader issue of _routing docs, where a "progressive disclosure" system (e.g. I've built one here as an agent skill) needs to keep context low and still access stuff on-demand. ICM is very concerned with this. I think giving agents cli tooling via icm to create, edit, and navigate these hieraarchy routers more efficiently would be phenomenal.

Could you please explain the idea of hierarchy routers a bit more? Ideally with an example?

[gimalay]

Most of the OKF/ICM systems ideas are just unix-style folders, given semantic meaning.
I think it would be useful to let the iwe refactor commands take parameters to place sections into subfolders, not just external files

I agree, this makes a lot of sense. Currently extract refactor places the files in the same directory. I can add a parameter to the extract action in the configuration which will control if sub-directory needs to be created for extracted files.

[gimalay]

If following ICM, that would also allow users to provide a numbered folder prefix, which auto-sorts the workflow position. letting users re-sort the workflow with iwe could be useful as well, e.g. if I swap two headings in an index.md, then iwes could update the number prefixes on the corresponding subfolders.

This is an interesting use-case, I wonder how we can represent this in a generic way (some sort of configuration/action) and not make it ICM specific.

[rtbs-dev]

Not sure. Maybe we let the user handle optional numbering on there end, And instead we use dot notation (like yq/jq query syntax)... If a header has a dot in it, we default to a nested folder. Similarly for subheaders.

E.g.

# 01_workflow_start.resources
Contents 

Pulling out that section could put a file at ./01_workflow_start/resources.md

Then, for something like

# My header 

## .subheader
Contents

Pulling out the lvl 2 subheader could add a file at ./my_header/subheader.md

Of course that kinda makes possible renders of the markdown files a bit ugly, using dot notation. Another way is to just use header hierarchy, where a new code action or something makes the refactor behavior nest subsections into correspondingly nested folder hierarchy when pulled out.

[rtbs-dev]

For numbering specifically, maybe we just let users config an option for whether or not header order should be preserved in the filesystem (i.e. with numeric prefixes)

[gimalay]

The first thing I can think of is first-class support for manipulating/reading/accessing document metadata. I've been involved in an excellent discussion on this topic jgm/djot#35 (comment), though that was djot-specific. (sidenote, I think there are peculiarities of the djot design that lend itself even better to AI parsing and interaction than MD, so maybe that's a future feature...)

iwe has a query language which supports YAML frontmatter metadata based queries and updates.

In the discussion you linked the general ideal is to a support of multiple metadata blocks per document and different metadata languages (such as TOML). I think it's all possible as long as this functionality is supported by the format. I'm actually thinking about adding multi format support for iwe so it can work with both Markdown and Djot formats.

[rtbs-dev]

Djot support would be absolutely amazing. Very excited to use it for my knowledge bases

[gimalay]

Here it is: #323

[gimalay]

Obviously it's one way to do the obsidian thing and build a specific syntax and datastructure/query for metadata, but I wonder if wee could bypass that by allowing the user/ai system to make use of cli tools like yq and similar to let the user put arbitrary strucutred data in a meta block (defaulting to yaml, as in OKF spec).

yq can already parse and query frontmatter from the Markdown documents using --frontmatter parameter. How would see iwe support for this?

[rtbs-dev]

Yes this is great, I guess my preference is that iwe natively gets ai agents using mcp to use yq when available to handle all frontmatter writes/reads, rather than attempting to parse itself. I've had a lot of issues when I don't configure agents to use tooling like this. Plus it saves a lot of context (not needing to read the file just to get/edit is metadata)

[gimalay]

iwe itself parses the frontmatter and support reach queries agains it. You can find more details here: https://www.reddit.com/r/iwe/comments/1t1m93k/markdown_as_a_database_memory_for_humans_and_ai/

Speaking about yq, if agent is working in the directly with the files it can use yq directly and don't need anything from iwe CLI to support this. Am I missing something?

For the MCP, agent can use the built in queries. I'm not sure if it makes sense to expose yq through iwe MCP.

[gimalay]

Secondly, letting the generalization away from yaml-only let's iwe support literate programming blocks (like org-babel), which let's users+AI tools store relevant scripts/tools inside the iwe mdd files, as tagged code "blocks". The tooling for making these as a human already exists well for e.g. myst, jupytext, rmarkdown, qmd files.

This would be awesome. As far as I know, rmakdown adds special syntax which is not Markdown compatible. The key question here is how to support this kind of functionality and stay Markdown compatible.

[rtbs-dev]

Not sure, my best thought right now is to use pandoc (and consequently quarto and myst) syntax for block metadata like {.code}, etc. I know tools like logseq and org-noter do something like this for managing pdf annotations too, and Ive been hoping for an lsp that does the same for markdown.