Feature Request: Default Markdown NLWeb Ingestion

[tim13337]

Problem

When ingesting Markdown content into NLWeb (e.g., via RSS feeds or direct file uploads), the absence of a default transformation mechanism leads to:

• Loss of Semantic Meaning: Markdown elements like headings, lists, and code blocks are processed as raw text, missing the chance to map them to Schema.org types (e.g., a setup section as a HowTo type).
• Reduced Vector Search Accuracy: Without structured metadata, vector search fails to disambiguate entities (e.g., “bug fix” as software vs. insect repellent)

Background

At iunera, we’ve encountered similar challenges in past projects involving structured enterprise data and big data processing. Our solution was to enrich Markdown content by creating a default transformation pipeline to convert it into structured data, which proved highly efficient in internal and client projects (we then could mix it up with structured data from other sources jsonld-schemaorg-javatypes).

While previously unpublished, we believe this approach can significantly boost vector search in NLWeb. iunera is generally willing to contribute this solution to NLWeb to enhance its ingestion capabilities.

Impact

NLWeb currently attracts tech-savvy users who often host content in Git repositories with Markdown READMEs. Supporting Markdown ingestion would unlock immense potential to attract early adopters by enabling them to index content in their most commonly used format, thereby broadening NLWeb’s user base and enhancing its conversational capabilities.

Proposed Solution

Add a default Markdown-to-JSON-LD transformation pipeline to NLWeb’s ingestion process, converting Markdown content into Schema.org JSON-LD for improved semantic indexing. This can leverage existing tools like iunera’s json-ld-markdown (demo) as a foundation.

Questions for Discussion

• Is Markdown ingestion within the scope and interest of the NLWeb project?
• Does the community see value in Markdown ingestion, or is this primarily an enterprise need we’ve identified?
• Which Schema.org types (e.g., for comparison tables, headings, lists, code blocks) would have the greatest impact on vector search accuracy?
• Should NLWeb integrate an existing tool like json-ld-markdown or develop a custom pipeline?

Expected Outcome

• A standardized Markdown ingestion pipeline for NLWeb.
• Enhanced vector search accuracy and conversational responses through semantic context (e.g., mapping a setup guide to a HowTo type).
• Community-driven mappings, allowing developers to contribute custom transformations in line with NLWeb’s collaborative ethos.

[rvguha]

Here is the issue with Markdown --> Schema.org.

Markdown is primarily about formatting/presentation. Schema.org is only about semantics. The schema.org types (Event, Person, Product, Recipe) etc. cannot be captured by a direct translation from the markdown. I am hopeful that we can do it by invoking an LLM.

[tim13337]

@rvguha yes- that is the true and problem since all semantic web attempts since the RDF and OWL days.. to get proper data.
What I mean by markdown to Json-ld/schema.org transformation is to at least transform the obvious structures.
I did not find any tool doing at least this before any entity determination.

E.g. it is trivial to detect via rules:

• what a license markdown file is,
• what a howto is,
• what a recipe is
• how a comparison table looks like
  and that how the subsections of either look like in most cases.

When someone uses NLWeb for own data they commonly know how they structured their semantic information markdown.

For the entities detection LLMs can help.
However, a low hanging fruit for is pre-processing in just simple rule based mapping and allowing semantic annotations by the transformers to encourage more semantics in markdown files. That is desperately required.

Key improvement for entity mining with LLMs is that there is over time more "training"/ "labeled schema.org data" there that we can do that matching better.
This is where the past RDF and OWL also failed on...

What would really help research and practice in entity mining would be the possibility to simple hint "entity" detection in markdown files (or any other files).

One approach can be simple annotations after a word with @.

E.g.:
[NLWeb]@{SoftwareApplication,@id=#nlweb} leverages JSON-LD for conversational AI, improving response accuracy.
-> Simple tagging with an @ and Schema.org properties.

Key thing why I see markdown as so interesting is because it is effortless to create.

So the NLWeb Team would propose some markdown annotations for ingestion that enables easy entity annotations, that would be a hug thing to train/improve entity detection in general..