[clang-doc] Add a Markdown parser by evelez7 · Pull Request #155887 · llvm/llvm-project

evelez7 · 2025-08-28T17:06:31Z

Initial implementation of a Markdown parser and Markdown AST. Currently, the parser only recognizes emphasis and strong syntax. It also only creates paragraph blocks. The AST is allocated via a BumpPtrAllocator. It attempts to conform to the CommonMark standard

The parser ingests lines and creates preprocessing LineNodes that later are converted to Nodes that form the AST.

evelez7 · 2025-08-28T17:06:49Z

[clang-doc] Add a Markdown parser #155887 👈 (View in Graphite)
main

This stack of pull requests is managed by Graphite. Learn more about stacking.

github-actions · 2025-08-28T17:11:22Z

✅ With the latest revision this PR passed the C/C++ code formatter.

ilovepi · 2025-08-29T18:21:24Z

+  /// @param IdxOrigin the index of * or _ that might start a delimiter run.
+  /// @return A pair denoting the type of run and the index where the run stops
+  std::pair<std::optional<DelimiterContext>, size_t>
+  processDelimiters(SmallString<64> &Line, const size_t &Origin = 0);


Suggested change

/// @param IdxOrigin the index of * or _ that might start a delimiter run.

/// @return A pair denoting the type of run and the index where the run stops

std::pair<std::optional<DelimiterContext>, size_t>

processDelimiters(SmallString<64> &Line, const size_t &Origin = 0);

/// @param Origin the index of * or _ that might start a delimiter run.

/// @return A pair denoting the type of run and the index where the run stops

std::pair<std::optional<DelimiterContext>, size_t>

processDelimiters(SmallString<64> &Line, const size_t &Origin = 0);

The doc name seems stale. I do wonder if you should just use Start or StartIdx, though.

ilovepi · 2025-08-29T18:23:50Z

+  return Result;
+}
+
+void MarkdownParser::parseLine(SmallString<64> &Line, Node *Current) {


can line also just be StringRef here? maybe I missed it, but I don't see you manipulating it at all.

ilovepi · 2025-08-29T20:39:34Z

+      if (Closer->Length >= 2 && Opener->Length >= 2) {
+        // We have at least one strong node.
+        Closer->Length -= 2;
+        Opener->Length -= 2;
+        Emphasis = new (Arena) Node();
+        Emphasis->Type = MDType::Strong;
+        auto *Child = createTextNode(Text);
+        Child->Parent = Emphasis;
+        Emphasis->Children.push_back(Child);
+      } else if (Closer->Length == 1 && Opener->Length == 1) {
+        Closer->Length -= 1;
+        Opener->Length -= 1;
+        Emphasis = new (Arena) Node();
+        Emphasis->Type = MDType::Emphasis;
+        auto *Child = createTextNode(Text);
+        Child->Parent = Emphasis;
+        Emphasis->Children.push_back(Child);


The difference in these blocks seems to be the MDType, and the length adjustment. maybe just use a helper? or just set those bits on their own to avoid repeating mostly the same code?

ilovepi · 2026-02-26T18:29:40Z

+  char Preceeding = (Origin == 0) ? ' ' : Line[Origin - 1];
+  char Proceeding = (Idx >= Line.size()) ? ' ' : Line[Idx];


Suggested change

char Preceeding = (Origin == 0) ? ' ' : Line[Origin - 1];

char Proceeding = (Idx >= Line.size()) ? ' ' : Line[Idx];

char Leading = (Origin == 0) ? ' ' : Line[Origin - 1];

char Trailing = (Idx >= Line.size()) ? ' ' : Line[Idx];

Maybe this is a bit easier to follow/not accidentally typo? happy with any name, but these are really just 1 char apart right now...

ilovepi · 2026-02-26T18:37:10Z

+  if (LeftFlanking && RightFlanking)
+    return {DelimiterContext{LeftFlanking, RightFlanking, true, true}, Idx};
+  if (LeftFlanking)
+    return {DelimiterContext{LeftFlanking, RightFlanking, true, false}, Idx};
+  if (RightFlanking)
+    return {DelimiterContext{LeftFlanking, RightFlanking, false, true}, Idx};
+  return {std::nullopt, 0};


Suggested change

if (LeftFlanking && RightFlanking)

return {DelimiterContext{LeftFlanking, RightFlanking, true, true}, Idx};

if (LeftFlanking)

return {DelimiterContext{LeftFlanking, RightFlanking, true, false}, Idx};

if (RightFlanking)

return {DelimiterContext{LeftFlanking, RightFlanking, false, true}, Idx};

return {std::nullopt, 0};

if(!LeftFlanking && !RightFlanking)

return {std::nullopt, 0};

return {DelimiterContext{LeftFlanking, RightFlanking, LeftFlanking, RightFlanking}, Idx};

I think this is equivalent, right?

ilovepi · 2026-02-26T18:37:52Z

+  for (const auto *Node : Text) {
+    TextNode->Content.append(Node->Content);
+  }


Suggested change

for (const auto *Node : Text) {

TextNode->Content.append(Node->Content);

}

for (const auto *Node : Text)

TextNode->Content.append(Node->Content);

evelez7 · 2026-03-05T18:49:56Z

I've rewritten a large part of the initial implementation idea. The biggest change is that I've separated the Markdown AST, parser, and HTML generation (which doesn't exist here yet). The parser is independent of the AST. The AST owns the arena and string saver. The motivation behind the separation is that a single parser object could be reused for different ASTs in the future or several ASTs could be parsed in parallel. Then, these ASTs could be fed to different engines and not just HTML. You can see an example of the usage in the tests.

There's still a lot of work to be done here but I thought some initial feedback on the design direction could be helpful, especially the problems with arena allocations and memory leaks that I had last time. It might also motivate discussions for the recent RFC or help GSoC applicants.

github-actions · 2026-03-05T18:59:24Z

🐧 Linux x64 Test Results

The build failed before running any tests. Click on a failure below to see the details.

tools/clang/tools/extra/clang-doc/CMakeFiles/obj.clangDoc.dir/MDParser.cpp.o

FAILED: tools/clang/tools/extra/clang-doc/CMakeFiles/obj.clangDoc.dir/MDParser.cpp.o
sccache /opt/llvm/bin/clang++ -D_DEBUG -D_GLIBCXX_ASSERTIONS -D_GLIBCXX_USE_CXX11_ABI=1 -D_GNU_SOURCE -D_LIBCPP_HARDENING_MODE=_LIBCPP_HARDENING_MODE_EXTENSIVE -D__STDC_CONSTANT_MACROS -D__STDC_FORMAT_MACROS -D__STDC_LIMIT_MACROS -I/home/gha/actions-runner/_work/llvm-project/llvm-project/build/tools/clang/tools/extra/clang-doc -I/home/gha/actions-runner/_work/llvm-project/llvm-project/clang-tools-extra/clang-doc -I/home/gha/actions-runner/_work/llvm-project/llvm-project/clang/include -I/home/gha/actions-runner/_work/llvm-project/llvm-project/build/tools/clang/include -I/home/gha/actions-runner/_work/llvm-project/llvm-project/build/include -I/home/gha/actions-runner/_work/llvm-project/llvm-project/llvm/include -gmlt -fPIC -fno-semantic-interposition -fvisibility-inlines-hidden -Werror -Werror=date-time -Werror=unguarded-availability-new -Wall -Wextra -Wno-unused-parameter -Wwrite-strings -Wcast-qual -Wmissing-field-initializers -pedantic -Wno-long-long -Wc++98-compat-extra-semi -Wimplicit-fallthrough -Wcovered-switch-default -Wno-noexcept-type -Wnon-virtual-dtor -Wdelete-non-virtual-dtor -Wsuggest-override -Wstring-conversion -Wno-pass-failed -Wmisleading-indentation -Wctad-maybe-unsupported -fdiagnostics-color -ffunction-sections -fdata-sections -fno-common -Woverloaded-virtual -Wno-nested-anon-types -O3 -DNDEBUG -std=c++17 -UNDEBUG -fno-exceptions -funwind-tables -fno-rtti -MD -MT tools/clang/tools/extra/clang-doc/CMakeFiles/obj.clangDoc.dir/MDParser.cpp.o -MF tools/clang/tools/extra/clang-doc/CMakeFiles/obj.clangDoc.dir/MDParser.cpp.o.d -o tools/clang/tools/extra/clang-doc/CMakeFiles/obj.clangDoc.dir/MDParser.cpp.o -c /home/gha/actions-runner/_work/llvm-project/llvm-project/clang-tools-extra/clang-doc/MDParser.cpp
/home/gha/actions-runner/_work/llvm-project/llvm-project/clang-tools-extra/clang-doc/MDParser.cpp:229:15: error: enumeration value 'Document' not handled in switch [-Werror,-Wswitch]
229 |       switch (CurrentBlockType) {
|               ^~~~~~~~~~~~~~~~
1 error generated.

If these failures are unrelated to your changes (for example tests are broken or flaky at HEAD), please open an issue at https://github.com/llvm/llvm-project/issues and add the infrastructure label.

ilovepi

Took a first pass. Will give more feedback tomorrow.

ilovepi · 2026-03-06T00:53:28Z

+static bool isEmpty(StringRef Line) {
+  if (Line.size() == 0)
+    return true;
+  if (Line.size() == 1 && Line[0] == ' ')
+    return true;
+  for (auto &Char : Line) {
+    if (Char != ' ')
+      return false;
+  }
+  return true;
+}


I think I don't follow. Can you get something similar with

Suggested change

static bool isEmpty(StringRef Line) {

if (Line.size() == 0)

return true;

if (Line.size() == 1 && Line[0] == ' ')

return true;

for (auto &Char : Line) {

if (Char != ' ')

return false;

}

return true;

}

static bool isEmpty(StringRef Line) {

return Line.trim().empty();

}

ilovepi · 2026-03-06T00:54:59Z

+}
+
+static MDType determineBlockType(StringRef Line) {
+  StringRef TrimmedLine = Line.ltrim(' ');


What about other whitespace characters? like \t? Also, is ther a particular spec I should look at while reviewing?

ilovepi · 2026-03-06T00:55:52Z

+    return MDType::Paragraph;
+  case '>':
+    return MDType::Paragraph;
+  default:
+    return MDType::Paragraph;


Are these all intended to be the same? If so, why not fall through all of them?

ilovepi · 2026-03-06T00:58:41Z

+StringRef ASTContext::intern(std::string &String) {
+  return SSaver.save(String);
+}
+StringRef ASTContext::intern(StringRef String) { return SSaver.save(String); }


Do yo need these overloads? I thought Twine would cover all of them...

ilovepi · 2026-03-06T00:59:50Z

+std::optional<std::pair<DelimiterContext, size_t>>
+MarkdownParser::processDelimiters(StringRef Line, const size_t &Start) {
+  size_t Idx = Start;
+  while (Idx < Line.size() && Line[Idx] == Line[Start]) {


you can avoid the call to size() in the loop header.

evelez7 force-pushed the users/evelez7/clang-doc-markdown-parser branch from c9e91c7 to 5a66d89 Compare August 28, 2025 17:08

evelez7 requested review from ilovepi and petrhosek August 28, 2025 17:09

evelez7 changed the title ~~[clang-doc] markdown parser~~ [clang-doc] add Markdown parser implementation Aug 28, 2025

evelez7 changed the title ~~[clang-doc] add Markdown parser implementation~~ [clang-doc] add Markdown parser Aug 28, 2025

ilovepi reviewed Aug 29, 2025

View reviewed changes

ilovepi reviewed Feb 26, 2026

View reviewed changes

[clang-doc] markdown parser

571e578

evelez7 force-pushed the users/evelez7/clang-doc-markdown-parser branch from 5a66d89 to 571e578 Compare March 5, 2026 18:45

evelez7 changed the title ~~[clang-doc] add Markdown parser~~ [clang-doc] Add a Markdown parser Mar 5, 2026

evelez7 requested a review from ilovepi March 5, 2026 18:46

ilovepi reviewed Mar 6, 2026

View reviewed changes

github-actions Bot deleted the users/evelez7/clang-doc-markdown-parser branch September 7, 2026 08:10

		char Preceeding = (Origin == 0) ? ' ' : Line[Origin - 1];
		char Proceeding = (Idx >= Line.size()) ? ' ' : Line[Idx];

Conversation

evelez7 commented Aug 28, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

evelez7 commented Aug 28, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Aug 28, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

evelez7 commented Mar 5, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Mar 5, 2026

🐧 Linux x64 Test Results

Uh oh!

ilovepi left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

This branch has not been deployed

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

evelez7 commented Aug 28, 2025 •

edited

Loading

evelez7 commented Aug 28, 2025 •

edited

Loading

github-actions Bot commented Aug 28, 2025 •

edited

Loading

evelez7 commented Mar 5, 2026 •

edited

Loading