Skip to content

fix(adapters): build a TOC for documents larger than 400 KB - #86

Merged
ekalinin merged 5 commits into
masterfrom
fix/large-markdown-chunking
Aug 20, 2026
Merged

fix(adapters): build a TOC for documents larger than 400 KB#86
ekalinin merged 5 commits into
masterfrom
fix/large-markdown-chunking

Conversation

@ekalinin

Copy link
Copy Markdown
Owner

Fixes #25.

GitHub's /markdown/raw API refuses to render a payload larger than 400 KB, so
gh-md-toc failed on any local document above that size. The same call backs STDIN
and remote raw Markdown, so all three paths were affected.

What changed

internal/core/mdsplit cuts a document into chunks below the limit. A chunk
normally ends at a blank line outside a fenced code block, an HTML comment or a raw
HTML block - that is where GitHub ends a block too, so every chunk parses the way the
same lines parse inside the whole document. When no such point fits, the cut falls on
a line boundary and the open block is closed at the end of the chunk and reopened at
the start of the next one, which keeps # comments inside a shell listing from
becoming headings. A setext underline is never separated from its text, and a single
line larger than the limit is reported as an error instead of being broken silently.

HTMLConverter.Convert now returns one HTML string per request. Documents up to
384 KB take the same single request as before; larger ones are split and posted chunk
by chunk through the new HttpPostBody / RemotePoster.PostBody. The threshold is
below 400 KB whether GitHub means 400*1024 or 400000 bytes.

Generator.Grab became variadic and renumbers anchors. GitHub appends -N to the
Nth repeat of a slug within one rendered document, so a document converted in pieces
restarts that numbering in every piece. The base slug is recovered with the same
counter GitHub used inside the chunk, then numbered again against a counter that spans
the whole document. With a single input nothing is renumbered and behaviour is
unchanged.

Verification

Checked against the live API with a 451 KB document that previously failed: 798 TOC
entries, anchors usage through usage-398 running unbroken across chunk boundaries,
no duplicate anchors, and no phantom headings from the fenced code blocks.

golangci-lint run ./... is clean and go test -race ./... passes. The README gained
a Large documents section, so e2e-tests/want.md and want3.md were regenerated;
the remote e2e steps read README.md from master, so run them on this branch with
make e2e E2E_REF=$(git rev-parse HEAD).

Trade-offs

  • A large document costs one API request per chunk, so the rate limit arrives sooner.
    Documented in the README next to the token section.
  • A heading that uses a link reference definition (# [Project][1]) whose definition
    lands in another chunk renders literally and its slug differs from a single-request
    render. Rare enough that collecting every definition into every chunk was not worth
    the cost.

@codecov

codecov Bot commented Aug 20, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 78.60082% with 52 lines in your changes missing coverage. Please review.
✅ Project coverage is 82.34%. Comparing base (914bbc1) to head (5c11630).

Files with missing lines Patch % Lines
internal/core/mdsplit/mdsplit.go 76.25% 23 Missing and 15 partials ⚠️
internal/adapters/htmlconverter.go 71.42% 5 Missing and 3 partials ⚠️
internal/core/toc/anchors.go 86.66% 2 Missing and 2 partials ⚠️
internal/adapters/remoteposter.go 50.00% 1 Missing and 1 partial ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##           master      #86      +/-   ##
==========================================
- Coverage   83.64%   82.34%   -1.30%     
==========================================
  Files          35       37       +2     
  Lines         954     1184     +230     
==========================================
+ Hits          798      975     +177     
- Misses        101      132      +31     
- Partials       55       77      +22     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@ekalinin
ekalinin merged commit b090af7 into master Aug 20, 2026
3 of 5 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[BUG] Can't work well when the text > 400KB

1 participant