fix(adapters): build a TOC for documents larger than 400 KB - #86
Merged
Conversation
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## master #86 +/- ##
==========================================
- Coverage 83.64% 82.34% -1.30%
==========================================
Files 35 37 +2
Lines 954 1184 +230
==========================================
+ Hits 798 975 +177
- Misses 101 132 +31
- Partials 55 77 +22 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #25.
GitHub's
/markdown/rawAPI refuses to render a payload larger than 400 KB, sogh-md-tocfailed on any local document above that size. The same call backs STDINand remote raw Markdown, so all three paths were affected.
What changed
internal/core/mdsplitcuts a document into chunks below the limit. A chunknormally ends at a blank line outside a fenced code block, an HTML comment or a raw
HTML block - that is where GitHub ends a block too, so every chunk parses the way the
same lines parse inside the whole document. When no such point fits, the cut falls on
a line boundary and the open block is closed at the end of the chunk and reopened at
the start of the next one, which keeps
#comments inside a shell listing frombecoming headings. A setext underline is never separated from its text, and a single
line larger than the limit is reported as an error instead of being broken silently.
HTMLConverter.Convertnow returns one HTML string per request. Documents up to384 KB take the same single request as before; larger ones are split and posted chunk
by chunk through the new
HttpPostBody/RemotePoster.PostBody. The threshold isbelow 400 KB whether GitHub means 400*1024 or 400000 bytes.
Generator.Grabbecame variadic and renumbers anchors. GitHub appends-Nto theNth repeat of a slug within one rendered document, so a document converted in pieces
restarts that numbering in every piece. The base slug is recovered with the same
counter GitHub used inside the chunk, then numbered again against a counter that spans
the whole document. With a single input nothing is renumbered and behaviour is
unchanged.
Verification
Checked against the live API with a 451 KB document that previously failed: 798 TOC
entries, anchors
usagethroughusage-398running unbroken across chunk boundaries,no duplicate anchors, and no phantom headings from the fenced code blocks.
golangci-lint run ./...is clean andgo test -race ./...passes. The README gaineda
Large documentssection, soe2e-tests/want.mdandwant3.mdwere regenerated;the remote e2e steps read README.md from
master, so run them on this branch withmake e2e E2E_REF=$(git rev-parse HEAD).Trade-offs
Documented in the README next to the token section.
# [Project][1]) whose definitionlands in another chunk renders literally and its slug differs from a single-request
render. Rare enough that collecting every definition into every chunk was not worth
the cost.