Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Fix buffer overflows and other errors found with fuzzying. #168

Merged
merged 9 commits into from
Jan 6, 2022
Merged

Fix buffer overflows and other errors found with fuzzying. #168

merged 9 commits into from
Jan 6, 2022

Conversation

TCKnet
Copy link
Contributor

@TCKnet TCKnet commented Aug 29, 2021

No description provided.

TCKnet and others added 2 commits August 29, 2021 12:50
Fix buffer overflow on input found with fuzzying.
c85694b
@TCKnet
Update md4c.c
c3c900b 
Fix an incorrect assertion in md_analyze_permissive_email_autolink, which fails on [[<4@w>]].
@codecov
Copy link

codecov bot commented Sep 2, 2021 •
edited

Codecov Report

Merging #168 (42c1bb5) into master (7f05330) will increase coverage by 0.03%.
The diff coverage is 94.11%.

Impacted file tree graph

@@            Coverage Diff             @@
##           master     #168      +/-   ##
==========================================
+ Coverage   94.33%   94.36%   +0.03%     
==========================================
  Files           3        3              
  Lines        3088     3087       -1     
==========================================
  Hits         2913     2913              
+ Misses        175      174       -1
Impacted Files Coverage Δ
src/md4c.c 94.27% <94.11%> (+0.03%) ⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 7f05330...42c1bb5. Read the comment docs.

@TCKnet TCKnet changed the title Fix buffer overflow on input found with fuzzying. Fix buffer overflow on input found with fuzzying, fix incorrect assertion. Sep 2, 2021
@TCKnet TCKnet changed the title Fix buffer overflow on input found with fuzzying, fix incorrect assertion. Fix buffer overflows and other errors found with fuzzying. Oct 6, 2021
@TCKnet
Update md4c.c
daa2f8c 
More fixes.
@TCKnet
Update md4c.c
42c1bb5 
More fixes: one endless loop and few assertion failing on fuzzed inputs.
@mity mity merged commit f436c30 into mity:master Jan 6, 2022
@mity
Copy link
Owner

mity commented Jan 6, 2022

Thanks.

(And sorry for the long reaction time.)

mity added a commit that referenced this pull request Jan 10, 2024
@mity
Revert most of PR #168
ebbb12e 
i.e of the commit f436c30.

It added bunch of checks all over the place, but most of them
shouldn't be needed: If they are true, our internal state is
already broken. In other words, those checks are hiding real bugs
and making debugging harder.

Hopefully the underlying bugs are already fixed in some of previous
commits addressing some fuzzing issues, like these:

 * d775b51
 * c6942ef
netbsd-srcmastr pushed a commit to NetBSD/pkgsrc that referenced this pull request Jan 25, 2024
md4c: update to 0.5.1.
d2b981f 
## Version 0.5.1

Changes:

 * LaTeX math extension (`MD_FLAG_LATEXMATHSPANS`) now requires that opener
   mark is not immediately preceded with alpha-numeric character and similarly
   that closer mark is not immediately followed with alpha-numeric character.

   So for example `foo$ x + y = z $` is not recognized as LaTeX equation
   anymore because there is no space between `foo` and the opening `$`.

 * Table extension (`MD_FLAG_TABLES`) now recognizes only tables with no more
   than 128 columns. This limit has been imposed to prevent a pathological
   case of quadratic output size explosion which could be used as DoS attack
   vector.

 * We are now more strict with `MD_FLAG_PERMISSIVExxxAUTOLINKS` family of
   extensions with respect to non-alphanumeric characters, with the aim to
   mitigate false positive detections.

   Only relatively few selected non-alphanumeric are now allowed in permissive
   e-mail auto-links (`MD_FLAG_PERMISSIVEEMAILAUTOLINKS`):
     - `.`, `-`, `_`, `+` in user name part of e-mail address; and
     - `.`, `-`, `_` in host part of the e-mail address.

   Similarly for URL and e-mail auto-links (`MD_FLAG_PERMISSIVEURLAUTOLINKS` and
   `MD_FLAG_PERMISSIVEWWWAUTOLINKS`):
     - `.`, `-`, `_` in host part of the URL;
     - `/`, `.`, `-`, `_` in path part of the URL;
     - `&`, `.`, `-`, `+`, `_`, `=`, `(`, `)` in the query part of the URL
       (additionally, if present, `(` and `)` must form balanced pairs); and
     - `.`, `-`, `+`, `_` in the fragment part of the URL.

   Furthermore these characters (with some exceptions like where they serve as
   delimiter characters, e.g. `/` for paths) are generally accepted only when
   an alphanumeric character both precedes and follows them (i.e. these cannot
   be "stacked" together).

Fixes:

 * Fix several bugs where we haven't properly respected already resolved spans
   of higher precedence level in handling of permissive auto-links extensions
   (family of `MD_FLAG_PERMISSIVExxxAUTOLINKS` flags), LaTeX math extension
   (`MD_FLAG_LATEXMATHSPANS`) and wiki-links extension (`MD_FLAG_WIKILINKS`)
   of the form `[[label|text]]` (with pipe `|`). In some complex cases this
   could lead to invalid internal parser state and memory corruption.

   Identified with [OSS-Fuzz](https://github.com/google/oss-fuzz).

 * [#222](mity/md4c#222):
   Fix strike-through extension (`MD_FLAG_STRIKETHROUGH`) which did not respect
   same rules for pairing opener and closer marks as other emphasis spans.

 * [#223](mity/md4c#223):
   Fix incorrect handling of new-line character just at the beginning and/or
   end of a code span where we were not following CommonMark specification
   requirements correctly.


## Version 0.5.0

Changes:

 * Changes mandated by CommonMark specification 0.30.

   Actually there are only very minor changes to recognition of HTML blocks:

   - The tag `<textarea>` now triggers HTML block (of type 1 as per the
     specification).

   - HTML declaration (HTML block type 4) is not required to begin with an
     upper-case ASCII character after the `<!`. Any ASCII character is now
     allowed. Also it now doesn't require a whitespace before the closing `>`.

   Other than that, the newest specification mainly improves test coverage and
   clarifies its wording in some cases, without affecting the implementation.

   Refer to [CommonMark
   0.30 notes](https://github.com/commonmark/commonmark-spec/releases/tag/0.30)
   for more info.

 * Make Unicode-specific code compliant to Unicode 15.1.

 * Update list of entities known to the HTML renderer from
   https://html.spec.whatwg.org/entities.json.

New Features:

 * Add extension allowing to treat all soft break as hard ones. It has to be
   explicitly enabled with `MD_FLAG_HARD_SOFT_BREAKS`.

   Contributed by [l-m](https://github.com/l1mey112).

 * Structure `MD_SPAN_A_DETAIL` now has a new member `is_autolink`.

   Contributed by [Jens Alfke](https://github.com/snej).

 * `md2html` utility now supports command line options `--html-title` and
   `--html-css`.

   Contributed by [Andreas Baumann](https://github.com/andreasbaumann).

Fixes:

 * [#163](mity/md4c#163):
   Make HTML renderer to emit `'\n'` after the root tag when in the XHTML mode.

 * [#165](mity/md4c#165):
   Make HTML renderer not to percent-encode `'~'` in URLs. Although it does
   work, it's not needed, and it can actually be confusing with URLs such as
   `http://www.example.com/~johndoe/`.

 * [#167](mity/md4c#167),
   [#168](mity/md4c#168):
   Fix multiple instances of various buffer overflow bugs, found mostly using
   a fuzz testing. Contributed by [dtldarek](https://github.com/dtldarek) and
   [Thierry Coppey](https://github.com/TCKnet).

 * [#169](mity/md4c#169):
   Table underline now does not require 3 characters per table column anymore.
   One dash (optionally with a leading or tailing `:` appended or prepended)
   is now sufficient. This improves compatibility with the GFM.

 * [#172](mity/md4c#172):
   Fix quadratic time behavior caused by unnecessary lookup for link reference
   definition even if the potential label contains nested brackets.

 * [#173](mity/md4c#173),
   [#174](mity/md4c#174),
   [#212](mity/md4c#212),
   [#213](mity/md4c#213):
   Multiple bugs identified with [OSS-Fuzz](https://github.com/google/oss-fuzz)
   were fixed.

 * [#190](mity/md4c#190),
   [#200](mity/md4c#200),
   [#201](mity/md4c#201):
   Multiple fixes of incorrect interactions of indented code block with a
   preceding block.

 * [#202](mity/md4c#202):
   We were not correctly calling `enter_block()` and `leave_block()` callbacks
   if multiple HTML blocks followed one after another; instead previously
   such blocks were merged into one.

   (This may likely impact only applications interested in Markdown's AST,
   and not just converting Markdown to other formats like HTML.)

 * [#210](mity/md4c#210):
   The `md2html` utility now handles nested images with optional titles
   correctly.

 * [#214](mity/md4c#214):
   Tags `<h2>` ... `<h6>` incorrectly did not trigger HTML block.

 * [#215](mity/md4c#215):
   The parser incorrectly did not accept optional tabs after setext header
   underline.

 * [#217](mity/md4c#217):
   The parser incorrectly resolved emphasis in some situations, if the emphasis
   marks were enclosed by punctuation characters.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
2 participants
@TCKnet @mity