SyntaxHighlighter Code block: improper encoding/decoding of HTML entities #152

alexsanford · 2020-06-12T15:30:44Z

Originally reported internally by @tbradsha

Steps to reproduce

Create a SyntaxHighlighter Code block, any code language.
Add the following contents (pulled from http://www.htmlhelp.org/reference/html40/entities/):

+-----------------------------------------------------------------------+----------+---------+--------+---------------------------+---+---+
|                               Character                               |  Entity  | Decimal |  Hex   | Rendering in Your Browser |   |   |
+-----------------------------------------------------------------------+----------+---------+--------+---------------------------+---+---+
| Entity                                                                | Decimal  | Hex     |        |                           |   |   |
| no-break space = non-breaking space                                   | &nbsp;   | &#160;  | &#xA0; |                           |   |   |
| inverted exclamation mark                                             | &iexcl;  | &#161;  | &#xA1; | ¡                         | ¡ | ¡ |
| cent sign                                                             | &cent;   | &#162;  | &#xA2; | ¢                         | ¢ | ¢ |
| pound sign                                                            | &pound;  | &#163;  | &#xA3; | £                         | £ | £ |
| currency sign                                                         | &curren; | &#164;  | &#xA4; | ¤                         | ¤ | ¤ |
| yen sign = yuan sign                                                  | &yen;    | &#165;  | &#xA5; | ¥                         | ¥ | ¥ |
| broken bar = broken vertical bar                                      | &brvbar; | &#166;  | &#xA6; | ¦                         | ¦ | ¦ |
| section sign                                                          | &sect;   | &#167;  | &#xA7; | §                         | § | § |
| diaeresis = spacing diaeresis                                         | &uml;    | &#168;  | &#xA8; | ¨                         | ¨ | ¨ |
| copyright sign                                                        | &copy;   | &#169;  | &#xA9; | ©                         | © | © |
| feminine ordinal indicator                                            | &ordf;   | &#170;  | &#xAA; | ª                         | ª | ª |
| left-pointing double angle quotation mark = left pointing guillemet   | &laquo;  | &#171;  | &#xAB; | «                         | « | « |
| not sign                                                              | &not;    | &#172;  | &#xAC; | ¬                         | ¬ | ¬ |
| soft hyphen = discretionary hyphen                                    | &shy;    | &#173;  | &#xAD; |                           |   |   |
| registered sign = registered trade mark sign                          | &reg;    | &#174;  | &#xAE; | ®                         | ® | ® |
| macron = spacing macron = overline = APL overbar                      | &macr;   | &#175;  | &#xAF; | ¯                         | ¯ | ¯ |
| degree sign                                                           | &deg;    | &#176;  | &#xB0; | °                         | ° | ° |
| plus-minus sign = plus-or-minus sign                                  | &plusmn; | &#177;  | &#xB1; | ±                         | ± | ± |
| superscript two = superscript digit two = squared                     | &sup2;   | &#178;  | &#xB2; | ²                         | ² | ² |
| superscript three = superscript digit three = cubed                   | &sup3;   | &#179;  | &#xB3; | ³                         | ³ | ³ |
| acute accent = spacing acute                                          | &acute;  | &#180;  | &#xB4; | ´                         | ´ | ´ |
| micro sign                                                            | &micro;  | &#181;  | &#xB5; | µ                         | µ | µ |
| pilcrow sign = paragraph sign                                         | &para;   | &#182;  | &#xB6; | ¶                         | ¶ | ¶ |
| middle dot = Georgian comma = Greek middle dot                        | &middot; | &#183;  | &#xB7; | ·                         | · | · |
| cedilla = spacing cedilla                                             | &cedil;  | &#184;  | &#xB8; | ¸                         | ¸ | ¸ |
| superscript one = superscript digit one                               | &sup1;   | &#185;  | &#xB9; | ¹                         | ¹ | ¹ |
| masculine ordinal indicator                                           | &ordm;   | &#186;  | &#xBA; | º                         | º | º |
| right-pointing double angle quotation mark = right pointing guillemet | &raquo;  | &#187;  | &#xBB; | »                         | » | » |
| vulgar fraction one quarter = fraction one quarter                    | &frac14; | &#188;  | &#xBC; | ¼                         | ¼ | ¼ |
| vulgar fraction one half = fraction one half                          | &frac12; | &#189;  | &#xBD; | ½                         | ½ | ½ |
| vulgar fraction three quarters = fraction three quarters              | &frac34; | &#190;  | &#xBE; | ¾                         | ¾ | ¾ |
| inverted question mark = turned question mark                         | &iquest; | &#191;  | &#xBF; | ¿                         | ¿ | ¿ |
| Latin capital letter A with grave = Latin capital letter A grave      | &Agrave; | &#192;  | &#xC0; | À                         | À | À |
| Latin capital letter A with acute                                     | &Aacute; | &#193;  | &#xC1; | Á                         | Á | Á |
| Latin capital letter A with circumflex                                | &Acirc;  | &#194;  | &#xC2; | Â                         | Â | Â |
| Latin capital letter A with tilde                                     | &Atilde; | &#195;  | &#xC3; | Ã                         | Ã | Ã |
| Latin capital letter A with diaeresis                                 | &Auml;   | &#196;  | &#xC4; | Ä                         | Ä | Ä |
| Latin capital letter A with ring above = Latin capital letter A ring  | &Aring;  | &#197;  | &#xC5; | Å                         | Å | Å |
| Latin capital letter AE = Latin capital ligature AE                   | &AElig;  | &#198;  | &#xC6; | Æ                         | Æ | Æ |
| Latin capital letter C with cedilla                                   | &Ccedil; | &#199;  | &#xC7; | Ç                         | Ç | Ç |
| Latin capital letter E with grave                                     | &Egrave; | &#200;  | &#xC8; | È                         | È | È |
| Latin capital letter E with acute                                     | &Eacute; | &#201;  | &#xC9; | É                         | É | É |
| Latin capital letter E with circumflex                                | &Ecirc;  | &#202;  | &#xCA; | Ê                         | Ê | Ê |
| Latin capital letter E with diaeresis                                 | &Euml;   | &#203;  | &#xCB; | Ë                         | Ë | Ë |
| Latin capital letter I with grave                                     | &Igrave; | &#204;  | &#xCC; | Ì                         | Ì | Ì |
| Latin capital letter I with acute                                     | &Iacute; | &#205;  | &#xCD; | Í                         | Í | Í |
| Latin capital letter I with circumflex                                | &Icirc;  | &#206;  | &#xCE; | Î                         | Î | Î |
| Latin capital letter I with diaeresis                                 | &Iuml;   | &#207;  | &#xCF; | Ï                         | Ï | Ï |
| Latin capital letter ETH                                              | &ETH;    | &#208;  | &#xD0; | Ð                         | Ð | Ð |
| Latin capital letter N with tilde                                     | &Ntilde; | &#209;  | &#xD1; | Ñ                         | Ñ | Ñ |
| Latin capital letter O with grave                                     | &Ograve; | &#210;  | &#xD2; | Ò                         | Ò | Ò |
| Latin capital letter O with acute                                     | &Oacute; | &#211;  | &#xD3; | Ó                         | Ó | Ó |
| Latin capital letter O with circumflex                                | &Ocirc;  | &#212;  | &#xD4; | Ô                         | Ô | Ô |
| Latin capital letter O with tilde                                     | &Otilde; | &#213;  | &#xD5; | Õ                         | Õ | Õ |
| Latin capital letter O with diaeresis                                 | &Ouml;   | &#214;  | &#xD6; | Ö                         | Ö | Ö |
| multiplication sign                                                   | &times;  | &#215;  | &#xD7; | ×                         | × | × |
| Latin capital letter O with stroke = Latin capital letter O slash     | &Oslash; | &#216;  | &#xD8; | Ø                         | Ø | Ø |
| Latin capital letter U with grave                                     | &Ugrave; | &#217;  | &#xD9; | Ù                         | Ù | Ù |
| Latin capital letter U with acute                                     | &Uacute; | &#218;  | &#xDA; | Ú                         | Ú | Ú |
| Latin capital letter U with circumflex                                | &Ucirc;  | &#219;  | &#xDB; | Û                         | Û | Û |
| Latin capital letter U with diaeresis                                 | &Uuml;   | &#220;  | &#xDC; | Ü                         | Ü | Ü |
| Latin capital letter Y with acute                                     | &Yacute; | &#221;  | &#xDD; | Ý                         | Ý | Ý |
| Latin capital letter THORN                                            | &THORN;  | &#222;  | &#xDE; | Þ                         | Þ | Þ |
| Latin small letter sharp s = ess-zed                                  | &szlig;  | &#223;  | &#xDF; | ß                         | ß | ß |
| Latin small letter a with grave = Latin small letter a grave          | &agrave; | &#224;  | &#xE0; | à                         | à | à |
| Latin small letter a with acute                                       | &aacute; | &#225;  | &#xE1; | á                         | á | á |
| Latin small letter a with circumflex                                  | &acirc;  | &#226;  | &#xE2; | â                         | â | â |
| Latin small letter a with tilde                                       | &atilde; | &#227;  | &#xE3; | ã                         | ã | ã |
| Latin small letter a with diaeresis                                   | &auml;   | &#228;  | &#xE4; | ä                         | ä | ä |
| Latin small letter a with ring above = Latin small letter a ring      | &aring;  | &#229;  | &#xE5; | å                         | å | å |
| Latin small letter ae = Latin small ligature ae                       | &aelig;  | &#230;  | &#xE6; | æ                         | æ | æ |
| Latin small letter c with cedilla                                     | &ccedil; | &#231;  | &#xE7; | ç                         | ç | ç |
| Latin small letter e with grave                                       | &egrave; | &#232;  | &#xE8; | è                         | è | è |
| Latin small letter e with acute                                       | &eacute; | &#233;  | &#xE9; | é                         | é | é |
| Latin small letter e with circumflex                                  | &ecirc;  | &#234;  | &#xEA; | ê                         | ê | ê |
| Latin small letter e with diaeresis                                   | &euml;   | &#235;  | &#xEB; | ë                         | ë | ë |
| Latin small letter i with grave                                       | &igrave; | &#236;  | &#xEC; | ì                         | ì | ì |
| Latin small letter i with acute                                       | &iacute; | &#237;  | &#xED; | í                         | í | í |
| Latin small letter i with circumflex                                  | &icirc;  | &#238;  | &#xEE; | î                         | î | î |
| Latin small letter i with diaeresis                                   | &iuml;   | &#239;  | &#xEF; | ï                         | ï | ï |
| Latin small letter eth                                                | &eth;    | &#240;  | &#xF0; | ð                         | ð | ð |
| Latin small letter n with tilde                                       | &ntilde; | &#241;  | &#xF1; | ñ                         | ñ | ñ |
| Latin small letter o with grave                                       | &ograve; | &#242;  | &#xF2; | ò                         | ò | ò |
| Latin small letter o with acute                                       | &oacute; | &#243;  | &#xF3; | ó                         | ó | ó |
| Latin small letter o with circumflex                                  | &ocirc;  | &#244;  | &#xF4; | ô                         | ô | ô |
| Latin small letter o with tilde                                       | &otilde; | &#245;  | &#xF5; | õ                         | õ | õ |
| Latin small letter o with diaeresis                                   | &ouml;   | &#246;  | &#xF6; | ö                         | ö | ö |
| division sign                                                         | &divide; | &#247;  | &#xF7; | ÷                         | ÷ | ÷ |
| Latin small letter o with stroke = Latin small letter o slash         | &oslash; | &#248;  | &#xF8; | ø                         | ø | ø |
| Latin small letter u with grave                                       | &ugrave; | &#249;  | &#xF9; | ù                         | ù | ù |
| Latin small letter u with acute                                       | &uacute; | &#250;  | &#xFA; | ú                         | ú | ú |
| Latin small letter u with circumflex                                  | &ucirc;  | &#251;  | &#xFB; | û                         | û | û |
| Latin small letter u with diaeresis                                   | &uuml;   | &#252;  | &#xFC; | ü                         | ü | ü |
| Latin small letter y with acute                                       | &yacute; | &#253;  | &#xFD; | ý                         | ý | ý |
| Latin small letter thorn                                              | &thorn;  | &#254;  | &#xFE; | þ                         | þ | þ |
| Latin small letter y with diaeresis                                   | &yuml;   | &#255;  | &#xFF; | ÿ                         | ÿ | ÿ |
+-----------------------------------------------------------------------+----------+---------+--------+---------------------------+---+---+


+--------------------------------------------+----------+---------+----------+---------------------------+---+---+
|                 Character                  |  Entity  | Decimal |   Hex    | Rendering in Your Browser |   |   |
+--------------------------------------------+----------+---------+----------+---------------------------+---+---+
| Entity                                     | Decimal  | Hex     |          |                           |   |   |
| quotation mark = APL quote                 | &quot;   | &#34;   | &#x22;   | "                         | " | " |
| ampersand                                  | &amp;    | &#38;   | &#x26;   | &                         | & | & |
| less-than sign                             | &lt;     | &#60;   | &#x3C;   | <                         | < | < |
| greater-than sign                          | &gt;     | &#62;   | &#x3E;   | >                         | > | > |
| Latin capital ligature OE                  | &OElig;  | &#338;  | &#x152;  | Œ                         | Œ | Œ |
| Latin small ligature oe                    | &oelig;  | &#339;  | &#x153;  | œ                         | œ | œ |
| Latin capital letter S with caron          | &Scaron; | &#352;  | &#x160;  | Š                         | Š | Š |
| Latin small letter s with caron            | &scaron; | &#353;  | &#x161;  | š                         | š | š |
| Latin capital letter Y with diaeresis      | &Yuml;   | &#376;  | &#x178;  | Ÿ                         | Ÿ | Ÿ |
| modifier letter circumflex accent          | &circ;   | &#710;  | &#x2C6;  | ˆ                         | ˆ | ˆ |
| small tilde                                | &tilde;  | &#732;  | &#x2DC;  | ˜                         | ˜ | ˜ |
| en space                                   | &ensp;   | &#8194; | &#x2002; |                           |   |   |
| em space                                   | &emsp;   | &#8195; | &#x2003; |                           |   |   |
| thin space                                 | &thinsp; | &#8201; | &#x2009; |                           |   |   |
| zero width non-joiner                      | &zwnj;   | &#8204; | &#x200C; | ‌                         | ‌ | ‌ |
| zero width joiner                          | &zwj;    | &#8205; | &#x200D; | ‍                         | ‍ | ‍ |
| left-to-right mark                         | &lrm;    | &#8206; | &#x200E; | ‎                         | ‎ | ‎ |
| right-to-left mark                         | &rlm;    | &#8207; | &#x200F; | ‏                         | ‏ | ‏ |
| en dash                                    | &ndash;  | &#8211; | &#x2013; | –                         | – | – |
| em dash                                    | &mdash;  | &#8212; | &#x2014; | —                         | — | — |
| left single quotation mark                 | &lsquo;  | &#8216; | &#x2018; | ‘                         | ‘ | ‘ |
| right single quotation mark                | &rsquo;  | &#8217; | &#x2019; | ’                         | ’ | ’ |
| single low-9 quotation mark                | &sbquo;  | &#8218; | &#x201A; | ‚                         | ‚ | ‚ |
| left double quotation mark                 | &ldquo;  | &#8220; | &#x201C; | “                         | “ | “ |
| right double quotation mark                | &rdquo;  | &#8221; | &#x201D; | ”                         | ” | ” |
| double low-9 quotation mark                | &bdquo;  | &#8222; | &#x201E; | „                         | „ | „ |
| dagger                                     | &dagger; | &#8224; | &#x2020; | †                         | † | † |
| double dagger                              | &Dagger; | &#8225; | &#x2021; | ‡                         | ‡ | ‡ |
| per mille sign                             | &permil; | &#8240; | &#x2030; | ‰                         | ‰ | ‰ |
| single left-pointing angle quotation mark  | &lsaquo; | &#8249; | &#x2039; | ‹                         | ‹ | ‹ |
| single right-pointing angle quotation mark | &rsaquo; | &#8250; | &#x203A; | ›                         | › | › |
| euro sign                                  | &euro;   | &#8364; | &#x20AC; | €                         | € | € |
+--------------------------------------------+----------+---------+----------+---------------------------+---+---+

What I expected

The text would show as above on the live site and in the editor.

What happened instead

In the editor: HTML, decimal, and hex entities are rendered as their respective characters
On the live site: decimal and hex entities are rendered to their respective characters

To complicate matters further, &, <, and > seem to behave in even more non-standard ways:

&:

Editor: changes everything to &
Live: renders decimal and hex as actual character, and changes actual character as &

<:

Editor: changes everything to <
Live: renders everything as actual character

>:

Editor: renders everything as actual character
Live: renders decimal and hex as actual character, and changes actual character as >

Browser / OS version

Chrome 75 on macOS 10.14.5

Context / Source

Reported in #13419156-hc. Was able to replicate on my end. This appears to be a regression per the user report, and I'd say this is a pretty serious bug that directly leads post data corruption/loss.

The text was updated successfully, but these errors were encountered:

adnanafzal565 · 2020-07-02T04:41:10Z

Same here.

alexsanford added [Type] Bug [Status] Queued In the queue of issues to work on next. [Pri] High Wide impact, no workaround. labels Jun 12, 2020

donnapep mentioned this issue Jul 28, 2020

Fix escaping issues with HTML entities #160

Merged

yscik closed this as completed in #160 Sep 7, 2020

This was referenced Nov 13, 2020

Release 3.5.6 #170

Closed

Fix escaping for shortcodes and URLs #177

Merged

jom mentioned this issue Nov 17, 2020

Fix content attribute selector #178

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

SyntaxHighlighter Code block: improper encoding/decoding of HTML entities #152

SyntaxHighlighter Code block: improper encoding/decoding of HTML entities #152

alexsanford commented Jun 12, 2020

adnanafzal565 commented Jul 2, 2020

SyntaxHighlighter Code block: improper encoding/decoding of HTML entities #152

SyntaxHighlighter Code block: improper encoding/decoding of HTML entities #152

Comments

alexsanford commented Jun 12, 2020

Steps to reproduce

What I expected

What happened instead

Browser / OS version

Context / Source

adnanafzal565 commented Jul 2, 2020