Implement strict mode (replacing unicodeTextInMathMode) #1278

edemaine · 2018-04-30T15:14:29Z

Add new "strict" setting (default value false) that can take a boolean (whether to throw an error or silently ignore), string ("ignore", "warn", or "error"), or a function possibly returning such a value. This enables a variety of ways of handling or ignoring transgressions from "true" LaTeX behavior, making KaTeX easier to use while still providing the ability for strict LaTeX adherance.

Resolve #1226, implementing that spec, for two existing transgressions from regular LaTeX:

src/functions/kern.js had some errors and warnings about use of (units in) math vs. text mode commands.
The former setting unicodeTextInMathMode (not in any released version) needed to be set to true to enable Unicode text symbols in math mode.

Now these are controlled by the strict setting. By default, KaTeX is now very permissive, but if desired, the user can request warnings or errors.

codecov · 2018-04-30T15:26:22Z

Codecov Report

Merging #1278 into master will decrease coverage by 0.11%.
The diff coverage is 78.88%.

@@            Coverage Diff             @@
##           master    #1278      +/-   ##
==========================================
- Coverage   83.99%   83.87%   -0.12%     
==========================================
  Files          62       64       +2     
  Lines        4017     4081      +64     
  Branches      670      689      +19     
==========================================
+ Hits         3374     3423      +49     
- Misses        546      559      +13     
- Partials       97       99       +2

Impacted Files	Coverage Δ
src/ParseNode.js	`100% <ø> (ø)`	⬆️
src/MacroExpander.js	`92.39% <ø> (-0.25%)`	⬇️
test/Warning.js	`100% <100%> (ø)`
src/Lexer.js	`100% <100%> (ø)`	⬆️
src/environments/array.js	`43.63% <100%> (+0.77%)`	⬆️
src/macros.js	`85.09% <100%> (+0.09%)`	⬆️
src/Parser.js	`92.95% <100%> (+0.07%)`	⬆️
src/functions/kern.js	`70.58% <50%> (-10.67%)`	⬇️
src/Settings.js	`71.87% <64%> (-28.13%)`	⬇️
test/setup.js	`75% <75%> (ø)`
... and 3 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 4801ab8...ca212c8. Read the comment docs.

Add new "strict" setting (default value false) that can take a boolean (whether to throw an error or silently ignore), string ("ignore", "warn", or "error"), or a function possibly returning such a value. This enables a variety of ways of handling or ignoring transgressions from "true" LaTeX behavior, making KaTeX easier to use while still providing the ability for strict LaTeX adherance. Resolve KaTeX#1226, implementing that spec, for two existing transgressions from regular LaTeX: * src/functions/kern.js had some errors and warnings about use of (units in) math vs. text mode commands. * The former setting unicodeTextInMathMode (not in any released version) needed to be set to true to enable Unicode text symbols in math mode. Now these are controlled by the strict setting. By default, KaTeX is now very permissive, but if desired, the user can request warnings or errors.

ylemkimon · 2018-05-02T13:55:15Z

README.md

 - `maxSize`: `number`. If non-zero, all user-specified sizes, e.g. in `\rule{500em}{500em}`, will be capped to `maxSize` ems. Otherwise, users can make elements and spaces arbitrarily large (the default behavior).
+- `strict`: `boolean` or `string` or `function` (default: `false`). If `false` or `"ignore`", ignore features that make writing LaTeX convenient but are not actually supported by (Xe)LaTeX (similar to MathJax). If `true` or `"error"` (LaTeX compatibility mode), throw an error for any such transgressions. If `"warn"`, warn about such behavior via `console.warn`. Provide a custom function `handler(errorCode, errorMsg, token)` to default behavior depending on the type of transgression (summarized by the string code `errorCode` and detailed in `errorMsg`).  A list of such features and their `errorCode`s:


Doesn't compatibility mode usually mean non-strict mode?

Agreed, "compatibility" maybe isn't the right term. Perhaps "adherence" or "faithfulness"?

I've pushed a rewording of this paragraph that in particular uses "faithfulness" instead of "compatibility".

kevinbarabash

Thanks for aggregating various settings into strict. Looking forward to getting this merged in so that we can continue to build out functionality while have a consistent way to provide a strict mode to users who need it.

kevinbarabash · 2018-05-08T00:36:47Z

src/Settings.js

+     * Report nonstrict (non-LaTeX-compatible) input.
+     * Can safely not be called if `this.strict` is false in JavaScript.
+     */
+    nonstrict(errorCode: string, errorMsg: string, token?: Token | ParseNode) {


I like this. Maybe rename token to tokenOrParseNode.

I was matching ParseError's constructor. If I change one, I should probably change both... I could also see something like where or context being a good name.

kevinbarabash · 2018-05-08T00:37:07Z

src/Settings.js

+                "LaTeX-incompatible input and strict mode is set to 'error': " +
+                `${errorMsg} [${errorCode}]`, token);
+        } else if (strict === "warn") {
+            typeof console !== "undefined" && console.warn(


When is console not defined?

I can't think of such an environment, but presumably they might exist (or the global console could be overwritten?). In any case, I'm following existing code from the KaTeX codebase, in particular:

typeof console !== "undefined" && console.warn( "No character metrics for '" + value + "' in style '" + fontName + "'");

It might be good to define a katex.warn or something that can be overwritten by the user, but defaults to console.warn.

Oh wow, I didn't realize we were doing that elsewhere.

kevinbarabash · 2018-05-08T00:39:33Z

static/main.js

+        if (query.strict.match(/^(1|t|y|e)/)) {
+            options.strict = "error";
+        } if (query.strict && query.strict.match(/^(w)/)) {
+            options.strict = "warn";


I like the simplicity of this.

kevinbarabash · 2018-05-08T00:40:42Z

src/Settings.js


 import type { MacroMap } from "./macros";

+export type StrictFunction =
+    (errorCode: string, errorMsg: string, token?: Token | ParseNode) =>
+    ?(boolean | string);


Exposing this level of customizability to users seems like overkill.

This level of customizability lets users allow some nonstrict behavior and not others. @ylemkimon at least seemed to think this would be useful, and it's not exactly hard to support. Thoughts?

Another point for allowing a function is that it enables custom warning functionality, if console.warn isn't the desired warning output. For example, this would make our own testing easier, as jest doesn't really handle warnings well.

I think it's nice to have customizability as long as we have simpler options to set them. We may add lots of features to the non-strict mode, and strict option may become too generic.

kevinbarabash · 2018-05-08T00:42:11Z

src/Settings.js

 export type SettingsOptions = {
    displayMode?: boolean;
    throwOnError?: boolean;
    errorColor?: string;
    macros?: MacroMap;
    colorIsTextColor?: boolean;
-    unicodeTextInMathMode?: boolean;
+    strict?: boolean | string | StrictFunction;


I think it's sufficient if strict?: boolean | "ignore" | "warn" | "error".

I replaced string with "ignore" | "warn" | "error" but left StrictFunction for now.

edemaine · 2018-05-08T14:23:30Z

If possible (in particular, if it doesn't slow things down much), I think it would be good to merge this before the next release, because it makes behavior more permissive than the current release. So far we haven't released with the unicodeTextInMathMode setting, which on master has a default value of false, which is less permissive than the current release.

In other words, if we release now, we might break some code. If we merge this first, we shouldn't, because of more permissive defaults. (Alternatively, we could change the default unicodeTextInMathMode to true.)

Also, this PR removes the unicodeTextInMathMode setting, so it'd be nice not to add a setting and then remove it in the next release, if we can avoid it.

edemaine · 2018-05-08T15:46:01Z

By the way, we might also discuss whether the right default for strict is

false/"ignore", as it is currently in this PR; or
"warn", which feels a little "safer", so at least people learn about the idea of strictness and then can make a decision to leave it or set to true or false?

ylemkimon · 2018-05-09T12:47:48Z

src/Parser.js

+                this.settings.nonstrict("unicodeTextInMathMode",
+                    `Unicode text character "${text[0]}" used in math mode`,
+                    nucleus);
+            }


How about allowing all unicodes(non-ASCII) on non-strict mode? (#1217, allowAllSymbols)

I was wondering about that, but planning to deal with it when we return to discussing #1217. allowAllSymbols feels like more of a hack, not to be encouraged -- the rendering will probably be poor in general, and it's really a sign that we need to add symbols to the fonts. So I was thinking of keeping that a separate option... but I'm open to suggestions. (And maybe it is worth discussing here, in case we want to tweak the strict functionality to deal with it...)

I think that's where the customizability should come in. Normal users would intend used symbols to be shown, but advanced users or who worries about the layout would want to know unsupported symbol has been used.

OK, I agree. I think this is more motivation to set the default strict value to "warn", so at least people get a sense of why the rendering is funny in these cases.

edemaine · 2018-05-10T15:18:10Z

I've gone ahead and changed the default setting for strict to "warn".

I also added functionality to check for warnings in Jest, turning them into a special type of Error. These changes are also a first step toward #1126.

edemaine · 2018-05-11T18:07:31Z

@ylemkimon Thanks for the review!
@kevinbarabash Are you OK with this being merged, with the following decisions?

Default strict setting is "warn"
Allow functions to be specified in strict argument

edemaine · 2018-05-13T02:33:41Z

~~OK, I propose merging this first, then I can add nonstrict behavior to \\.~~ @kevinbarabash , are you OK with leaving function support in?

edemaine · 2018-05-13T18:27:11Z

The changes to make strict mode control \newline behavior are extensive, so I'm going to put them in a separate PR.

edemaine force-pushed the strictmode branch from 732f4b4 to 1b6db8e Compare April 30, 2018 15:31

ylemkimon reviewed May 2, 2018

View reviewed changes

edemaine added 3 commits May 4, 2018 16:48

Rewrite strict description

a1f292a

Merge branch 'master' into strictmode

f8da0cc

Add tests for strict functions

4af450a

kevinbarabash reviewed May 8, 2018

View reviewed changes

edemaine added 2 commits May 8, 2018 11:41

Stricter type for strict

35d9876

Merge branch 'master' into strictmode

f8400f6

Merge branch 'master' into strictmode

d79f653

ylemkimon reviewed May 9, 2018

View reviewed changes

ylemkimon added the discussion label May 9, 2018

Switch default strict setting to "warn"

c1694f0

ronkok mentioned this pull request May 10, 2018

Support \raise and \lower #1167

Closed

edemaine added 2 commits May 11, 2018 09:41

Merge branch 'master' into strictmode

222f6eb

Fix new flow error

8b8d794

edemaine added the GH Review: review-needed label May 11, 2018

ylemkimon approved these changes May 11, 2018

View reviewed changes

ylemkimon removed the GH Review: review-needed label May 11, 2018

Fix another flow bug

905bfd8

ylemkimon and others added 3 commits May 12, 2018 18:28

Merge branch 'master' into strictmode

890288d

Merge branch 'master' into strictmode

b10418f

Merge branch 'master' into strictmode

22895dc

edemaine mentioned this pull request May 13, 2018

0.10.x release planning #1311

Closed

edemaine mentioned this pull request May 13, 2018

Support for top-level \newline and \\ in inline math #1298

Merged

Merge remote-tracking branch 'upstream/master' into strictmode

ca212c8

edemaine added GH Review: review-needed and removed discussion labels May 13, 2018

edemaine force-pushed the strictmode branch from ec88380 to ca212c8 Compare May 13, 2018 18:26

edemaine merged commit 7ab4f76 into KaTeX:master May 13, 2018

ronkok mentioned this pull request Jun 5, 2018

[plugin system] Add a utility function (setFontMetrics) to extend builtin fontMetrics #1269

Merged

ylemkimon mentioned this pull request Aug 18, 2018

Release v0.10.0-rc #1634

Merged

1 task

snyk-bot mentioned this pull request Feb 23, 2020

[Snyk] Upgrade katex from 0.9.0 to 0.11.1 saurabharch/Rocket.Chat#1

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Implement strict mode (replacing unicodeTextInMathMode) #1278

Implement strict mode (replacing unicodeTextInMathMode) #1278

edemaine commented Apr 30, 2018

codecov bot commented Apr 30, 2018 •

edited

ylemkimon May 2, 2018

edemaine May 2, 2018

edemaine May 4, 2018

kevinbarabash left a comment

kevinbarabash May 8, 2018

edemaine May 8, 2018

kevinbarabash May 8, 2018

edemaine May 8, 2018

kevinbarabash May 8, 2018

kevinbarabash May 8, 2018

kevinbarabash May 8, 2018

edemaine May 8, 2018 •

edited

edemaine May 8, 2018

ylemkimon May 9, 2018

kevinbarabash May 8, 2018

edemaine May 8, 2018

edemaine commented May 8, 2018 •

edited

edemaine commented May 8, 2018 •

edited

ylemkimon May 9, 2018

edemaine May 9, 2018

ylemkimon May 9, 2018

edemaine May 9, 2018

edemaine commented May 10, 2018

edemaine commented May 11, 2018

edemaine commented May 13, 2018 •

edited

edemaine commented May 13, 2018

		- `maxSize`: `number`. If non-zero, all user-specified sizes, e.g. in `\rule{500em}{500em}`, will be capped to `maxSize` ems. Otherwise, users can make elements and spaces arbitrarily large (the default behavior).
		- `strict`: `boolean` or `string` or `function` (default: `false`). If `false` or `"ignore`", ignore features that make writing LaTeX convenient but are not actually supported by (Xe)LaTeX (similar to MathJax). If `true` or `"error"` (LaTeX compatibility mode), throw an error for any such transgressions. If `"warn"`, warn about such behavior via `console.warn`. Provide a custom function `handler(errorCode, errorMsg, token)` to default behavior depending on the type of transgression (summarized by the string code `errorCode` and detailed in `errorMsg`). A list of such features and their `errorCode`s:

Implement strict mode (replacing unicodeTextInMathMode) #1278

Implement strict mode (replacing unicodeTextInMathMode) #1278

Conversation

edemaine commented Apr 30, 2018

codecov bot commented Apr 30, 2018 • edited

Codecov Report

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kevinbarabash left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

edemaine May 8, 2018 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

edemaine commented May 8, 2018 • edited

edemaine commented May 8, 2018 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

edemaine commented May 10, 2018

edemaine commented May 11, 2018

edemaine commented May 13, 2018 • edited

edemaine commented May 13, 2018

codecov bot commented Apr 30, 2018 •

edited

edemaine May 8, 2018 •

edited

edemaine commented May 8, 2018 •

edited

edemaine commented May 8, 2018 •

edited

edemaine commented May 13, 2018 •

edited