utils: <math> formulas rendered to SVGs without using LaTeX tools #1432

BoboTiG · 2022-11-16T11:38:01Z

Fixes #1427.
Fixes #1198.
Closes #1209.

Tests to pass before merging (the rendering is good, but not the display):

$ python -m wikidict fr --gen-dict "cercle unité" --output issue-1427
$ python -m wikidict en --gen-dict "Wallis product,primitive recursion,Horner's rule" --output issue-1427

sourcery-ai · 2022-11-16T11:38:16Z

Sourcery Code Quality Report

❌ Merging this PR will decrease code quality in the affected files by 1.86%.

Quality metrics	Before	After	Change
Complexity	2.80 ⭐	3.54 ⭐	0.74 👎
Method Length	104.38 🙂	99.12 🙂	-5.26 👍
Working memory	7.60 🙂	8.36 🙂	0.76 👎
Quality	68.24% 🙂	66.38% 🙂	-1.86% 👎

Other metrics	Before	After	Change
Lines	640	651	11

Changed files	Quality Before	Quality After	Quality Change
wikidict/constants.py	86.39% ⭐	85.77% ⭐	-0.62% 👎
wikidict/utils.py	67.18% 🙂	64.93% 🙂	-2.25% 👎

Here are some functions in these files that still need a tune-up:

File	Function	Complexity	Length	Working Memory	Quality	Recommendation
wikidict/utils.py	clean	4 ⭐	304 ⛔	10 😞	49.10% 😞	Try splitting into smaller methods. Extract out complex expressions
wikidict/utils.py	transform	8 ⭐	132 😞	10 😞	57.08% 🙂	Try splitting into smaller methods. Extract out complex expressions
wikidict/utils.py	process_templates	9 🙂	148 😞	8 🙂	58.61% 🙂	Try splitting into smaller methods
wikidict/utils.py	table2html	6 ⭐	82 🙂	11 😞	63.73% 🙂	Extract out complex expressions
wikidict/utils.py	get_word_of_the_day	1 ⭐	142 😞	9 🙂	64.85% 🙂	Try splitting into smaller methods

Legend and Explanation

The emojis denote the absolute quality of the code:

⭐ excellent
🙂 good
😞 poor
⛔ very poor

The 👍 and 👎 indicate whether the quality has improved or gotten worse with this pull request.

Please see our documentation here for details on how these metrics are calculated.

We are actively working on this report - lots more documentation and extra metrics to come!

Help us improve this quality report!

BoboTiG · 2022-11-16T11:39:38Z

Question 1: Should we use a LRU cache on render_formula()?

Question 2: Should we store optimized SVGs for later runs? So we could run tests without calling the REST API.

lasconic · 2022-11-16T12:18:22Z

lru_cache: My understanding is that it will cache the results of a function when the function is called several times with the same args during a program run. It will not persists between run. I believe we don't call this function often with the same argument. Right ? So I would say lru_cache will not really speed up the process, but hey, we could try it.

store svgs: if it doesn't make the test moot, sure, we would store them.

lasconic · 2022-11-16T12:24:40Z

wikidict/utils.py

+
+    svg_optimized = scourString(svg_raw, options=SCOUR_OPTS)
+    return (
+        f'<img src="data:image/svg+xml;utf8,{quote(svg_optimized)}"/>'


You tried to embed the SVG directly ? without img tag ?

Not yet, good idea 👍

The size is better but the display is messy :-/

On the other hand, it displays perfectly here:

Maybe "cercle unité" contains raw text that scales badly. Will check the SVG content.

Checked english words, the display is just perfect using <svg>. Let's hope we can fix the display issue for "cercle unité" (and potentially other words, but we made a good progress still).

Hm interesting ... Saving the SVG to a file: it is displayed properly via the OS viewer.

OK, it's scoup that is doing something with IDs, and brwosers don't like.

BoboTiG · 2022-11-16T13:39:19Z

lru_cache: My understanding is that it will cache the results of a function when the function is called several times with the same args during a program run. It will not persists between run. I believe we don't call this function often with the same argument. Right ? So I would say lru_cache will not really speed up the process, but hey, we could try it.

I was thinking about duplicate formulas, but it will be near to zero at the end, mostly because the cache won't be shared accross locales. Let's forget that idea.

store svgs: if it doesn't make the test moot, sure, we would store them.

It would make sense to store them: 2 HTTP calls saved for each formula. I'll implement it right now.

Root cause was shortening IDs via Scour.

BoboTiG · 2022-11-16T15:39:28Z

@lasconic ready for final review.

Only one tiny concern about the potential size of svg_cache.py, but I guess there are not thousands of formulas to keep in the cache.

lasconic · 2022-11-16T15:51:13Z

PNG master

SVG this PR.

It's crispy but it's too large no ? the last one is cut off.

BoboTiG · 2022-11-16T15:56:41Z

Let's see if other formulas are cut off too. I would say it's quite correct now (far better than what we used to have).

BoboTiG · 2022-11-16T16:05:45Z

Do you want to dig into scaling down the SVGs, maybe?

BoboTiG · 2022-11-16T16:08:13Z

I updated the cache with all formulas used in french. I'll update with all locales shortly.

BoboTiG · 2022-11-16T17:19:32Z

cb61ade fixed the issue where we altered <math> formulas, especially on the SV dict.

BoboTiG · 2022-11-16T17:50:07Z

I'm finished with the cache.

Maybe before merging we could find a way to compress svg_cache.py? An idea:

using md5(formula) for the keys
compressing SVG data as .tar.bz2 or something more powerfull from the standard modules. FTR gzip is badly perfoming.

Maybe not a good idea. The file is only 2 MiN, we can live with that.

BoboTiG · 2022-11-16T20:06:55Z

Ah, about the cut off, I didn't a big part of the picture was hidden. That's a shame.

lasconic · 2022-11-16T20:11:25Z

instead of md5, maybe SHA1 ? It seems wikimedia does this : https://github.com/wikimedia/restbase/blob/ecef17bda6f4efc0d6e187fb05b1eeb389bf7120/sys/post_data.js#L13

SVGZ is gzip no ? It doesn't perform well in term of speed ?

For the cut off, it's a pity but the real estate on kobo is so small... and we can't change the margins... We could change the font size but it would need style for every word.

BoboTiG · 2022-11-16T20:35:45Z

instead of md5, maybe SHA1 ? It seems wikimedia does this : https://github.com/wikimedia/restbase/blob/ecef17bda6f4efc0d6e187fb05b1eeb389bf7120/sys/post_data.js#L13

Weird, I can't replicate the hash.

formula = "V^n"
print(sha1(formula.encode()).hexdigest())

It gives 1617fcdcebf4aa277a74b0c3bb4772ea65d8966f, but eb483cb4ed353a8b15938aa2d994cc5f0fb9055a is returned by the service.
EDIT: Oh, it must be because of the normalization.

SVGZ is gzip no ? It doesn't perform well in term of speed ?

I didn't try yet, but the issue is not about perfs but storage on our side. Maybe not even a issue, I just raised the concern.

For the cut off, it's a pity but the real estate on kobo is so small... and we can't change the margins... We could change the font size but it would need style for every word.

What about we live with it? :)

lasconic · 2022-11-16T20:49:15Z

They do the hash on a normalized json string it includes the type and the formula https://github.com/wikimedia/restbase/blob/ecef17bda6f4efc0d6e187fb05b1eeb389bf7120/sys/mathoid.js#L33

BoboTiG · 2022-11-16T21:11:24Z

No luck so far. I would just let the cache as-is until it becomes problematic.

BoboTiG · 2022-11-16T21:21:40Z

I finally could replicate the hash.

We need to use the normalized TeX, it's available in the 1^st call (formula is V^n for the test):

{
    "success": true,
    "checked": "V^{n}",
    "requiredPackages": [],
    "identifiers": ["V", "n"],
    "endsWithDot": false,
}

(See the checked key.)

Then, we need to use the full query, as you found out, but also without spaces between items:

d = json.dumps({"q": "V^{n}", "type": "tex"}, indent=None)
d = d.replace(" ", "")
print(sha1(d.encode()).hexdigest())

I know, the POC is ugly, and will not work for all cases :)

BoboTiG · 2022-12-13T20:19:04Z

Well, we "just" need to fix the display issue, then we will be good to merge.
Do you have an idea on how to do that? Maybe using some CSS (sadly)?

lasconic · 2022-12-13T20:22:24Z

Which display issue ? If it's the cut out formula, then I don't think there is a fix... The screen of the kobo is just too small for some formula. If we make it smaller, it will become not readable.

BoboTiG · 2022-12-13T20:24:51Z

I was hoping to have such CSS hack: svg {max-width:100%} :D

I will still try one thing or two. In the eventuallity it's not possible to fix, do you propose to close the PR, and forget SVG stuff?

BoboTiG · 2022-12-13T20:26:03Z

Formulas are not so present in dictionaries. If some are cut, I'm OK with that.

utils: <math> formulas rendered to SVGs without using LaTeX tools

77ce352

BoboTiG requested a review from lasconic November 16, 2022 11:38

lasconic reviewed Nov 16, 2022

View reviewed changes

Use standalone <svg>

392a622

BoboTiG added 5 commits November 16, 2022 15:50

Use a cache for optimized SVG files

7f04278

Move more code to svg.py

39ff77c

Clean-up

1b36151

doc: fix typo

b2249f7

fix display issue on browser

ec8600b

Root cause was shortening IDs via Scour.

BoboTiG marked this pull request as ready for review November 16, 2022 15:37

Clean-up

f50aaee

Move the cache to a specific file

a7a2819

BoboTiG added 9 commits November 16, 2022 17:16

cache: add formulas from Norway dict

99a2487

cache: add formulas from CA dict

8c46740

cache: add formulas from IT dict

500e72e

cache: add formulas from SV dict

6ee4c48

cache: add formulas from EL dict

5befef3

cache: add formulas from ES dict

75867e8

cache: add formulas from DE dict

22823d9

cache: add formulas from EN dict + sort

da2a5af

cache: missing piece

901383d

BoboTiG added 3 commits November 16, 2022 17:51

cache: add formulas from SV dict

fdf6c34

cache: add missed formulas from EN dict

d40772d

Prevent altering <math> formulas

cb61ade

BoboTiG added 2 commits November 16, 2022 18:38

cache: missing SV formulas

bf60d3d

cache: last SV formulas

71571b4

Better defaults, and docstring

7cdb977

Merge branch 'master' into fix-1427

14cf341

BoboTiG merged commit 7a6ffbf into master Dec 14, 2022

BoboTiG deleted the fix-1427 branch December 14, 2022 08:10

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

utils: <math> formulas rendered to SVGs without using LaTeX tools #1432

utils: <math> formulas rendered to SVGs without using LaTeX tools #1432

BoboTiG commented Nov 16, 2022 •

edited

Loading

sourcery-ai bot commented Nov 16, 2022 •

edited

Loading

BoboTiG commented Nov 16, 2022

lasconic commented Nov 16, 2022

lasconic Nov 16, 2022

BoboTiG Nov 16, 2022

lasconic Nov 16, 2022

BoboTiG Nov 16, 2022

BoboTiG Nov 16, 2022

BoboTiG Nov 16, 2022 •

edited

Loading

This comment was marked as outdated.

BoboTiG Nov 16, 2022

BoboTiG Nov 16, 2022

BoboTiG Nov 16, 2022

BoboTiG commented Nov 16, 2022

BoboTiG commented Nov 16, 2022 •

edited

Loading

lasconic commented Nov 16, 2022

BoboTiG commented Nov 16, 2022

BoboTiG commented Nov 16, 2022

BoboTiG commented Nov 16, 2022 •

edited

Loading

BoboTiG commented Nov 16, 2022

BoboTiG commented Nov 16, 2022 •

edited

Loading

BoboTiG commented Nov 16, 2022

lasconic commented Nov 16, 2022 •

edited

Loading

BoboTiG commented Nov 16, 2022 •

edited

Loading

lasconic commented Nov 16, 2022

BoboTiG commented Nov 16, 2022

BoboTiG commented Nov 16, 2022 •

edited

Loading

BoboTiG commented Dec 13, 2022

lasconic commented Dec 13, 2022

BoboTiG commented Dec 13, 2022 •

edited

Loading

BoboTiG commented Dec 13, 2022 •

edited

Loading

utils: <math> formulas rendered to SVGs without using LaTeX tools #1432

utils: <math> formulas rendered to SVGs without using LaTeX tools #1432

Conversation

BoboTiG commented Nov 16, 2022 • edited Loading

sourcery-ai bot commented Nov 16, 2022 • edited Loading

Sourcery Code Quality Report

Legend and Explanation

BoboTiG commented Nov 16, 2022

lasconic commented Nov 16, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

BoboTiG Nov 16, 2022 • edited Loading

Choose a reason for hiding this comment

This comment was marked as outdated.

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

BoboTiG commented Nov 16, 2022

BoboTiG commented Nov 16, 2022 • edited Loading

lasconic commented Nov 16, 2022

BoboTiG commented Nov 16, 2022

BoboTiG commented Nov 16, 2022

BoboTiG commented Nov 16, 2022 • edited Loading

BoboTiG commented Nov 16, 2022

BoboTiG commented Nov 16, 2022 • edited Loading

BoboTiG commented Nov 16, 2022

lasconic commented Nov 16, 2022 • edited Loading

BoboTiG commented Nov 16, 2022 • edited Loading

lasconic commented Nov 16, 2022

BoboTiG commented Nov 16, 2022

BoboTiG commented Nov 16, 2022 • edited Loading

BoboTiG commented Dec 13, 2022

lasconic commented Dec 13, 2022

BoboTiG commented Dec 13, 2022 • edited Loading

BoboTiG commented Dec 13, 2022 • edited Loading

BoboTiG commented Nov 16, 2022 •

edited

Loading

sourcery-ai bot commented Nov 16, 2022 •

edited

Loading

BoboTiG Nov 16, 2022 •

edited

Loading

BoboTiG commented Nov 16, 2022 •

edited

Loading

BoboTiG commented Nov 16, 2022 •

edited

Loading

BoboTiG commented Nov 16, 2022 •

edited

Loading

lasconic commented Nov 16, 2022 •

edited

Loading

BoboTiG commented Nov 16, 2022 •

edited

Loading

BoboTiG commented Nov 16, 2022 •

edited

Loading

BoboTiG commented Dec 13, 2022 •

edited

Loading

BoboTiG commented Dec 13, 2022 •

edited

Loading