Improvements to the tex rendering pipeline #362

JonasIsensee · 2020-06-03T12:18:50Z

This small PR relaxes some type restrictions and implements a few small helper functions in the tex rendering pipeline.

There are no changes to JMarkdown2tex
TexMinted and Tex now have a template field and by default use the md2pdf.tpl template
The template was modified to allow for additional \usepackage{...} dependencies (defined in docformat.tex_deps)
TexMinted now produces runnable tex files but because Highlights.jl only works with lstlisting the standard code highlighting is deactivated.
Tex does not produce a runnable tex file because there is currently no code to define the custom code environments.

aviatesk · 2020-06-03T12:32:37Z

nice, sorry I'll be busy until weekend but I'll make sure to review on this then.
If you have any higher view question, I will try to respond quickly though.

aviatesk

Left few feedbacks, but in general this is really nice enhancements !

aviatesk · 2020-06-07T15:11:40Z

src/rendering/texformats.jl

@@ -185,7 +198,7 @@ function format_chunk(chunk::DocChunk, docformat::JMarkdown2tex)
    return docformat.keep_unicode ? out : uc2tex(out)
 end

-function format_output(result, docformat::JMarkdown2tex)
+function format_output(result, docformat::TexFormat)


I think this will produce "invalid" output texminted format, right ?
Hightlights.jl will include its own characters around some special characters e.g. _, so that it will result in weird output for minted output.
How about defining new format_output for TexMinted ?

Well spotted. I hadn't figured this out.

fixed it.
However it did make my code have more duplicate lines again.

aviatesk · 2020-06-07T15:14:41Z

src/rendering/texformats.jl

+highlight_str(docformat::TexFormat) = ""
+highlight_str(docformat::JMarkdown2tex) =
+    get_highlight_stylesheet(MIME("text/latex"), docformat.highlight_theme)


Maybe defining render_doc for JMarkdown2tex and TexFormat separately will make more sense ?

That is probably true. This saved a handful of lines but of course it needs another new functions.

defined render_doc separately now

aviatesk · 2020-06-07T15:15:57Z

We can update our test when you think this PR is ready.

JonasIsensee · 2020-06-13T07:19:59Z

A more general thought:

Fixing these things I always have to consider what to do with the Tex format that uses juliacode as
an environment. In itself it is not functional and I am not sure if there is anyone using it.

I propose to remove it from the package.
That will make the source code shorter and more maintainable.
(After all - you can't really test something that is non-functional)
And once the API has been refactored we could bring it back as an
example in the docs on how to define your own custom rendering type.

aviatesk · 2020-06-13T10:16:35Z

I propose to remove it from the package.
That will make the source code shorter and more maintainable.
(After all - you can't really test something that is non-functional)
And once the API has been refactored we could bring it back as an
example in the docs on how to define your own custom rendering type.

yeah, this sounds totally fine. Or we can just make Tex works like JMarkdown2tex except it doesn't generate the final PDF document but just stays as a .tex file

JonasIsensee · 2020-06-13T12:46:54Z

yeah, this sounds totally fine. Or we can just make Tex works like JMarkdown2tex except it doesn't generate the final PDF document but just stays as a .tex file

This is actually precisely what md2tex does.

Weave.jl/src/rendering/texformats.jl

Lines 154 to 155 in 5c85315

    
           register_format!("md2tex", JMarkdown2tex()) 
        
           register_format!("md2pdf", JMarkdown2tex())

aviatesk · 2020-06-13T13:23:21Z

ah, yeah, I just thought about the "dispatch-base" external doc generation, but for now
let's just remove Tex format

JonasIsensee · 2020-06-13T13:54:58Z

Ah, that's probably true.
Generally this frees up the name tex. There is no real need for the standard tex pipeline JMarkdown2tex to have such a long name.

I imagine one could implement a

struct PDF <: WeaveFormat
    intermediate_format::TexFormat
    shell_cmd::String
end

that dispatches such that first the inner docformat is rendered, and then
runs xelatex or whatever.

aviatesk · 2020-06-13T14:13:50Z

Okay, cool, so do you feel this is ready for review/merge again ? Then I will look into details agains and update tests, etc.

JonasIsensee · 2020-06-13T14:28:02Z

Renaming JMarkdown2tex and introducing a PDF struct may be the topic of another PR.

This one didn't change too much so I think it is ready for review/merge.

aviatesk · 2020-06-13T14:51:41Z

src/rendering/texformats.jl

+function uc2tex(::TexFormat, s, escape = false)
+    for key in keys(latex_symbols)
+        if escape
+            s = replace(s, latex_symbols[key] => "|\$\\ensuremath{$(texify(key))}\$|")
+        else
+            s = replace(s, latex_symbols[key] => "\\ensuremath{$(texify(key))}")
+        end
+    end
+    return s
+end


something like:

uc2tex(::JMarkdown2tex, s, escape = false) = unicode2tex(s, escape, "*@") uc2tex(::TexFormat, s, escape = false) = unicode2tex(s, escape, "|\$") function unicode2tex(s, escape, starter, closer = reverse(starter)) for key in keys(latex_symbols) body = "\\ensuremath{$(texify(key))}" target = escape ? string(starter, body, closer) : body s = replace(s, latex_symbols[key] => body) end return s end

would look more cleaner ?

ah, that's much nicer !
Please put that instead 👍

or actually I don't think we can just define unicode2tex and don't use dispatch here; it only adds complexity here imho.

I'm sorry, I don't quite get what you are trying to say.

ups, sorry, very bad english, I tied to say:
I think we can just define unicode2tex and remove uc2tex. Using dispatches doesn't seem to be necessary here imho.

ah, I'm starting to understand.
The dispatch is indeed not necessary, but on the other hand, uc2tex is currently called 5 times and in one of those cases (

Weave.jl/src/rendering/texformats.jl

Line 179 in b569740

return docformat.keep_unicode ? out : uc2tex(docformat, out)

)

you can't hardcode the escape sequence. (Because the calling scope is used for all <:TexFormats.

There is however no real need for two names:

unicode2tex(::JMarkdown2tex, s, escape = false) = unicode2tex(s, escape, "*@") unicode2tex(::TexMinted, s, escape = false) = unicode2tex(s, escape, "|\$")) function unicode2tex(s, escape, starter, closer = reverse(starter)) for key in keys(latex_symbols) body = "\\ensuremath{$(texify(key))}" target = escape ? string(starter, body, closer) : body s = replace(s, latex_symbols[key] => body) end return s end

alternatively escape and the escape sequence starter and closer could be made part of the JMarkdown2tex and TexMinted struct

sounds fair, I will like whichever way you would like to choose.

aviatesk · 2020-06-13T19:23:05Z

fwiw, merging (or rebasing) master will get rid of legacy test suites and enable much faster gh-action CI.

JonasIsensee · 2020-06-13T21:50:48Z

oops, I broke the tests again.

codecov-commenter · 2020-06-14T05:32:04Z

Codecov Report

Merging #362 into master will decrease coverage by 1.53%.
The diff coverage is 60.00%.

@@            Coverage Diff             @@
##           master     #362      +/-   ##
==========================================
- Coverage   70.89%   69.35%   -1.54%     
==========================================
  Files          23       23              
  Lines        1254     1263       +9     
==========================================
- Hits          889      876      -13     
- Misses        365      387      +22

Flag	Coverage Δ
#unittests	`69.35% <60.00%> (-1.54%)`	⬇️

Impacted Files	Coverage Δ
src/rendering/rendering.jl	`88.88% <ø> (ø)`
src/rendering/texformats.jl	`66.31% <57.14%> (-22.19%)`	⬇️
src/Weave.jl	`46.00% <100.00%> (+0.54%)`	⬆️
src/run.jl	`80.08% <100.00%> (ø)`

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update bd447ec...092adb6. Read the comment docs.

aviatesk

LGTM, thanks for your great work, @JonasIsensee !

Reapply improvements to the tex rendering pipeline

630ac46

aviatesk reviewed Jun 7, 2020

View reviewed changes

fix math escapes in code and output

5c85315

Remove "tex" format

1135c5d

Merge branch 'master' into pr-362/JonasIsensee/texpipeline

972869d

aviatesk reviewed Jun 13, 2020

View reviewed changes

rm tests with "tex" format

b569740

JonasIsensee added 2 commits June 13, 2020 22:16

move unicode2tex escape characters to struct

12e148a

fix bug

3dfe817

aviatesk added 3 commits June 14, 2020 13:36

Merge branch 'master' into texpipeline

5ae1fd5

construct UNICODE2LATEX at precompile time

872f7c7

update test

092adb6

aviatesk approved these changes Jun 14, 2020

View reviewed changes

aviatesk merged commit 174585e into JunoLab:master Jun 14, 2020

aviatesk added breaking change for records PDF PDF output tex about TeX/LaTeX documents labels Jun 14, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Improvements to the tex rendering pipeline #362

Improvements to the tex rendering pipeline #362

JonasIsensee commented Jun 3, 2020

aviatesk commented Jun 3, 2020 •

edited

aviatesk left a comment

aviatesk Jun 7, 2020

JonasIsensee Jun 13, 2020

JonasIsensee Jun 13, 2020

aviatesk Jun 7, 2020

JonasIsensee Jun 13, 2020

JonasIsensee Jun 13, 2020

aviatesk commented Jun 7, 2020

JonasIsensee commented Jun 13, 2020

aviatesk commented Jun 13, 2020 •

edited

JonasIsensee commented Jun 13, 2020

aviatesk commented Jun 13, 2020

JonasIsensee commented Jun 13, 2020

aviatesk commented Jun 13, 2020

JonasIsensee commented Jun 13, 2020

aviatesk Jun 13, 2020

JonasIsensee Jun 13, 2020

aviatesk Jun 13, 2020

JonasIsensee Jun 13, 2020

aviatesk Jun 13, 2020

JonasIsensee Jun 13, 2020

JonasIsensee Jun 13, 2020

aviatesk Jun 13, 2020

aviatesk commented Jun 13, 2020

JonasIsensee commented Jun 13, 2020

codecov-commenter commented Jun 14, 2020 •

edited

aviatesk left a comment

Improvements to the tex rendering pipeline #362

Improvements to the tex rendering pipeline #362

Conversation

JonasIsensee commented Jun 3, 2020

aviatesk commented Jun 3, 2020 • edited

aviatesk left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

aviatesk commented Jun 7, 2020

JonasIsensee commented Jun 13, 2020

aviatesk commented Jun 13, 2020 • edited

JonasIsensee commented Jun 13, 2020

aviatesk commented Jun 13, 2020

JonasIsensee commented Jun 13, 2020

aviatesk commented Jun 13, 2020

JonasIsensee commented Jun 13, 2020

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

aviatesk commented Jun 13, 2020

JonasIsensee commented Jun 13, 2020

codecov-commenter commented Jun 14, 2020 • edited

Codecov Report

aviatesk left a comment

Choose a reason for hiding this comment

aviatesk commented Jun 3, 2020 •

edited

aviatesk commented Jun 13, 2020 •

edited

codecov-commenter commented Jun 14, 2020 •

edited