first version of xml, xhtml and epub export tagging support #34
Conversation
SYNBOL and SYNEOL now insert a <div class="syntaxlinegroup"> tag in the xml exporter output. The most tricky part was to ensure independent of whether linenumbering is on or off that each line is rendereed as distinct line by the browser and epub viewer. Finally solved by makine SYNEOL insert a <div class=break> tag which is used by the exporter anyway to make the browser render distinct lines. A separate css generated for each colorscheme defined. Ensuring that coloring of code is as much as possible matching between xml/xhtml/epub and pdf output. TODO: linenumbering breaks when placed inside float see example tests/vim/40-SYNBOL-SYNEOL-ln-tagging-export-float.tex. Likely some tag or some other additional settings have to be made that also in xml/xhtml output the line number injection works properly and line numbers are added before or after each break tag. Currently only tested with line numbers placed left. Other placements may cause even more errors.
|
This is very exciting! Thanks. I should have time next week, when I'll go over both your pull requests. |
|
This commit only contains the test files. Did you forget to include some files? |
| subtitle={}, | ||
| author={J.R.C.H.} | ||
| ] | ||
| \usemodule[amsl] |
There was a problem hiding this comment.
Why do you add amsl. That module is not needed for this test and is deprecated anyways.
There was a problem hiding this comment.
Copy paste from other source. Initially i thought to also demonstrate but that would be something for later more advanced test files if necesary at all. can go.
There was a problem hiding this comment.
Could you please remove this from all test files.
| ] | ||
| \usemodule[amsl] | ||
| \usemodule[vim] | ||
| \definevimtyping[Python][syntax=python,tab=4,numbering=no] |
There was a problem hiding this comment.
The standard style that I have followed in all the tests is to use capital case for language names. Could you change this to
\definevimtyping[PYTHON][....]
| \usemodule[vim] | ||
| \definevimtyping[Python][syntax=python,tab=4,numbering=no] | ||
| \starttext | ||
| This is a little test whether modifcation in \type{\SYNBOL} and \type{\SYNEOL} have any sideeffects |
| \setupbackend[ | ||
| export=yes, | ||
| xhtml=yes | ||
| ] |
There was a problem hiding this comment.
Could you also follow the formatting style for the rest of the tests. Basically:
\setupbackend
[
export=yes,
xhtml=yes,
]
| subtitle={}, | ||
| author={J.R.C.H.} | ||
| ] | ||
| \usemodule[amsl] |
| a=12 | ||
| def hello_world(hello) | ||
| print("simon says: {}".format(hello)) | ||
| hallo_welt("how do youdo") |
There was a problem hiding this comment.
Minor point: but shouldn't this be hello_world ... also youdo -> 'you do`
|
This commit only contains test files. There is no file which actually implements the tagging. Perhaps you forgot to add it to your branch. Could you please check it. Thanks |
Adds the files missed in last commit and updates the test files. Line numbering still a wild hack and not working within floats and framed environment. Still work in progress
|
Ouups missed to add them should be solved for now, squash in final commit |
|
Which version of ConTeXt ( With MkIV 2020.01.30, the files do not compile. For example: gives I get a similar error with LMTX 2020.04.27. |
|
Hi
[Inline Response to message by Aditya Mahajan, Mi, 2020-04-29 14:11
-0700: Answers and comments below corresponding cited paragraphs.]
Which version of ConTeXt (`context --version`) are you using?
With MkIV 2020.01.30, the files do not compile. For example:
```
context tests/vim/38-SYNBOL-SYNEOL-ln-tagging-export.tex
```
gives
```
tex error > tex error on line 8 in file tests/vim/38-SYNBOL-
SYNEOL-ln-tagging-export.tex: ! Undefined control sequence
<recently read> \dosynchronizeexport
***@***.*** ...ingparameter \c!alternative }
\setvalue {\e!start
raw#1}...
l.8 ...hon,tab=4,numbering=yes,numberlocation=right]
```
I get a similar error with LMTX 2020.04.27.
I currently use 2018 tex-life als 2020 texlife from jonathonf was not
yet ready and 2019 was broken. The standalone i did not try yet as i
currently use it in production and already had a texlife setup and did
not want to switch.
Maybe this would also solve some of my other problems. I will take care
about it in a few weeks when i hope to have finished my lecture notes
and thus safely can switch to an up to date and clean setup including
the recent release of t-vim.
Xristoph
|
|
I currently use 2018 tex-life als 2020 texlife from jonathonf was not
yet ready and 2019 was broken. The standalone i did not try yet as i
currently use it in production and already had a texlife setup and did
not want to switch.
Note that it is possible to install context-standalone in parallel to texlive.
In fact, you can have multiple versions of context-stanalone in parallel, so
you can keep a frozen version for long running projects but at the same time
use the latest version for the newer projects.
Maybe this would also solve some of my other problems. I will take care
about it in a few weeks when i hope to have finished my lecture notes
and thus safely can switch to an up to date and clean setup including
the recent release of t-vim.
No worries. In the meanwhile, I'll see if I can get a better understanding of
the export mechanism.
|
|
Ok just had some time to test home. By replacing But the exporter seemst to have some problems with newline characters as it complains about |
|
If I change that line, I do not get any elements in the exported XML. Did you make any changes in |
|
Thats exactly the point where I'm currently struggling with 2020 ConTeXt version (in 2018 version it seemed at least still to work by chance). The accompanying pdf looks as should be. The only thing which i do see with my naive trial and error approach is by turning on export trackers is that it has some problem that the code block would sort before the first paragraph and that seems to be disliked by the exporter. Do you have an advice which channel to use for asking these kind of questions? |
|
On Mon, 18 May 2020, hernot wrote:
That is why i have sent an email concerning the exporter to the mailing list. But I'm not sure if to ask development questions the normal mailing list is the best place. Especially when it comes to the need to understand how lowlevel things like the exporter work or how the document tree looks like it has to process and choke on.
Do you have an advice which channel to use for asking these kind of questions?
The mailing list is the right channel because all the experts are on that list. But you do have to break down the problem into bite sized chunks and include easily reproducible examples.
Please don't take this as a criticism, but in my opinion your latest question did not receive any response because there was no way to reproduce the error message. You had linked to your pull request, but not stated what needs to be run in order to get the error message. Also, asking someone else to discuss internal code in a module can be difficult. If you can translate that into a smaller, stand-alone question, then there is a higher chance of positive replies.
|
|
Exactly, i was too clueless and disoriented to be able to get anywhere close to be able to break down to anything helpfull. I do fear for now I'm just the apprentice having to whatch and learn and not much help otherwise. Cause even with the answers you got i'm still as clue less and lost as before. |
|
By the way where do i find documentatoin how to properly debug my contributions, how do is get to know what of my thoughts about context internal workings are wrong and where do i find documentation about the designprinciples of context any of its modules. Is the mailing list the only source? |
|
On Mon, 25 May 2020, hernot wrote:
By the way where do i find documentatoin how to properly debug my contributions, how do is get to know what of my thoughts about context internal workings are wrong and where do i find documentation about the designprinciples of context any of its modules.
The high-level interface is documented in the manuals. There was a bug which I reported in the mailing list. Hopefully, that will be fixed in the next release.
Is the mailing list the only source?
Yes. Mailing list + reading the source (which used to very easy to read pre-luatex, but now is more complicated because functionality is split between lua and tex).
The actual code for tagging in exports is pretty simple. I pushed a new branch called export which works with my bugfix. The only remaining issues are linenumbering and maintaining spaces (which already happens for `\starttyping`, so it is just a question of mimicking that functionality).
I am waiting for a new release with the bugfix, so that I can then use to test further.
Aditya
|
|
Okay, first step towards the missing features. I pushed a new version 4772457, which now fixes the spaces in export. Now looking at line numbers. |
|
Linenumbering is also sorted. Was a backend issue. I am waiting for the context distribution to be updated and then we can test the export functionality. |
|
See branch gets exported to |
|
Hi adityam
Ok that i would have never sorted out, given the necessity of
contibutitions to core ConTeXt. Too many improvements there, that
making vimtyping output appear nicely in XML and E-Pub output would
just be some adoption of module code only.
Thank You very much for taking over.
I's there anything left, after new version of ConTeXt including your
patches has been released, for me to support you eg. helping in writing
Tests etc. ?
Otherwise if nothing is left for me to support with, I'd suggest to
close my (this) pull-request and continue with your branch.
Best
Xristoph
|
|
On Mon, 1 Jun 2020, hernot wrote:
Ok that i would have never sorted out, given the necessity of
contibutitions to core ConTeXt. Too many improvements there, that
making vimtyping output appear nicely in XML and E-Pub output would
just be some adoption of module code only.
Thank You very much for taking over.
I's there anything left, after new version of ConTeXt including your
patches has been released, for me to support you eg. helping in writing
Tests etc. ?
I still haven't looked at generating CSS. Since the module only provides a two built-in color schemes, I was actually thinking of including a hand coded CSS rather than an automatically generated one. But I need to look into that.
I'd also appreciate your help with testing to make sure there is no corner case.
Thanks,
Aditya
|
Unless changes in colour management are as radical as for tagging in latest mkiv and lmtx than in theory my suggestion on how to create css from colourscheme setup in context and hook it to css output list should still work. In other words no need for static css and for special cases users still can manually attach css through cssfile option of export.
Ok please keep me posted on progress and on which dedicated tasks or issues my support would be needed and/or apriciated. |
|
The code now works with the latest version of ConTeXt. Can you test the Please note that I have not yet added the CSS functionality (haven't had the time). I'll try to test your code and then add something appropriate. Note that ConTeXt does generate a bare-bones CSS definitions in |
|
Yes i will test with latest an report any Yes i know about
Given these i could not fully figure i decided against utilizing this file for placing vimtyping related stuff. And yes I admit i might have been a bit lazy and lousy to not approach those via Mailing list who could have helped to obtain more clarity upon above points ;-) |
|
Ok seems to work eventhough all syntaxgroups are for now maked as verbatim lines. Are there any obstacles where your expertise would be needed or is that something i could play with at least? Just for understanding: shouldn't 41-export.tex be placed in tests/vim as it is a tests for vimtyping and not basic t-filter? or is vimtyping just the vehicle or better the environment allowing to test the underlying changes to t-filter? |
|
Ok seems to work eventhough all syntaxgroups are for now maked as verbatim lines.
What do you mean by that? Here I get:
<vimtyping detail="PYTHON">
<verbatimline><syntaxgroup detail="Comment"># Python program listing</syntaxgroup></verbatimline>
<verbatimline><syntaxgroup detail="Statement">def</syntaxgroup> <syntaxgroup detail="Function">foobar</syntaxgroup></verbatimline>
<verbatimline> <syntaxgroup detail="Function">print</syntaxgroup>(<syntaxgroup detail="String">"</syntaxgroup><syntaxgroup detail="String">Hello World</syntaxgroup><syntaxgroup detail="String">"</syntaxgroup>)</verbatimline>
</vimtyping>
<paragraph>Now the same example with line numbering</paragraph>
<vimtyping detail="PYTHON">
<verbatimline><linenumber>1</linenumber><syntaxgroup detail="Comment"># Python program listing</syntaxgroup></verbatimline>
<verbatimline><linenumber>2</linenumber><syntaxgroup detail="Statement">def</syntaxgroup> <syntaxgroup detail="Function">foobar</syntaxgroup></verbatimline>
<verbatimline><linenumber>3</linenumber> <syntaxgroup detail="Function">print</syntaxgroup>(<syntaxgroup detail="String">"</syntaxgroup><syntaxgroup detail="String">Hello World</syntaxgroup><syntaxgroup detail="String">"</syntaxgroup>)</verbatimline>
</vimtyping>
Note that each line is marked as a `<verbatimline>` and each syntax group is marked separately inside it.
Are there any obstacles where your expertise would be needed or is that something i could play with at least?
As far as the XML is concerned, I think that this is okay. So, please feel free to play around with this.
I haven't had the chance to look at the CSS part yet.
Just for understanding: shouldn't 41-export.tex be placed in tests/vim as it is a tests for vimtyping and not basic t-filter? or is vimtyping just the vehicle or better the environment allowing to test the underlying changes to t-filter?
That was a mistake. Thanks for catching it. Fixed.
|
|
Ah yes you are right should have read carefully. And the rest is a matter of css.
whereas in xhtml it seems to recognize the corresponding css settnings. As far as i remember i had similiar struggles but i think it should resolve by dropping div, but I'm not quit sure how i got rid of that strange behaviour anymore. I just remember that it was gone somehow. If you want to compare i think you would have to replace syntaxline by verbatimline. But not sure whether this helps. |
|
I don't have a good understanding of CSS and HTML. I'll let you figure that out :-)
…On Jun 17, 2020, 2:59 PM, at 2:59 PM, hernot ***@***.***> wrote:
Ah yes you are right should have read carefully. And the rest is a
matter of css.
In html it appears with every elment on a single line as if firefox
would only see block setting for vimtyping and not get the css div
entries.
<img
src="https://user-images.githubusercontent.com/11776325/84936635-d61ff880-b0da-11ea-9cde-f1edf617db89.png"
width=350>
whereas in xhtml it seems to recognize the corresponding css settnings.
<img
src="https://user-images.githubusercontent.com/11776325/84936742-02d41000-b0db-11ea-9bb0-3faffb7d5544.png"
width=350>
As far as i remember i had similiar struggles but i think it should
resolve by dropping div, but I'm not quit sure how i got rid of that
strange behaviour anymore. I just remember that it was gone somehow.
--
You are receiving this because you commented.
Reply to this email directly or view it on GitHub:
#34 (comment)
|
|
Ok I'll re figure and pullrequest if necessary against your branch as soon as i think i do have a usable solution. EDIT
That was one reason why i added the automatic adding of vimtyping styles, as either user has his own custom css not related at all to vimtyping or has no clue how to get colorscheme right. The following finding or suggestions i do have but for some would need your further help to make work properly at least for some.
In html it is necessary to end each syntaxline by following it by a In xhtml this is not necessary. I think here you will have to look into, how to tell core to inject either
Possibly line number injection should be changed a bit that linenumbers are placed within dedicated
EDIT But for xhtml i do fear that it would be necessary to enclose line numbers in a EDIT In other words if you are ok i can take care of proper css. And creating more export tests.
|
|
I'm nearly there to make things work as expected for proper display of typeset syntax in export to html, xhtml and xml. Only one little thing is to be solved where i do need your advice and guidance how to make it work without breaking anything. The t-vim module is based upon t-filter and the outer most set of
The class name is set to the passed Therefore the tag for vimtiyping should better use colorsheme name as tagdetail information like so
and in general
How should i, may i or do i have to modify or extend the related line in t-filter.tex to be able to select an alternative value for the tag detail value instead of the value stored in |
|
I can think of two possibilities:
1. I add a parameter, `tagdetail` and use
\dostarttagged{\externalfilterparameter\c!taglabel}{\externalfilterparameter\c!tagdetail}
Then `t-vim` can set tagdetail to the name of the colorscheme (or perhaps both colorscheme and language; I have seen CSS with `class="name1 name2 name3"`).
2. We use `\startelement[taglabel][colorscheme=name-of-colorscheme, language=name-of-language]`. This is nicer because we get a more detailed XML, but it will require some refactoring as the current interface doesn't allow passing the list of key-value pairs). One also needs to globally enable `\setupelements[properties=yes]` to get the tags, and I am not sure how this interacts with other modules.
So, I am leaning towards the first, as it is trivial to implement.
|
|
which ever option you prefer. Eventhough how often one really utilizes different colourscemes for listings of the same language. I think lets start with option 1 and if usecases occur for second than refactoring can be done. Or do you already have existing documents where colorscheme and language should be included within tagdetail? Concerning |
|
I'll implement something with the first option.
`\setupelements[properties=yes]` is global for everything. The way the context backend code is written, it is not possible to localize it effect.
|
|
Are you sure about |
|
Sorry,
\setupexport[properties=yes]
or
\setupexport[properties=prefix]
|
|
Ah now i get what you mean, but from reading epub-mkiv.pdf I'm not sure if that is relevant for anything beyond structures created through "[...]The default output reflects the structure present in the document. If that is not enough you can add your own structure [...]" And as i view it t-filter and t-vim output is structure which is already present in the document. So I'm not sure if the properties parameter of Cause epub is basically zip archive including all files required to properly display the document, with the xml describing the pages, and some more informatoin hosted by additional meta files. And xml and xhtml do not differ that much. Maybe Hans can shed some light on it. |
|
There are two ways to implement tagging. The first, which we currently follow in t-filter/t-vim is which gets translated to The other option is By default, this gets translated to But, if \setupexport[properties=yes] is set, this gets translated to ...However, if \setupexport[properties=prefix] is set, this gets translated to ...I get the impression that \dostarttagging is for internal code while \startelement is for user code. The limitation of \dostarttagging is that we can only have one option. (There is also something which captures the inheritance structure with If we use \startelement in t-filter, then (i) we need to enable \setupexport[properties=yes] but this might interfere with user code. For example, in a personal document, I might or might not want properties to be enabled globally. (ii) If the user sets \setupexport[properties=prefix], then the output of t-filter/t-vim gets affected. I don't like that. I have implemented a version in |
|
So we do have the same impression that |
|
So we do have the same impression that `\startelement \stopelement` is for document editors which want to add custom elements which have no corresponding item or element in other output formats like PDF.
Yes
I would opt for sticking with `\starttagged \stoptagged` for now as it is more consistent with standard ConTeXt and has less side effects. Could it be that the chain attributed are filled by the third variant `\starttaggedchained` Where the detail specifier is followed by a reference to another tag? Not sure if that has any value for vimtyping.
I agree.
But doesn't that for now deviate a bit from the initial idea/request to get comparable results from vimtyping on default pdf output and html, xhtml and xml output.
Please look at the current export-2 branch and see if something is missing (apart from CSS). In my opinion the generated XML is fine. If you agree, then I can merge this to the main branch and then we can see how to include CSS.
|
|
Oh i have an uptodate copy of your export-2 on my repo and in my local directory already css included. Just need to run some further tests and than i would place a new pull-request against your export-2 and close this one, if that is ok for you. That pullrequest would than include most if not everything needed for merging into main. |
|
Please place it against export-2. export-2 and dev have deviated, so I'll rebase locally before merging.
|
|
OK done by #38. |
late happy easter
Managed to add taggs and css files to support xml/xhtml export and epub generation
This is a very first version. But line numbering will still need some indeep attention. currently only tested with number location left. Which works outside floats but already beraks inside ()
And i do expect even wors breaking for any other remaining placement ()
(*) line numbers are not placed immediately infront or after trags but immediately after float tag and before end float tag.
But for some advice it should be fair and OK enough.