[xparse] Provide verbatim environment #591

Open
opened this issue Jun 17, 2019 · 13 comments

Skillmon commented Jun 17, 2019

 It would be nice if xparse would support a combination of the +v type and the b type arguments to provide verbatim environments.
Member

josephwright commented Jun 17, 2019

 If we want a letter: f as it's like filecontents?
Author

Skillmon commented Jun 17, 2019 • edited

 No idea on the letter. f might be misleading as it is an expansion type in expl3 but I can't think of something more reasonable. Also what would the behaviour of \begin{myverbatimenv} \begin{myverbatimenv} \end{myverbatimenv} \end{myverbatimenv}  be? Currently all the verbatim environments in LaTeX I know of would fail in this example, but the v type does care for matching braces if it's opened by a brace, should there be a matched environment option? EDIT: If the above (matching environments) isn't considered as the behaviour for the actual implementation (which I'd understand), it'd be nice to have a code-level macro to restart argument grabbing to implement it on a per-use-case-basis (no idea how this could be implemented robust though).
Member

PhelypeOleinik commented Jun 17, 2019

 No idea on the letter. f might be misleading as it is an expansion type in expl3 but I can't think of something more reasonable. Well, o is also an expansion type in expl3. I don't think there would be a problem...
Author

Skillmon commented Jun 17, 2019

 @PhelypeOleinik good point.
Contributor

FrankMittelbach commented Jun 17, 2019

 Am 17.06.19 um 11:16 schrieb Skillmon: No idea on the letter. |f| might be misleading as it is an expansion type in |expl3| but I can't think of something more reasonable. Also what would the behaviour of |\begin{myverbatimenv} \begin{myverbatimenv} \end{myverbatimenv} \end{myverbatimenv} | be? Currently all the verbatim environments in LaTeX I know of would fail in this example, but the |v| type does care for matching braces if it's opened by a brace, should there be a matched environment option? for braces that support seems reasonable as you often have braces in "verbatim code fragments" and if that is supported at least if they match that good. But to provide fairly complicated code that is only ever going to apply in its own documentation that seems overkill in my option so the above should just close the environment on the first |\end{myverbatimenv}| in my opinion.
Contributor

blefloch commented Jun 17, 2019

 Nestability is a bit of a detail. IIRC many verbatim packages read lines one by one and typeset them directly without waiting for the end of the environment. Should we provide something like that? Another issue is catcodes: it is actually not so convenient to get the lines in the same way as provided by \readlines, with ^^M at the end. I don't see right away what would be a good interface to make this customizable. Another question is whether the code should be clever enough to make the following work (by looking at \@currenvir)? \NewDocumentEnvironment{vrb}{f}{...}{...} \newenvironment{foo}{\vrb}{\endvrb} \begin{foo} test \end{foo}
Author

Skillmon commented Jun 17, 2019

 @blefloch for +v I currently use something like \cs_new:Npx \__mymodule_process_verb_newline:nnn #1 #2 #3 { \tl_set:Nn \exp_not:N \ProcessedArgument { #3 } \tl_replace_all:Nnn \exp_not:N \ProcessedArgument { \char_generate:nn { 13 } { 12 } \char_generate:nn { 13 } { 12 } } { #2 } \tl_replace_all:Nnn \exp_not:N \ProcessedArgument { \char_generate:nn { 13 } { 12 } } { #1 } }, that works pretty well to customize the behaviour of newlines (e.g. the double one replacing with \par for \scantokens). I don't think reading and outputting lines one by one would match the established behaviour of xparse's b type argument, and I'm unsure how a front-end for that behaviour inside \NewDocumentEnvironment should look like, should the begin and end code be executed for each line? Nesting of verbatim like environments was always a bit of a hassle, IIRC fancyvrb had a solution for that.
Member

PhelypeOleinik commented Jun 17, 2019

 IIRC many verbatim packages read lines one by one and typeset them directly without waiting for the end of the environment. Should we provide something like that? If I understand correctly, the purpose of combining b and v is to grab the environment verbatim (duh) and then process it somehow (e.g., apply formatting, line breaking, frame the whole thing, write to file, etc.) so I think that grabbing the entire environment is what we want, otherwise the entire thing could be set up as a normal environment with the verbatim catcode setting at the \begin part. Another issue is catcodes: it is actually not so convenient to get the lines in the same way as provided by \readlines, with ^^M at the end. I don't see right away what would be a good interface to make this customizable. Stupid idea, but what about putting everything in a seq variable, one item per line? It would be really easy to process it later in the code...
Contributor

wspr commented Jun 18, 2019

 If xparse were to provide any support here, I think it should be at quite a low level but with appropriate hooks to define more useful packages like fancyvrb. So if the environment handled the verbatim reading, and then wrapped each line in a macro pair like \xparse_verbatim_process_line_begin:w … \xparse_verbatim_process_line_end: With the idea these functions (by default no-op) would be defined appropriately by the code author. I don’t have much to say around implementation edge cases. I agree with Frank that nesting is probably unimportant, but handling things like \begin{verbatim} … \end{verbatim}foo varies between different approaches, and speed is probably of (some) concern.
Contributor

blefloch commented Jun 18, 2019

 Support for turning newlines or pairs of newlines to other things (like in Skillmon's code) could be done by an argument processor. The drawback is that it is more efficient to directly grab things with ^^M having catcode active and define it to whatever one wants. Perhaps for efficiency we should provide a separate set of commands for verbatim environments read line by line. \NewVerbatimEnvironment etc.
Author

Skillmon commented Jun 18, 2019

 @PhelypeOleinik would be easy enough to put it in a seq inside the environments code (\seq_set_split:Nxn \l_tmpa_seq { \char_generate:nn { 13 } { 12 } } { #1 }), no need to set this up as the default for the environment. I think providing a processor (or two, one that replaces all with a single tl, one that replaces multiple consecutive newlines with one tl, and single newlines with another tl) would be the best option.
Author

Skillmon commented Jun 18, 2019

 Prototypes of those processors could be something like the following: \documentclass[]{article} \usepackage{xparse} \ExplSyntaxOn \bool_new:N \l__xparse_is_in_bool \cs_new_protected:Npn \__xparse_replace_newline_loop:nnn #1 #2 #3 { \tl_if_in:NnTF \ProcessedArgument { #1 } { \bool_set_true:N \l__xparse_is_in_bool } { \bool_set_false:N \l__xparse_is_in_bool } \bool_while_do:nn { \l__xparse_is_in_bool } { \tl_replace_once:Nnn \ProcessedArgument { #1 } { #2 } #3 \tl_if_in:NnTF \ProcessedArgument { #1 } { \bool_set_true:N \l__xparse_is_in_bool } { \bool_set_false:N \l__xparse_is_in_bool } } } \cs_new_protected:Npx \ReplaceNewlineB #1 #2 #3 { \tl_set:Nn \exp_not:N \ProcessedArgument { #3 } \__xparse_replace_newline_loop:nnn { \char_generate:nn { 13 } { 12 } \char_generate:nn { 13 } { 12 } } { \exp_not:N \c__xparse_consecutive_newline_mark_tl } { \__xparse_replace_newline_loop:nnn { \exp_not:N \c__xparse_consecutive_newline_mark_tl \char_generate:nn { 13 } { 12 } } { \exp_not:N \c__xparse_consecutive_newline_mark_tl } {} \tl_replace_once:Nnn \exp_not:N \ProcessedArgument { \exp_not:N \c__xparse_consecutive_newline_mark_tl } { #2 } } \tl_replace_all:Nnn \exp_not:N \ProcessedArgument { \char_generate:nn { 13 } { 12 } } { #1 } } \cs_new_protected:Npx \ReplaceNewlineA #1 #2 { \tl_set:Nn \exp_not:N \ProcessedArgument { #2 } \tl_replace_all:Nnn \exp_not:N \ProcessedArgument { \char_generate:nn { 13 } { 12 } } { #1 } } \NewDocumentCommand \myvrbA { >{\ReplaceNewlineA{\\\null}}+v } { \group_begin: \ttfamily #1 \group_end: } \NewDocumentCommand \myvrbB { >{\ReplaceNewlineB{~}{\par}}+v } { \tl_rescan:nn {} { #1 } } \ExplSyntaxOff \begin{document} \myvrbA {this is verbatim material} \myvrbB {this is read verbatim but gets rescanned} \end{document} 
Contributor

FrankMittelbach commented Jun 18, 2019

 Am 18.06.19 um 03:12 schrieb Will Robertson: If xparse were to provide any support here, I think it should be at quite a low level but with appropriate hooks to define more useful packages like fancyvrb. So if the environment handled the verbatim reading, and then wrapped each line in a macro pair like I don't really think that xparse should provide *any* low-level functionality targeted for programmers ... that belongs elsewhere. xparse is about providing document level interfaces not programmer-level i.e. if people use xparse top-level for internal code (as I do sometime see) then something is wrong imho