expl3-generic.tex fails to load in ConTeXt MkII and MkIV

[Witiko]

The LaTeX interfaces documentation states:

latex3/l3kernel/doc/source3body.tex

Lines 58 to 62 in 2dcd78d

	The \pkg{expl3} modules are designed to be loaded on top of
	\LaTeXe{}. With an up-to-date \LaTeXe{} kernel, this material is loaded
	as part of the format. The fundamental programming code can also be loaded
	with other \TeX{} formats, subject to restrictions on the full range of
	functionality.

Since the latest weekly update of the texlive/texlive:latest Docker image, expl3-generic.tex fails to load in ConTeXt MkII:

$ docker run --rm -it texlive/texlive:latest
# apt update
# apt install ruby
# cat > example.tex
\input expl3-generic\relax
^D
# texexec example.tex
This is pdfTeX, Version 3.141592653-2.6-1.40.24 (TeX Live 2022) (preloaded format=cont-en 2022.10.30)  31 OCT 2022 10:08
entering extended mode
 restricted \write18 enabled.
 %&-line parsing enabled.
**example.tex
(./example.tex

ConTeXt  ver: 2021.03.05 19:11 MKII  fmt: 2022.10.30  int: english/english

system          : cont-new.mkii loaded
(/usr/local/texlive/2022/texmf-dist/tex/context/base/mkii/cont-new.mkii
systems         : beware: some patches loaded from cont-new.mkii
)
system          : cont-sys.rme loaded
(/usr/local/texlive/2022/texmf-dist/tex/context/user/mkii/cont-sys.rme
(/usr/local/texlive/2022/texmf-dist/tex/context/base/mkii/type-siz.mkii)
(/usr/local/texlive/2022/texmf-dist/tex/context/base/mkii/type-one.mkii))
system          : example.top loaded
(./example.top)
bodyfont        : 12pt rm is loaded
language        : patterns nl->texnansi:texnansi->1->2:2 nl->ec:ec->2->2:2 us->
ec:ec->3->2:3 de->texnansi:texnansi->4->3:3 de->ec:ec->5->3:3 da->ec:ec->6->2:2
 sv->ec:ec->7->2:2 af->ec:ec->8->2:2 gb->ec:ec->9->2:2 fr->texnansi:texnansi->1
0->2:2 fr->ec:ec->11->2:2 es->ec:ec->12->2:2 ca->ec:ec->13->2:2 it->texnansi:te
xnansi->14->2:2 it->ec:ec->15->2:2 la->ec:ec->16->2:2 pt->texnansi:texnansi->17
->2:2 pt->ec:ec->18->2:2 ro->ec:ec->19->2:2 pl->pl0:pl0->20->2:2 pl->ec:ec->21-
>2:2 pl->qx:qx->22->2:2 cs->il2:il2->23->2:2 cs->ec:ec->24->2:2 sk->il2:il2->25
->2:2 sk->ec:ec->26->2:2 hr->ec:ec->27->2:2 sl->ec:ec->28->2:2 tr->ec:ec->29->2
:2 tk->ec:ec->30->1:2 lt->l7x:l7x->31->2:2 agr->agr:agr->33->2:2 fi->ec:ec->34-
>2:2 hu->ec:ec->35->2:2 ru->t2a:t2a->37->2:2 uk->t2a:t2a->38->3:3  loaded
specials        : dvips loaded
\openout3 = `example.tui'.

\openout0 = `example-mpgraph.mp'.

systems         : system commands are enabled
language        : language en is active
specials        : loading definition file tpd
(/usr/local/texlive/2022/texmf-dist/tex/context/base/mkii/spec-tpd.mkii
specials        : loading definition file fdf
(/usr/local/texlive/2022/texmf-dist/tex/context/base/mkii/spec-fdf.mkii))
(/usr/local/texlive/2022/texmf-dist/tex/latex/l3kernel/expl3-generic.tex
Package: expl3 2022-10-26 L3 programming layer (loader)

(/usr/local/texlive/2022/texmf-dist/tex/latex/l3kernel/expl3-code.tex
Package: expl3 2022-10-26 L3 programming layer (code)

! Missing \endcsname inserted.
<to be read again> 
                   \bgroup 
\dotextmodediscretionary #1->\bgroup 
                                     \let \nextnextnext \egroup \def \next #...
<argument> __bool_|
                   _0:
\exp_args:Nc #1#2->\exp_after:wN #1\cs:w #2
                                           \cs_end: 
l.7288 \cs_new:cpn { __bool_|_0: }
                                   | { \__bool_get_next:NN \use_i:nnnn }
? X
 
Here is how much of TeX's memory you used:
 4312 strings out of 455439
 71317 string characters out of 5509318
 4870558 words of memory out of 9270903
 45695 multiletter control sequences out of 15000+600000
 82418 words of font info for 37 fonts, out of 8000000 for 9000
 581 hyphenation exceptions out of 8191
 35i,1n,31p,228b,542s stack positions out of 10000i,1000n,20000p,200000b,200000s
No pages of output.
PDF statistics:
 0 PDF objects out of 1000 (max. 8388607)
 0 named destinations out of 1000 (max. 500000)
 1 words of extra memory for PDF output out of 10000 (max. 10000000)

Furthermore, expl-generic.tex also fails to load in ConTeXt MkIV:

# context example.tex
open source     > level 1, order 1, name '/usr/local/texlive/2022/texmf-dist/tex/context/base/mkiv/cont-yes.mkiv'
system          > 
system          > ConTeXt  ver: 2021.03.05 19:11 MKIV  fmt: 2022.10.31  int: english/english
system          > 
system          > 'cont-new.mkiv' loaded
open source     > level 2, order 2, name '/usr/local/texlive/2022/texmf-dist/tex/context/base/mkiv/cont-new.mkiv'
system          > beware: some patches loaded from cont-new.mkiv
close source    > level 2, order 2, name '/usr/local/texlive/2022/texmf-dist/tex/context/base/mkiv/cont-new.mkiv'
system          > files > jobname 'example', input './example', result 'example'
fonts           > latin modern fonts are not preloaded
languages       > language 'en' is active
open source     > level 2, order 3, name '/mnt/example.tex'
open source     > level 3, order 4, name '/usr/local/texlive/2022/texmf-dist/tex/latex/l3kernel/expl3-generic.tex'
Package: expl3 2022-10-26 L3 programming layer (loader)
open source     > level 4, order 5, name '/usr/local/texlive/2022/texmf-dist/tex/latex/l3kernel/expl3-code.tex'
Package: expl3 2022-10-26 L3 programming layer (code)

tex error       > tex error on line 2641 in file /usr/local/texlive/2022/texmf-dist/tex/latex/l3kernel/expl3-code.tex: ! Missing number, treated as zero

<to be read again> 
^
l.2641   \tex_catcode:D `\^^
                          @ = 13

2631       {
2632         \__kernel_exp_not:w \exp_after:wN
2633           { \exp:w \__exp_eval_register:N #1 }
2634       }
2635     \cs_new:Npn \exp_not:v #1
2636       {
2637         \__kernel_exp_not:w \exp_after:wN
2638           { \exp:w \__exp_eval_register:c {#1} }
2639       }
2640     \group_begin:
2641 >>    \tex_catcode:D `\^^@ = 13
2642       \cs_new_protected:Npn \exp_end_continue_f:w { `^^@ }
2643       \if_cs_exist:N ^^@
2644       \else:
2645         \cs_new:Npn ^^@
2646           { \msg_expandable_error:nn { kernel } { bad-exp-end-f } }
2647       \fi:
2648       \cs_new:Npn \exp_end_continue_f:nw #1 { `^^@ #1 }
2649     \group_end:
2650     \cs_if_exist:NF \tex_expanded:D
2651       {

[u-fischer]

reset the catcode of ^ to 7 before loading expl3:

\catcode`\^=7
\input expl3-generic\relax

(I only tried with MkIV, I really don't see a point to test with MkII).

[Witiko]

reset the catcode of ^ to 7 before loading expl3

Thank you for the hotfix. Any chance for expl3 to reset the catcode internally as it did before the latest release of expl3?

On a tangential note: There are ConTeXt and plain TeX users of expl3. Would you be willing to have \input expl3-generic\relax executed by plain TeX and ConTeXt MkIV in the continous integration of LaTeX3? This would guarantee that expl3 can be at least loaded in these formats, as promised by the LaTeX3 interfaces documentation.

[josephwright]

reset the catcode of ^ to 7 before loading expl3

Thank you for the hotfix. Any chance for expl3 to reset the catcode internally as it did before the latest release of expl3?

We've been exploring in the context of the LaTeX2e kernel to what extend we could address the perennial issue of a non-fixed list of active chars. The idea then is to inherit as far as possible the catcode setup of the parent format. I'll have a look at this: we might want to have a cleaner internal approach to the ^^ syntax that doesn't rely on the catcode of ^ outside of a short, predictable range.

[josephwright]

On a tangential note: There are ConTeXt and plain TeX users of expl3. Would you be willing to have \input expl3-generic\relax executed by plain TeX and ConTeXt MkIV in the continous integration of LaTeX3?

MkIV is a bit tricky as it's not reliably updated in TeX Live (same for LTMX). But I really do wonder if the statement is true: I've seen no reports from ConTeXt end users (e.g. on the ConTeXt list), and the entire point of plain is to avoid using a sensible package/format setup ;)

[Witiko]

But I really do wonder if the statement is true: I've seen no reports from ConTeXt end users (e.g. on the ConTeXt list), and the entire point of plain is to avoid using a sensible package/format setup ;)

I would expect most ConTeXt end users to use document markup rather than expl3 directly. Therefore, they would only be affected if they used a module that uses expl3. Furthermore, this regression has only been introduced to CTAN in the previous week, so I would expect that many ConTeXt users have not yet updated to the latest version of expl3.

Definitely with you on the last point: expl3 can be difficult to sell to plain TeX users, since they can be ascetic in their tastes and sceptical of projects such as expl3 that are a radical departure from plain. I have given talks and written articles about expl3 for the Czech and Slovak TUG in the past year; plenty of plain TeX and ConTeXt users there.

[josephwright]

Interesting that Hans has gone for catcode-12 for ^ but 13 for |. Might ask him why :)

[FrankMittelbach]

12 seems odd, because it still breaks the ^^ notation, right? And if you aren't concerned about that then 13 seems a more logical catcode.

[josephwright]

@FrankMittelbach More the difference in treatment intrigues me. Hans likely doesn't care about ^^ syntax as he's say that it shouldn't be used in a document. (I'm not sure they need it at all in ConTeXt as messages, etc., are done by Lua not TeX.) I'm mainly surprised that there is the need for an active | (which is what leads to the error).

At the expl3 level, I could simply revert the change to catcode-resetting, or we could take a more 'guard against oddities' approach in the code, e.g. token_to_str:N | in place of just | in those c-type constructs.

[Witiko]

According to the ConTeXt manual (Section 7.8), the pipe character (|) is used to type compound words in ConTeXt. This is similar to the active characters set up by the different languages in the babel LaTeX package.