Improve fragile macro support (textcase.sty)

[Grief]

When textcase.sty is included with --includestyles option, conversion fails with the following errors:

Error:expected:{ Expected <general text> here
	at textcase.sty; line 49 col 59
	Next token is T_OTHER[1]
	In Core::Definition::Expandable[\upperca... from TeX.pool.ltxml line 2097
	 <= Core::Gullet[@0x55e35ac94910] <= Core::Definition::Primitive[\edef Ski... <= Core::Stomach[@0x55e35a917868] <= ...

Fatal:misdefined:\MakeTextUppercase Expansion of '\MakeTextUppercase' has unbalanced {}
	at textcase.sty; line 49 col 59
	Expansion is \begingroup\def\i{I}\def\j{J}\def\({$}\let\)\(\def#1{\NoCaseChange{#1}}
	\def\label#1{\label{#1}}\def\ref#1{\ref{#1}}\def\protect\@ensuremath#1{\protect\@ensuremath{#1}}\def\@@cite[cite]{[\@@bibref{Refnum}{#}{}{}]}1#{\toks@{\cite#1}\@citex}\def\@citex#1{{#1}}\def\reserved@a#1#2{\let#1#2\reserved@a}\reserved@a\oe\OE\o\O\ae\AE\dh\DH\dj\DJ\l\L\ng\NG\ss\SS\th\TH\reserved@b{\reserved@b\let\@@protect\protect\let\protect\@unexpandable@protect\afterassignment\let\protect\@@protect\edef\reserved@a{\endgroup\@skipmath#1$\valign$}\reserved@a
	In Core::Definition::Primitive[\edef Ski... from TeX.pool.ltxml line 1142
	 <= Core::Stomach[@0x55e35a917868] <= Core::Gullet[@0x55e35ac94910] <= Core::Definition::Constructor[\Requir... <= Core::Stomach[@0x55e35a917868] <= Core::Gullet[@0x55e35ac94910] <= Core::Definition::Constructor[\docume... <= Core::Stomach[@0x55e35a917868]
1 error; 1 fatal error

[dginev]

Should I assume this is a request for us to add support for textcase.sty?

A quick look at the error you've pasted may suggest that the \uppercase macro binding in TeX.pool isn't playing well with the raw latex definitions

[Grief]

I might be wrong but as far as I understand, --includestyles does some parsing of additional .cls and .sty files and applies definitions from these files the same as if they were ltxml bindings. At least some of styles and classes work that way. I think that there are tons of different classes and styles in the world and adding all of them via bindings is too huge task to implement. For me, it would be ok if these styles and classes won't lead to fatal errors at least. Even if the macro won't work (I'd be glad if it is) it should provide some output at least.

[dginev]

Yes, we are completely on the same page here. The viable way to cover as big a subsection of LaTeX as possible would be to improve the native raw TeX coverage to the point we can ingest all latex packages and classes without any issues - or as you say at least without errors.

For now we also add an explicit .ltxml bindings to LaTeX packages we have tested, where the binding itself loads the TeX directly, so that we mark them as "supported" in a more obvious way. See for example t1enc.sty.ltxml

So I am still leaning to interpret this issue as "improve latexml so that textcase.sty works"

[Grief]

"improve latexml so that textcase.sty works" is fine. I'd prefer to see it works without the binding but if it's not possible then ok, bound solution would be good enough too. Sorry, I'm into neither TeX nor Perl so LaTeXML is mostly just a magic black box for me. What can I add is that we have more than a thousand custom classes and styles (including some hard legacy specimens) most of them doesn't work automatically which is very very sad. I tried to implement some binding but for me it appears to be too hard. During my frustration period I found the class which is most actively used in the documents I have and raised the current issue against the problem with it because I cannot do much here. You did a lot of work guys, but a huge amount of work still remains. Hope you'll not abandon the development. Cheers!

[dginev]

We are not abandoning anything, to the contrary! 👍

Feel free to share other hard to interpret cases, we will try to add support as we can, very valuable to have isolated examples that fail, as the usual situation with latex packages is that you need to unravel a very substantial execution chain to find the exact breaking point (especially if we are talking about a badly implemented primitive in latexml).

And thanks for the details of your use case!

[dginev]

Making some progress in diagnozing the issue here, this is actually a very well-contained package for debugging.

Just wanted to remark that a binding will likely be a lot more reliable than using the native implementation, because it is quite beastly. As one example that made me think of Rube Goldberg machines:

The $\valign$ is just a fake math expression used to terminate the parsing done by @skipmath.

textcase.pdf docs

I am looking for a generic interpretation fix though.

[dginev]

There are two sticking points here after debugging:

1. The GeneralText parameter type is too restrictive for this use of uppercase and lowercase. When wrapped around, the braces are often not readily available, and moreover single token invocations do work, such as \uppercase s. Let's say I can fix this by making the parameter type a bit more permissive.

2. More mysteriously, there is a use of edef I can't quite understand.

\protected@edef\MakeTextUppercase#1{\MakeTextUppercase{#1}}

pdflatex is more than happy to interpret this, and go ahead, but latexml tries to expand the inner #1 and crashes (again in the GenericText parameter type). I need to read up on this type of edef use, but if anyone has tips, they are most welcome.

Edit: It is not a generic deficiency, I think this is specific to this macro.
Edit 2: It's really related to the recursive invocations via \reserved@a and \reserved@b that are terminated by \@gobble, and I'm trying to carefully figure out the details without drowning now.

[dginev]

Some more diagnostic progress, there seems to be an interesting expansion bug, in arguably the most complex line of the textcase style:

      \expandafter\reserved@a\@uclclist\reserved@b{\reserved@b\@gobble}%

In particular, and I have verified this, there is an active invocation of \@gobble on the closing brace as its first argument. This is quite peculiar, and has to be due to the expansion machinery specific to these recursive unwindings, as an outright direct \@gobble{}} invocation returns the obvious error right away.

Will inspect further.

[brucemiller]

On 02/26/2018 10:01 AM, Deyan Ginev wrote: More mysteriously, there is a use of edef I can't quite understand. ***@***.***\MakeTextUppercase#1{\MakeTextUppercase{#1}} |

When I took a brief look at this issue before, this was the first error I encountered, and I suspect is the main one. Critical is that it's `\protected@edef`, not just `\edef`, and I suspect that LaTeXML isn't dealing quite properly with `\protect`. See https://tex.stackexchange.com/questions/5693/minimal-protectededef-example

[dginev]

Thanks for the tip! Reading into it some more, it seems in latexml we currently have \protect to always map to \relax, while the SO thread you linked to claims the definition tends to vary with execution context.

That said, I added a few \show\protect calls to the .sty file and reran my test with pdflatex, and it prints \relax each time. Maybe there is something more mischievous going on, but the basic protect-related infrastructure seems to be matching the latex definitions. I will carefully follow the execution chain, see where it diverges.

[brucemiller]

Actually, \protected@edef redefines \protect, so within the expansion of the definition, it should be a more protected \protect. Was just rereading the stackexchange page, and the referenced one about moving arguments. I'm trying to figure out how to create a test for LaTeXML that would correctly distinguis between fragile, robust, moving, etc. Clearly Expand() does the expansion without the execution (until later).

[dginev]

It's quite remarkable the amount of rope we need to do nothing, but keep the rest of the boilerplate unharmed.

I can mostly follow what you're looking into, but it sounds like we need a machinery upgrade rather than a simple fix, which is what I was hoping for when I took a sneak-peek here. Good to have it all hashed out!