etoolbox.sty support

[dginev]

Addresses #478

After a long time staring, there were only minor tweaks needed to get latexml to properly interpret the etoolbox.sty tex code and pass a basic test.

I have not tested on complex examples yet, and there are 4 warnings that indicate the begin/end hooks from etoolbox can't attach yet. But at the very least we're past the Fatal failures. I need to find myself a good torture test for etoolbox next.

[dginev]

I reworked this since it felt weird to unread an undef value in the empty input case.

[dginev]

strange oversight, adding fordefn back in makes the parameter read errors much saner (as opposed to wrong argument of undef)

[dginev]

I hit a Perl die here while playing around, patched it.

[dginev]

Technically, newcommand is defined in terms of argdef, not the other way around. etoolbox uses argdef directly, and I was surprised to see a simple aliasing to (latexml's) newcommand had great results.

[dginev]

This is a bit of a perverse change. The warning messages of etoolbox have a trailing \@gobble that seems to intend to remove the trailing line break. So to avoid an argument error, I had to add the trailing newlines in. Meanwhile I also noticed \MessageBreak had to be locally bound to ^^J (and expandable), but wasn't, so fixed that too.

[brucemiller]

Awesome! Looks like we're working on this from both ends :> I'd given up (quite easily) reading in etoolbox.sty, but started writing (w/o testing) bindings for it. A lot of macros and similar patterns. If we can interpret the sty, some will still be useful, I think. The document & environment hooks will need special attention.

[dginev]

Nice! This PR certainly reads in the native package successfully, but whether it does it correctly - I'm not so sure! Especially when it comes to the extra hooks etoolbox introduces. So your work may definitely be the key to get things working in full 👍

I'm having trouble locating a great test case "for free" though, maybe I'll write my own - but perfectly happy to borrow yours if you have one? :>

[brucemiller]

Ha! That's the thing, I wrote no tests at all, just coding, coding, coding :>

[dginev]

Pushing incremental changes here, as I expand the test coverage.

[dginev]

A side-remark here in passing - I've seen a number of errors due to "No more Input" that have to do with the Expand technique of using a standalone Mouth for the expansion. Some of the low-level tricks in the sty files end up reaching beyond the argument, so Expand may need an upgrade that can grab the next Mouth from the gullet when its sandbox is not enough?

[dginev]

Not something to address now, I will eventually have a perfect example that shows this, which we can use to upgrade. But bookkeeping to remember.

[dginev]

@brucemiller I got to the hooks section of the documentation, but your bindings may be called for here. Here is a simple test that should get you a document that says "hello world"

\documentclass{article}
\usepackage{etoolbox}

% --- preamble/postamble document hooks
\AfterPreamble{\secret\newline\renewcommand{\secret}{world}}
\AfterEndPreamble{\secret\newline}
\AtEndPreamble{\def\secret{hello}}
\AfterEndDocument{never printed}

\begin{document}
\end{document}

The --includestyles run relies on the etoolbox \patchcmd and \appto macros to override all of \begin, \end, \document and \enddocument for the hooks. I'm wondering if it would be better to make the pool bindings resilient to such redefinitions, or instead to ensure they are locked in a way where raw TeX interpretation can't override them...

[dginev]

Note from exploring the showcase examples - turns out babel's italian.ldf requires etoolbox explicitly now, and uses its hooks. So all the more reason to think it's time to add support.

[dginev]

A summary comment in the PR on the status of this binding:

• A lot of discussion happened in private email exchange with Bruce
• While debugging, I stumbled on some fundamental limitations with latexml-defined macros, which expand to Perl CODE(...) blocks, and have no obvious analogue to the \meaning-based parser in etoolbox
• This implies a weakened form of \patchcmd, and defining some of the patch-dependent commands directly in Perl
• Nevertheless, the PR improves on the native interpretation of the .sty file definition, and Bruce has improved master to facilitate that along the way.

At this point I am going through the documentation and writing as many tests as I can, at which point this PR should be good to go.

[dginev]

I now consider this PR with sufficient testing coverage, and welcome review and corrections. CC @brucemiller

[dginev]

Something I would like to understand better is whether this behavior of skipping spaces after the conditional test is something that should be done for all conditional types.

In other words, while patching \ifnum was all it took to get the etoolbox machinery to work, I wonder if it isn't better to patch DefConditional itself centrally, so that you can never declare a conditional that doesn't skip spaces after its test.

Would like to read up on this in the texbook possibly... Just a note for now.

[brucemiller]

I've just pushed a commit that fixes \numexpr (and friends) so that they ignore spaces after expansion; I think this is the reason you were tempted to add the SkipSpaces to some of the \if commands; you should be able to revert that change.

I'm also bugged by the change to object::Equals; Without this change, it seems that the only affect this has on the test cases is that pdflatex considers \begin to be "patchable" (apparently that \patchcmd should succeed?), whereas LaTeXML does not. However, I have to wonder: is \begin patchable in LaTeXML?

[dginev]

Wonderful, rebasing the branch now and checking quickly...

[dginev]

Sadly this was a high difficulty rebase, but with some extra care I think I got it right - in particular the same test tex->xml pair succeeds, and reviewing the final diff looks correct. Removing the skipped spaces indeed keeps the tests passing with the latest master, which is excellent.

[dginev]

As to the patchable cases, the real win are binding-defined expansions that do not take arguments, such as \stop, which the master-branch Object.pm wouldn't successfully compare via the \meaning parsing.

A minimal example document is:

\documentclass{article}
  \usepackage{etoolbox}
\begin{document}
\ifpatchable{\stop}{}{}{not}  patchable
\end{document}

This type of patching will succeed with the latexml run as well, so we do in fact succeed in boosting the raw interpretation. The macros that are impossible to patch are those that have one or more arguments, as the current serialization won't show the expected pattern by the meaning parse.

pdflatex prints "patchable", and so does the current etoolbox branch. However, using the Object.pm from the master branch prints the opposite "not patchable", demonstrating the issue.

And my main claim is that this is solving a general discrepancy, which covers patchable macros only in the specific case of etoolbox. Other packages use \meaning-parsing for other types of techniques as well, so the upgrade could have long-term gain for raw interpretation.

[brucemiller]

Maybe I'm misunderstanding, or forgot how patching was handled; Are you saying that \begin is patchable by \patchcmd?

[dginev]

I did not say that, no, I did not discuss the \begin case at all.

[dginev]

If I was to rephrase, I would prefer that Object.pm was capable of noticing a "tokenized" CODE(...) pointer returned by \meaning, as a general point of enhancing the technique I dubbed "\meaning-parsing", that is common in raw tex bindings.

If, separately from that, you would prefer that \ifpatchable should be reimplemented in a way that is completely accurate with the latexml implementation of patching, rather than aim for pdflatex-proximity, I can do that as well. But I see it as a higher level concern that is specific to etoolbox.

My perspective from the start was to try and get as much as possible of etoolbox.sty operational as raw tex, and I'm quite happy I stumbled on a couple of generic sticking points that will improve the raw interpretation for a potential number of additional packages.

[brucemiller]

It's always a dilemma; sometimes pretending to be (say) pdflatex is best, but we can't do everything pdflatex can. Likewise depends on what \ifpatchable might be used for; if it's used to check whether to use \patchcmd, then \ifpatchable should do false when we can't patch.

[dginev]

Great, I'm on the same page w.r.t the dilemma between following pdflatex behavior and being honest about what latexml can do. For the binding, I had - and just slightly enhanced - the \patchcmd macro, which is the user-facing feature in question, honestly produce Info messages that it can not patch non-expandable, as well as CODE-expandable, definitions. And return the false case.

Since \patchcmd is the user-facing macro, I am hoping this is both honest and provides the right messages to avoid confusion.

Meanwhile, the \ifpatchable macro is a bit lower-level, and has no further uses in etoolbox itself, which is why I leaned towards keeping it with its "raw tex" interpretation. Let me know if that is acceptable, for now, or if I can add any further upgrades.

I also added a lint pass to my last commit, so it is best seen with the ?w=1 flag on, as in the following link:
ce62b29?w=1

[brucemiller]

Great, cause now I'm having second thoughts! :>
Basically, I'm still inclined to think that we should claim something's patchable, but I'm also guessing that somewhere below here is effectively asking "are these two things the same definition". And even if one's been mangled by \meaning, one might expect LaTeXML to recognize them as still the "same", right?

[dginev]

And even if one's been mangled by \meaning, one might expect LaTeXML to recognize them as still the "same", right?

That's it! Exactly my point, yes. Could you be implying you're coming closer to liking the Object patch?

[brucemiller]

On 07/09/2018 08:54 PM, Deyan Ginev wrote: Could you be implying you're coming closer to liking the Object patch?

Accepting, but *not* liking!