\AtBeginDocument tokens from preamble get executed after the ones from an \AtBeginDocument in a package

[jfbu]

Brief outline of the bug

In debugging matters related to latex3/babel#190, which involved some usage of \AtBeginDocument by package numprint (or french.ldf) I encountered a very surprising fact that my own \AtBeginDocument added material ended up being executed after those emitted from packages.

(as I was not expecting this, it took me some time to figure out the puzzling illogical results of my actions)

With TeXLive 2019 this was not the case.

Real life involves me potentially writing to author of frenchmath package to tell him a possible way to overcome incompatibility of ncccomma with numprint in babel+french contexts, but now what I could say is complicated from the fact that one has to test LaTeX version.

_{if my advice had been for the doc to tell users to employ themselves AtBeginDocument... as I expect ordering still obeys expectation from inside packages.}

And to use I guess the hook mechanism with recent LaTeX.

I expect to be told the team is aware and has discussed it, but I did read the successive LaTeX News entries discussing hooks and have no remembrance that this rather striking fact has been explained there. (and quickly seaching for \AtBeginDocument in ltnews.pdf did not change that feeling).

I guess if not many breakage has been reported it is because most LaTeX users do not use \AtBeginDocument.

Minimal example showing the bug

\RequirePackage{latexbug}
\begin{filecontents*}{testpackage.sty}
\ProvidesPackage{testpackage}[2022/09/14 v1.0 JFB]
\AtBeginDocument{\show\SECOND}
\end{filecontents*}
\documentclass{article}
\AtBeginDocument{\show\FIRST}
\usepackage{testpackage}
\begin{document}
Nothing
\end{document}

Console output from executing latex

With TL2022:

> \SECOND=undefined.
\__hook begindocument ->\show \SECOND 
                                      \__hook_toplevel begindocument \__hook...
l.9 \begin{document}
                    
? 
> \FIRST=undefined.
\__hook_toplevel begindocument ->\show \FIRST 
                                              
l.9 \begin{document}
                    
? 

With TL2019:

> \FIRST=undefined.
\@begindocumenthook ...@series@setup \show \FIRST 
                                                  \show \SECOND \cs_if_exist...
l.9 \begin{document}
                    
? 
> \SECOND=undefined.
\@begindocumenthook ...\show \FIRST \show \SECOND 
                                                  \cs_if_exist:cT {ver@color...
l.9 \begin{document}
                    
? 

Log file (required) and possibly PDF file

testatbegindoc.log

[u-fischer]

code with the top-level label is executed last. This behaviour is intentional and documented in lthooks-code.pdf in section 2.1.6 The top-level label. It is also shown if you use \ShowHook on your example:

-> The hook 'begindocument':
> Code chunks:
>     testpackage -> \show \SECOND 
> Document-level (top-level) code (executed last):
>     -> \show \FIRST 
> Extra code for next invocation:
>     ---
> Rules:
>     ---
> Execution order:
>     testpackage.
<recently read> }

[mrpiggi]

This would work

%\RequirePackage{latexbug}
\begin{filecontents*}{testpackage.sty}
\ProvidesPackage{testpackage}[2022/09/14 v1.0 JFB]
\AtBeginDocument{\show\SECOND}
\end{filecontents*}
\documentclass{article}
\AddToHook{package/testpackage/before}
  {\AtBeginDocument{\show\FIRST}}
\usepackage{testpackage}
\begin{document}
Nothing
\end{document}

The top-level part could already been spotted at the original log: \__hook_toplevel begindocument ->\show \FIRST but the output of \ShowHook is even more lucid.

[u-fischer]

@mrpiggi I would prefer to give the code a label, or - in the real world if I want to ensure to be before the package code - a label + a hookrule

\AtBeginDocument[mylabel]{\show\FIRST}
\DeclareHookRule{begindocument}{mylabel}{before}{testpackage}

[FrankMittelbach]

The whole purpose of the hook management is to introduce "management" of hook code and not make it dependent on load ordering as it was the case in the past (for the few hooks that LaTeX had). The situation in the past was that packages competed for placement order giving incompatible advice (or worse couldn't be used together) because the only resolution posibility was to change the load order and with more than 2 packages involved that often doesn't work.

In contrast the hook management labels all code and if necessary an order can be defined on this level --- regardless of the order the packages are loaded.

Hook material that is set in the preamble is special (or rather the "top-level" label is special), because that is always executed last (or first if it is a reversed hook). The rationale is that user code wants to alter package code if code is added in the preamble (not to "signal" something to a package - which would be difficult to do with hooks) and the user might want to put that in a single file.

All this is documented in lthooks-doc, so yes it is by design.

And yes, since the hook management introduces a new (consistent) management concept there are naturally some differences to how things worked in the past and so depending on the situation one may need code that differs from before and after hook mgt was introduced.

If there is an incompatibility between two packages in how they add code to a hook (in that case begindocument) then both packages have now their code labeled (implicitly through the package name, or explicitly if they add a label). Thus, all that is necessary is to specify a hook management rule how the code should be ordered ... that is precisely the use case for the hook management. Either package can declare a rule for this and that will then work whether or not both packages are loaded.

This way nobody has to put something into the preamble to resolve the conflict or enforce a specific ordering of the packages, or both; they can handle that between themselves.

(as an aside: this is also the reason why misspelling a hook gives you no warning ... the hook management is happy to set up stuff for hooks that are not yet defined (because they come from other packages that may or may not get loaded) it therefore assumes you mean a hook that is defined later)

[mrpiggi]

I am just curious: Are there plans to implement some conflict detection mechanism? Currently, if a contradictory hook rule is defined, the very last one is applied without any notification and therefore, loading order is back in the game. This would certainly involve a lot of effort, especially with circular dependencies in mind. I don't have a actual use case, just thinking.

\begin{filecontents*}{pkgA.sty}
\ProvidesPackage{pkgA}
\AtBeginDocument{\show\pkgA}
\DeclareHookRule{begindocument}{.}{before}{pkgB}
\end{filecontents*}

\begin{filecontents*}{pkgB.sty}
\ProvidesPackage{pkgB}
\AtBeginDocument{\show\pkgB}
\DeclareHookRule{begindocument}{.}{before}{pkgA}
\end{filecontents*}

\documentclass{article}
\usepackage{pkgB}
\usepackage{pkgA}
\ShowHook{begindocument}
\begin{document}
Nothing
\end{document}

[FrankMittelbach]

No there are no such plans. It would mean checking that programmers do not make mistakes at the cost of each and every invocation runs extra tests. If pkgA and pkgB can't agree on who should go first then there is something seriously wrong :-) and they should sort it out, rather then each invocation of \DeclareHookRule makes such a test.

Besides, what happens if that errors? It would not be easy for a user to fix it does (and he or she shouldn't have to). Remember that this error doesn't show up for the maintainers of pkgA/B unless they actually make a test with both packages.

User mistakes are rather unlikely and therefore the is no such test.

But if our assumption is badly wrong that pkg maintainers understand who should go first and only add a rule when it makes sense and is needed, maybe we have to change our mind then, but for now ... no :-)

[jfbu]

I understand the management of hooks is powerful concept. And that it is documented. What surprised me is that old legacy usage of \AtBeginDocument was not left standing but transmogrified (I hope this is not pejorative, I am not native speaker) into the new hook management system. The ltnews said

At the moment the integration is lightweight, overwriting definitions made earlier during format generation (though this will change after more thorough testing). For that reason the documentation isn’t in its final form either and you have to read through three different documents:

The documents looks technical and so at that time I figured it would always be time to read them when I would get the time. I would never have inferred such a sweeping modification as executing user level \AtBeginDocument after all those of packages... now I dramatized a bit above when saying that if no breakage was reported it was because few users were aware of \AtBeginDocument; breakage occurred in my use case because I was trying to inject something which had to happen in-between two packages.

As background, I am sometimes contributing to some project for which users have no knowledge of LaTeX and are on often on Linux boxes were distributions of TeX/LaTeX lag by a few years ; so things requiring checking format version is never very good, from that point of view.

(posting now, your last comment appeared above and I have not read it yet)

[FrankMittelbach]

Am 15.09.22 um 22:34 schrieb Jean-François B.: I understand the management of hooks is powerful concept. And that it is documented. What surprised me is that old legacy usage of |\AtBeginDocument| was not left standing but transmogrified (I hope this is not pejorative, I am not native speaker) into the new hook management system. sometimes it is about time to drop "alte Zöpfe" as the German saying goes. We are not doing this lightly but this is one of the cases where it seems to us adequate. Making \AtBeginDocument to work differently to all other hooks in the future, just because it could have been used in a somewhat strange way in the past is not really helpful, it only complicates the behavior and the explanation of it. Now the behavior is consistent (and it should do what is expected in nearly every real use case) As I said it is really hard to make a good use case for the need to run \AtBeginDocument *before* a package is loaded and expect it to have an influence on on that upcoming package. I can understand that this surprised you, but that doesn't mean that it useful to keep that behavior as an exception to the overall mechanism. And yes, that means you have to check formats if you want to write code for older releases, but in the long run that is better than building models with inconsistent behavior for future generation of users.

[FrankMittelbach]

Am 15.09.22 um 22:34 schrieb Jean-François B.:

The documents looks technical and so at that time I figured it would always be time to read them when I would get the time. I would never have inferred such a sweeping modification as executing user level |\AtBeginDocument| after all those of packages... now I dramatized a bit above when saying that if no breakage was reported it was because few users were aware of |\AtBeginDocument|; breakage occurred in my use case because I was trying to inject something which had to happen in-between two packages.

but you realize that you don't have to after reading the documentation finally? If you really want to inject something between the \AtBeginDocument call of pkgA and the \AtBeginDocument of pkgB in the preamble all you have to do is to go \usepackage{pkgA} \AtBeginDocument[pkgA]{my injected code} \usepackage{pkgB} my point before is that the above is *not* something a user is very likely to do and if so he has to do it like that from now on.