\fmtversion can get set to arbitrary value by latexrelease

[blefloch]

Brief outline of the bug

When loading latexrelease with an arbitrary date, that date is used for \fmtversion. It would be better if \fmtversion was always a really existing version, otherwise package authors doing \@ifl@t@r\fmtversion{some actual date of a release}, or those doing \@ifl@t@r{some actual date of a release}\fmtversion would get incorrect information about what (latexrelease-updated) version of the kernel is being run.

Minimal example showing the bug

\RequirePackage{latexbug}
\RequirePackage[2017/01/03]{latexrelease}
\documentclass{article}
\begin{document}
\fmtversion
\end{document}

Log file (required) and possibly PDF file

Experiment283.log

(reported by jfbu)

[car222222]

What “actual date” should be in \fmtversion?
Can Joseph fix this quickly?

[davidcarlisle]

I am not sure there is an issue to fix here.

by design the user level date option takes any date as it's not reasonable for anyone to remember the exact nominal release version dates.

The package could keep a list and force it to the nearest previous version but I'm not sure that is necessary, it is simpler to understand if the release date gets set to the date specified in the argument.

It is true that this doesn't rollback to a historically accurate state but (unless you rollback your entire tex system) that's always true. I think the latexrelease rollback just needs to be easy to distribute (a single file) and easy to understand.

[blefloch]

@davidcarlisle How should package authors test if the LaTeX kernel in use (including rollback/forward) has a certain feature? A natural thing to do is use \@ifl@t@r\fmtversion{some actual date of a release} or \@ifl@t@r{some actual date of a release}\fmtversion, but if \fmtversion can be an arbitrary date, package authors have to be much more careful about how they do their tests, and it needs to be documented.

[davidcarlisle]

@blefloch I think the onus on the person doing the rollback to roll back far enough. Typically people only roll back if they get some error, and at that point they need to roll back to a date that gives them the desired behaviour, so if a package is doing the "wrong thing" for formats after a certain date, then the user can roll back to any date before that.

If I was testing for a feature in a package It would be more robust to check for a command being defined, or having a specific definition than to do a date check.

I only see 15 files in tl2019 (other than base latex) using @IFL@t@r to test against \fmtversion, and some of those, like footmisx.sty: \@ifl@t@r\fmtversion{1999/12/01} are checking dates that are too old to have any accurate rollback anyway, LaTeX time jumped from 0000-00-00 to 2015-01-01 in a blur..

[FrankMittelbach]

I think this is as designed and conceptually working well enough so that I would not alter that behavior.

[blefloch]

I really think that being able to set the *format* date to an arbitrary value cannot be good design. There is a finite list of different kernel versions to support. In principle a package author should be able to do \str_case_e:nn { \fmtversion } { {2015/12/01} { ... } {2018/12/01} { ... } } (or whatever the actual dates of our releases are).

[davidcarlisle]

@blefloch I'm not sure that the list of dates is as clear cut, especially now that there are -dev releases and anyone can build from the git sources, which may have a format date that is never actually used.

Also (as comes up all the time in browser testing) I think version testing in that way is something of an anti-pattern that we should discourage rather than encourage.

If a package is doing

\str_case_e:nn { \fmtversion }
  {
    {2015/12/01} { ... }
    {2018/12/01} { ... }
  }

Then it has two code paths because presumably it needs some features available to use the second branch. The format version test is a rather weak test for that, the document may already have compensated for a missing feature in an older release by

\makeatletter
 ... some code 
\makeatother

in the preamble, or one of the packages loaded. It is better to test explicitly for the features that you need.

Apart from anything else we do not currently force that all \IncludeInRelease guards exactly match final public release dates. Looking in changes.txt it looks like 2018 had two releases, 2018-04-01 and 2018-12-01 but I see at least one \IncludeInRelease{2018/10/10}% in latexrelease, that could be adjusted to say 2018-12-01 but it is an extra burden on authoring, and there are probably more examples, that one just came up on a quick look by eye.

[FrankMittelbach]

Am 10.11.19 um 15:36 schrieb Bruno Le Floch:

I really think that being able to set the *format* date to an arbitrary value cannot be good design. There is a finite list of different kernel versions to support. In principle a package author should be able to do \str_case_e:nn { \fmtversion } { {2015/12/01} { ... } {2018/12/01} { ... } } (or whatever the actual dates of our releases are).

We could of course provide such a list as part of the kernel code, and could use it to turn a request into the nearest format date, but as David said it is to really a robust way to decide on what features are available. besides, currently this is all tied to the requested rollback date meaning that the user might really want to rollback to a date different from a format date because some package happened to have an update then and uses rollback on package level. Of course this could be untied, but right now partly depends on both becoming the same.

[jfbu]

I will reason without looking at actual LaTeX release dates. Assume 2012/01/01 and 2012/04/01 were LaTeX release dates with none other in-between. Imagine some user does \RequirePackage{latexrelease}[2012/02/01] not knowing exact LaTeX release dates, but perhaps knowing that some 2012/04/01 change breaks some package or some own macros. If I understand correctly (not really read the docs) the effect of this declaration is that all changes strictly after 2012/02/01 are disabled, hence it is exactly as if using LaTeX release 2012/01/01 but with \fmtversion expanding to 2012/02/01.

Now some package, whose author is more informed on release dates, has used the syntax \@ifl@t@r{2012/01/01}\fmtversion{A}{B}. As \fmtversion will be 2012/02/01 the branch B is taken. Hence the package author believes that the package is used with a LaTeX release strictly later than 2012/01/01, i.e. at least with 2012/04/01. Hence the package author will believe some LaTeX code is present whereas in fact it is not.

Hope I get this right, because I should have at least looked at doc of latexrelease to check if above description is correct.

Checking if some LaTeX internal macro is this or that is very cumbersome and bulky in a package source. Checking a release date is the more natural thing. Thus the release date must be trustable. As it is available via \fmtversion, the latter must be trustable.

Else some precise guidelines to package authors must be given about how exactly to check for which LaTeX kernel is used.

[FrankMittelbach]

Now some package, whose author is more informed on release dates, has used the syntax \@ifl@t@r{2012/01/01}\fmtversion{A}{B}. As \fmtversion will be 2012/02/01 the branch B is taken. Hence the package author believes that the package is used with a LaTeX release strictly later than 2012/01/01, i.e. at least with 2012/04/01. Hence the package author will believe some LaTeX code is present whereas in fact it is not.

Hope I get this right, because I should have at least looked at doc of latexrelease to check if above description is correct.

I would have loved to be able to tell you you should have looked, but unfortunately, at least on the definition of \@ifl@ter the documentation is wrong so looking wouldn't have helped (though it is documented elsewhere I'm fairly sure).

Anyway, technically this author of yours is not more knowledgeable than your user, because he would have tested for the wrong thing. The correct meaning for \@ifl@ter is "equal or later" so your line \@ifl@t@r{2012/01/01}\fmtversion{A}{B} is testing if release 2012/01/01 is present and that is true on that day and on any later day, so B would be correctly chosen in either case.

And if you think about it, that makes sense: you write such a line because you know that you want a feature for release X and at that time you may not even know what is in X+1 or when that is going to happen and you may not care either. So you like to use the date from release X and you don't want to specify one day be fore that release date.

By the way the fact that this macro is having several @ signs already indicates that it wasn't meant (initially) for package writers, the official way to get what you want is

\NeedsFormat{latex2e}[2012/01/01]

And this is quite clear in my opinion.

The same is used in \usepackage or \documentclass. If you write (today)

\documentclass{article}[2019/10/25]

you get no complains because that is the date in current article class, but if you write

\documentclass{article}[2019/10/26]

or any other "later" date you get

LaTeX Warning: You have requested, on input line 1, version
               `2019/10/26' of document class article,
               but only version
               `2019/10/25 v1.4k Standard LaTeX document class'
               is available.

which is precisely what \@ifl@ter implements.

Again \@ifl@teris not a package developer command (still we should correct its documentation).

[blefloch]

|\NeedsFormat{latex2e}[2012/01/01] |

This is the right thing to do if a package requires some feature introduced in 2012/01/01, but not if a package needs to run different code according to what the current release is. For that, I don't see how to avoid using \@IFL@t@r or run some complicated test of whether a definition is specifically this or that.

[jfbu]

Anyway, technically this author of yours is not more knowledgeable than your user, because he would have tested for the wrong thing. The correct meaning for \@ifl@ter is "equal or later" so your line \@ifl@t@r{2012/01/01}\fmtversion{A}{B} is testing if release 2012/01/01 is present and that is true on that day and on any later day, so B would be correctly chosen in either case.

The package author deliberately wants to check it the LaTeX format has a date less or equal to 2012/01/01. He then executes branches A or B depending on circumstances.

The alternative for the package author (who is knowledgeable on LaTeX release dates) would have been to use \@ifl@t@r\fmtversion{2012/04/01}{B}{A} but the package author did it the \@ifl@t@r{2012/01/01}\fmtversion{A}{B} way. If \fmtversion only could expand to some actual LaTeX release date, and I recall here we are playing with 2012/01/01 and 2012/04/01 being consecutive such dates, the two conditional tests are the same. But not if \fmtversion F is only constrained by D1<= F< D2 rather than being forcefully set to D1.

Now, I am not knowledgeable about how if at all latexrelease package can be used by packages, and it might be that user request date F is not a specification of a LaTeX release, but only a point in time. Quite possible, then I understand better some of David and Frank comments. But then there must be some \userrequestedpointintime thing, and I still believe \fmtversion should be constrained to be a "format version" as its name indicates, for those package authors who do not use services of latexrelease and possibly ignore it completely perhaps because they wrote the package some years ago and that nobody told them back then that it was bad habit to use \@ifl@t@r{date}\fmtversion and they should have exclusively used \@ifl@t@r\fmtversion{date}.

[FrankMittelbach]

> |\NeedsFormat{latex2e}[2012/01/01] | This is the right thing to do if a package requires some feature introduced in 2012/01/01, but not if a package needs to run different code according to what the current release is. For that, I don't see how to avoid using ***@***.***@***@***.*** or run some complicated test of whether a definition is specifically this or that.

quite right, 2e doesn't have an exposed interface for this because in 1993 we didn't expected a need for that ... all I I was commenting on is that \@IFL@t@r was not meant to be an interface which is why they name is not that accurate, eg not \@iflaterorequaltodate Remember back then size was counting everywhere even the pool size ie csanme length.

[josephwright]

On balance, the feeling is the behaviour here is right: latexrelease is setting the version of the format to what you would have based on the rollback request.

[blefloch]

(I think this is not a sensible choice.) There needs to be documented somewhere how package authors should test for a format version.

[davidcarlisle]

@blefloch I think this actually makes it easier for a package author to do that. If for example you want to branch based on an old or new version of \@ifundefined then you can look at the guards used and see that for that command there are two versions with guards

%<latexrelease>\IncludeInRelease{2018-04-01}{\@ifundefined}

and

%\IncludeInRelease{0000-00-00}{\@ifundefined}

So you can test for the format date being later than or equal 2018-04-01 whether or not there was ever a public released format really isn't relevant.

[jfbu]

@davidcarlisle if the dates in the guards are not constrained to match offical release dates then possibly a user can break LaTeX by requiring a date in between two guard-dates, if some functionality depends on coordination. I.e. something could be broken if feature A is there but not feature B, which is possible with arbitrary user-requested date.

[davidcarlisle]

@jfbu I don't understand your concern, a user may legitimately request a version between actual release dates, either to trigger a dated change in some package, or just because they don't know the older dates, if I want to force my texlive 2019 latex to act the same as a co-author's texlive 2017, I might just put

\RequirePackage[2017/01/01]{latexrelease}

and wind both systems back to some arbitrary date. It will be far harder to understand the effects of that declaration if it built in a mechanism to snap to the nearest previous actual release date.

[jfbu]

@davidcarlisle imagine you merge vairous branches into master, each introduced features or changes with some guard dates which are not the final release dates. Then the upstream testing will be done only at time of everything being merged. It may depend on the ordering of the sequence of merge into master. Imagine you have branches A, B, C tested on top of master. Branch A gets merged into master, continuous intgration testing passes. Then Branch B, then branch C. But the guard dates do not have the same order. Then a user can end-up with a LaTeX which has never been tested.

[FrankMittelbach]

Am 21.11.19 um 21:07 schrieb Bruno Le Floch:

(I think this is not a sensible choice.) There needs to be documented somewhere how package authors should test for a format version.

really just like they did in th past \ @ ifl@t@r\fmtversion{1999/12/01} {code executed if release is 1999/12/01 or later} {code executed if we are before 1999/12/01} the only issue is the the command really should have been called \@if@equal@or@later ... but that is "wrong" since 1993 but point is the code executes the same branch if you are between two releases regardless of whether \fmtversion contains the first date or a date inbetween.

[jfbu]

if it built in a mechanism to snap to the nearest previous actual release date.

not the nearest. The date D with D = DN <= user date < D(N+1)

[FrankMittelbach]

@FrankMittelbach were there guidelines for that? why not \@ifl@t@r{date}\fmtversion?

in the code. I already said previously that the documentation of that command is somewhat poor, but it is

\@ifl@ter {date A}{date B}{ code if A is equal or later B}{else}

and so the other way around is wrong -- and all package authors who had pushed to ctan in the past got it right.

It could still be documented better and I had made myself a note to do exactly that, but otherwise, it works just fine (and has done so for many years and the rollback doesn't change it, at least I haven't seen a convincing argument why)

[jfbu]

@FrankMittelbach I concur all package authors who needed to use \@ifl@t@r checked what it did and understood the "if later or equal to" thing. And possibly nobody did ever usage with \fmtversion second, but why would that have been intrinsically bad? if the LaTeX release date is prior to or equal to <date> then do something else do something else. Why should that be unnatural? It is dangerous indeed if \fmtversion does not necessarily match a tested LaTeX.

[FrankMittelbach]

Again as David said, one reason is that packages and the format may have different release dates, so if you have 1 LaTeX release in a year but in summer you have an update to xyz package

and a some point you find out you need to go back to what LaTeX looked like in summer of that you ie have the package xyz as it was after june andthe LaTeX release that was in june then

\usepackage[2015/07/01]{latexrelease}

does that for you, and that was the intention

[jfbu]

@FrankMittelbach I am obviously lost because I said there were only two LaTeX releases in my fictional world, year 2015. One on 2015/01/01 and the next one on 2015/12/01.

was there? I understood your fictional world had 2 release one in april one in october

Checking back on my comment I did say

But if (in some vritual world) there was only one 2015/01/01 and then one 2015/12/01 LaTeX release.

back to quoting your comment:

...
What we do now is this:

* right now we have release 2019/10/01 running

* _every (!)_ change with rollback that we work on for 2020/02/02 in develop has a InRelease guard of exactly that date, so there will be no internal dates any more so when we release in 2020/02 these dates and the rollbck roll forward match

ah!

maybe under that assumption my explanations make more sense?

definitely yes thanks! but then @davidcarlisle's earlier comment which triggered my activity here yesterday evening less so :)

In fact today, we even changed all such guards to move from 2020/02/01 to 2020/02/02 because we like the number :-) and will will for a change release on the second of feb

then, it looks as if all my concerns got preemptively addressed (and I was right to go to bed), especially as you promised adding some code comments on \fmtversion usage :)

[blefloch]

* /every (!)/ change with rollback that we work on for 2020/02/02 in develop has a InRelease guard of exactly that date, so there will be no internal dates any more so when we release in 2020/02 these dates and the rollbck roll forward match * this is the process we are working against by now and ofr some release already (it wasn't so day one, you live and learn :-)) maybe under that assumption my explanations make more sense?

That assumption has presumably been correct for a few releases by now; is it correct for all times? If so, my concerns can be completely address by commenting somewhere that \@IFL@t@r\fmtversion{date} is the right way to test that a format is later or equal to that date, while \@IFL@t@r{date}\fmtversion does NOT check that a format is before or equal to the date.

[FrankMittelbach]

maybe under that assumption my explanations make more sense?

definitely yes thanks! but then @davidcarlisle's earlier comment which triggered my activity here yesterday evening less so :)

well both David and I are developing/supporting LaTeX for more than 30 years, so in that case when you think of something you might forget changes in behavior (usually that happens to me not David though). The "internal" dates is what it has been all the time prior to introducing rollback and still during the first rollback year I guess, but it isn't any longer and it will stay this way, so rollback guards and nominal release dates will match.

If somebody would make a pull request and updates all early guards to match the formal releases even that could be done. But I guess this is overkill. As David said, a roll back is never a 100% emulation of what was there at the time in TL because many package don't participate (yet) and for the simple reason that you can't really roll everything back in practical terms.

But we are getting better over time here and syncing the InRelease dates with the nominal was an important step. I don't trust my memory here, but I think we do this since we introduced package rollback.

[FrankMittelbach]

That assumption has presumably been correct for a few releases by now; is it correct for all times?

is anything all the time? but yes, as we go further towards well-defined procedures this is unlikely to change

If so, my concerns can be completely address by commenting somewhere that @IFL@t@r\fmtversion{date} is the right way to test that a format is later or equal to that date, while @IFL@t@r{date}\fmtversion does NOT check that a format is before or equal to the date.

that's the intention.

[FrankMittelbach]

fix doc of \@ifl@ter

[davidcarlisle]

Rather than (only) fixing up the documentation of \@ifl@t@r if we are going to document that as a public interface it should probably be used to define an \@iffmtlaterthanorequal{...} that we tell package authors to use. The three @ form was never intended to be a top level interface it was just the internal support for \@ifpackagelater and \@ifclasslater so an \@ifformatlater (or perhaps add orequal to the name) would make a more consistent interface.

[davidcarlisle]

(like some other things latexrelease.sty would probably need to always add that, so format tests didn't fail if you roll back)

[FrankMittelbach]

We could do add this command and it would be good to have in theory, but would it help here? it would need at least 3-5 years to catch on and during that time packages would need to check if it exists to use it so you end up with horrible constructs like \ifx ... \@undefined ... around it and it would be necessary for Overleaf etc because their kernel is way behind.

So on the whole I think staying with the admittenly bad name is actually better.

[jfbu]

$ cd base
$ grep '%<latexrelease>\\IncludeInRelease' *dtx > guarddates
$ open -a Emacs guarddates
... do some quick search replace inside that file to remove all but non 0000 guard dates
$ sort -u guarddates 
2015/01/01
2015/10/01
2016/02/01
2016/03/31
2017-04-15
2017/01/01
2017/04/15
2018-04-01
2018/04/01
2018/10/10
2018/12/01
2019/10/01

(2020/02/02 is missing because I did the above on master branch not on develop branch)

My idea was then to manipulate the output of git tag semi-manually to recover release dates.

Then write a small Python script or actually do it manually as the above list is short and write a shell script with a number of sed -i commands which replaces each guard date by the earliest release date which is after (or equal :)) to it.

(the sed -i would be in a for loop over all dtx files and it would substitute a complete %<latexrelease>\\IncludeInRelease... stuff not blindly arbitrary dates; looping over all dtx files and doing all the sed -i on each file is of course very non-optimized but it will take a milli-second once the script if written)

Then git diff and spend some time checking up manually and git commit -m "Make all latexrelease dates coincide with actual release dates".

I have not done it because I don't know how amsmath is supposed to be handled etc... hence I will end up ask myself questions which would be superfluous but blocking to me. Idem with respect to tools etc...

[davidcarlisle]

@jfbu packages such a amsmath although sometimes released alongside base follow their own release cycles so certainly shouldn't be adjusted to latex release dates.

But even in base I honestly do not understand your concern, taking one of the dates at random from your list, say 2017/04/15 then I don't know without checking whether we pushed a release with that date to ctan, but I can't see whether it matters whether we did or did not. It may just have been an internal version and we incremented the date again before a release, but now, years later, I can't see it matters either way, and in fact there are some advantages to keeping the guards as written, as that lets you get back to that version. Just because the version wasn't released to ctan doesn't mean it didn't exist or didn't run through our full test suite.

As Frank says these days if we push a release date back we edit the current IncludeInRelease date guards to match, to save confusion, but I think that is a minor process change and there is no issue to solve here.

[jfbu]

@davidcarlisle I am only applying a kind of generic functorial approach to the underlying context, and if you say that there is no issue, I am 100% willing to believe you of course, and I do, so please be reassured I hold no concerns. I am clearly not familiar with the development model of LaTeX, and have more background in projects fully using git branches philosophy where there is no linear history and no certified version apart from releases.

[stale]

This issue has been automatically marked as stale because it has not had recent activity.

[FrankMittelbach]

Provide 3 CamelCase commands to branching based on format/package/class date deprecating (but keeping) the internal interfaces that have been used in the past