Update packages that rely on 7z.exe being in the Sys.BINDIR folder

[davidanthoff]

EDIT: Please take a look at the end of this issue for what the TODO is at this point, after discussing this issue.

If I understand the recent changes to this, then the idea for julia 1.3 is to move 7z.exe and 7z.dll to the libexec folder, right? That is how I'm interpreting #33493.

But, I think that is a change that will break a whole bunch of packages that have an assumption built in that 7z.exe is actually located in the Sys.BINDIR folder. Examples are here, here and here (I believe there are more packages that have the same issue). I think those are way too central packages to break, so I think really 7z.exe and 7z.dll need to stay in the Sys.BINDIR folder, given that there is a promise that Julia 1.x versions are not breaking.

I believe this should be blocking for 1.3.

Did PkgEval pick this up? Or is PkgEval only run regularly on Linux? I think this is an example of why it would be crucial to run PkgEval on Windows as part of the regular release process.

CC @staticfloat

[KristofferC]

Just make a PR to fix the package? It is like a one line change?

[musm]

I think the best solution is to have Sys.LIBEXECDIR ref #33493 (comment)

[davidanthoff]

Just make a PR to fix the package? It is like a one line change?

That is not the problem. The problem is that there is a promise that old code will work on newer versions of Julia 1.x, and this breaks that promise. And as far as I can tell, this will be very widespread: pretty much anything that has a binary dependency is probably affected on Windows, right?

The particular scenario I have in mind is this: someone published a replication code for a paper. It is a zip file, hosted on something like zenodo.org, with a Project.toml and Manifest.toml in the root. The replication instructions say to to pkg> activate . and pkg> instantiate and then run a script. With this change, that won't work anymore. I think the backwards compat promise really needs to cover this scenario, at least for the academic setting this is really a core use-case.

In general, I really hope that the interpretation of "we don't break things" is not interpreted "if there is a simple one line fix that can be applied in user code for a breaking change in Julia, we are ok to include that in a Julia 1.x release". I feel strongly that backwards compat should work under the assumption that old code needs to run without any modifications. There might obviously be corner cases, and I think a good criterion in that case is to look at how large spread the impact of a change is. The impact of this particular change to me seems very large, because I think it will break pretty much anything that has a binary dependency on Windows.

[musm]

@davidanthoff I understand where you're coming from, but the number of packages this affects is actually quite small, and is less a problem than Julia pre version 1, since we have now transitioned to BinaryProvider.

I actually made PRs to roughly 8 package a while ago when we had to explicitly specify 7z.exe due to some internal changes. (These were all the public packages I found that use 7z.exe.)

Looking back only 7 of these packages need to get updated to accommodate the latest changes (the others have transitioned to BinaryProvider).

(1) The best solution is to provide Sys.LIBEXECDIR and a suitable Compat definition.

Affected packages include Electron.jl Taro.jl TimeZones.jl BinaryProvider.jl MXReader.jl Weber.jl BinDeps.jl

I'm more than happy to make suitable PRs to these packages to accommodate these packages.

But for the PRs to be robust to future changes we should probably first do (1).

[ararslan]

Or is PkgEval only run regularly on Linux?

I think it only actually works on Linux.

[KristofferC]

The problem is that there is a promise that old code will work on newer versions of Julia 1.x, and this breaks that promise.

That has never been a promise (it is impossible to fulfill since literally any change is possible to observe and therefore relied on by a package). The promise is to not break public API.

The particular scenario I have in mind is this: someone published a replication code for a paper.

Post the Julia version as well.

[davidanthoff]

the number of packages this affects is actually quite small, and is less a problem than Julia pre version 1, since we have now transitioned to BinaryProvider.

I don't think that is the right metric here. Any package that depends on BinaryProvider.jl, or BinDeps.jl is currently affected by this issue. That is a huge number of packages.

The problem is that there is a promise that old code will work on newer versions of Julia 1.x, and this breaks that promise.

That has never been a promise (it is impossible to fulfill since literally any change is possible to observe and therefore relied on by a package). The promise is to not break public API.

I understood @StefanKarpinski here as saying that essentially minor versions won't break things in the package ecosystem ("[...] which, in fact, do not break things in the package ecosystem as determined by running PkgEval to verify that there isn’t any breakage."). This change here clearly breaks a lot of things in the package system, so I think it should not be made. Now, I guess one can argue that this is an internal change, not part of the public API, and therefore ok. I'm not a fan of such an argument. To the users that are affect by this it doesn't matter whether this was ok by a narrow reading of some policy, at the end of the day things don't work for them and I think the goal should be to avoid that.

The particular scenario I have in mind is this: someone published a replication code for a paper.

Post the Julia version as well.

I don't think that is a good solution. I think it would be way more user friendly if newer 1.x versions simply run code that ran under a previous version of Julia. I'm pretty positive that most folks that read the official release plan process blog would expect that. Also, if this is a legit argument, then why do we have to care about backwards compat at all? We can always tell folks to just run things with an older version of Julia.

It seems to me that another factor in a decision like this is how bad a problem is fixed by this change. I agree that in principle this change is for the good, but on the other hand, are we aware of a single bug report where someone had a problem with the pre 1.3 approach? If not (or the number of users for whom this is a problem is low), I think that is one more argument to make this change in Julia 2.x.

I also found another bit of info re testing this: I think testing BinaryProvider.jl on the latest Windows 10 version won't unearth this problem: it turns out that MS started shipping tar as part of Windows 10 in system32 in more recent Windows 10 versions, so BinaryProvider.jl then just falls back to using that by default on Julia 1.3. But, I believe that both older versions of Windows 10, Windows versions pre 10, and julia 32 bit on Windows 10 don't take that path, and things fail. I think that explains why the 64 bit tests pass, but the 32 bit tests fail in the tests for SymbolServer.jl.

I think at the end of the day that changes that break existing package versions and affect a large number of packages (direct or indirect) shouldn't ship as a minor version.

[Keno]

There is no guarantee that code in 1.x will run as is in 1.x+1. The guarantee is that we won't change stable interfaces. The location or existence of 7z is not a stable interface. The packages that need to know about it will get fixed and users will be happy. For exact reproducibility the julia version always needs to be included. This is exactly the difference between something that can go into a patch release and something that can go into a minor release. I do agree getting the packages fixed should block the 1.3 release.

[KristofferC]

I don't think that is a good solution. I think it would be way more user friendly if newer 1.x versions simply run code that ran under a previous version of Julia.

Unless you vet all the dependencies to make sure they are not relying on internals of Julia you need to give the Julia version as well to ensure reproducibility.

then why do we have to care about backwards compat at all? We can always tell folks to just run things with an older version of Julia.

Because we want people that program against the public stable interface to have their code still work because that makes Julia a nicer language to use. But for things where you really want proper reproducibility you want to give Julia version + Manifest anyway.

[davidanthoff]

Ok, I have to say, I'm not happy with this policy, but ok.

In any case, can someone add this issue to the Julia 1.3 milestone, where the actual TODO would be to update all the packages that currently rely on the old location for 7z. As @Keno suggested, I think 1.3 should only be released once those packages have new versions tagged that are updated.

[musm]

relevant PR #33777

[KristofferC]

I think 1.3 should only be released once those packages have new versions tagged that are updated.

We don't hold the release for a bunch of other packages that rely on internals (but we try to open issues when we find them).

This issue has been opened for soon 3 weeks. Since the fix is so small, it either isn't that important, or everyone wants someone else to do the job. Just cutting the 1.3 release should make someone step up.

[davidanthoff]

bunch of other packages

Well, in this case that includes BinaryProvider.jl, which is a) very widely used and b) the official binary story from the core Julia team, as far as I can tell. Yes, you can tell a story that BinaryProvider.jl used non-public APIs and therefore it is ok to break it, but I think a much more user friendly approach is to just make sure it works before 1.3 ships.

it either isn't that important

I think it is important but not widely detected at this point. As far as I can tell, anything using BinaryProvider.jl will not work on any Windows version that is not of the latest kind (where tar ships out of the box). I don't have the latest numbers at hand, but if I remember our telemetry from the VS Code extension that is potentially a very large group of users. At the same time it seems also super likely that the crowd of folks that actually develop Julia is not running into this issue because a) all the CI Windows solutions run on the latest version of Windows (so the only breakages we see right now are on 32 bit Windows, where almost all builds for all my packages are broken because of this), b) devs might be on a recent Windows version, c) devs might have some of these unpacking programs in their PATH.

This issue has been opened for soon 3 weeks

Well, I would have fixed the problem in my packages, but we needed #33777 first and then a new RC build that includes that so that I can actually test it.

So, I think at least BinaryProvider.jl should really be in a working shape before 1.3 ships, and so I think that warrants adding this issue to the 1.3 milestone.

[staticfloat]

So, I think at least BinaryProvider.jl should really be in a working shape before 1.3 ships, and so I think that warrants adding this issue to the 1.3 milestone.

Does BinaryProvider v0.5.8 not work for you? It should be looking in both bin as well as libexec, as desired.

[davidanthoff]

Does BinaryProvider v0.5.8 not work for you? It should be looking in both bin as well as libexec, as desired.

Does it? This to me looks like it is only looking in bin. But I can't really test it, because the last binary for Julia I have is RC4, and I think there the whole story is broken because the exe and the dll are in different directories, right?

[staticfloat]

Ah, that's because the latest release is actually on a different branch, not on master. This was the easiest way to roll back a bad release without outright reverting commits, but those commits will probably never be fixed now that 1.3 is trying to eradicate BinaryProvider.

[davidanthoff]

Hehe, that was well hidden :) Great!

Is that the recommended current way of handling this in other packages as well? Or should I use #33777 instead?

Any chance we might get a new RC soonish that also includes #33777? That would make it a lot easier to fix this stuff in packages.

[musm]

Question: Should we include the LIBEXECDIR addition to Compat? (see: JuliaLang/Compat.jl#676) I'm having second thoughts because as was pointed out this is not part of the stable API, but at the same time it can help existing users more easily upgrade their packages?

[giordano]

@oxinabox went ahead and opened the following PRs:

• Fix 7zip path for julia 1.3 JuliaPackaging/BinDeps.jl#407
• error if read user input 100 times without success oxinabox/DataDeps.jl#105
• Fix 7zip path on Julia 1.3 JuliaTime/TimeZones.jl#230

[davidanthoff]

Ok, so the right code to use is:

if isdefined(Base, :LIBEXECDIR)
  const exe7z = joinpath(Sys.BINDIR, Base.LIBEXECDIR, "7z.exe")
else
  const exe7z = joinpath(Sys.BINDIR, "7z.exe")
end

right? I'll do that for Electron.jl and NodeJS.jl.

Having said that, two questions:

1. Why is LIBEXECDIR not in the Sys module like BINDIR?
2. Why is LIBEXECDIR not an absolute path, like BINDIR? Seems weirdly asymmetric...

More asking out of curiosity, I assume that at this point we don't want to change things again.

[giordano]

Having said that, two questions:

1. Why is LIBEXECDIR not in the Sys module like BINDIR?
2. Why is LIBEXECDIR not an absolute path, like BINDIR? Seems weirdly asymmetric...

More asking out of curiosity, I assume that at this point we don't want to change things again.

Lots of literature in #33777

[vtjnash]

Since 1.3 is done now, I'm going to say these questions are settled now, de facto