Ship remove Stdlibs in 1.9+

[LilithHafner]

The following script works on Julia 1.8.4

using DelimitedFiles
temp = tempname()
open(temp, "w") do io
    println(io, "1,2,3")
end
println(sum(readdlm(temp, ',', Int)))

Ideally it should work on all future Julia versions. In interactive use, once DelimitedFiles.jl is no longer a stdlib, the user will be prompted to install the package at the REPL. In package use, it should already be in the environment. As a scripts without a Project.toml, IIUC, the plan is to make this fail with ERROR: ArgumentError: Package DelimitedFiles not found in current path..

An alternate proposal from @IanButterworth on slack is

for scripts, if the package name cannot be resolved, and its in a special list known to Julia of names that used to be stdlibs, then auto-install it and carry on

Much like the interactive REPL prompt, but install & warn rather than prompt.

EDIT to clarify: This would be backed by downloading and loading DelimitedFiles with Pkg, not by packaging DelimitedFiles with Julia.

[brenhinkeller]

A second for Ian's proposal!

[KristofferC]

ERROR: ArgumentError: Package DelimitedFiles not found in current path..

A more accurate error message for this case, saying that the stdlib has been moved and to do Pkg.add("DelimitedFiles") is enough imho.

[Seelengrab]

Should the error also mention to consider using a project specific environment? I imagine the people hit by this could benefit from that the most.

[KristofferC]

I think that is a bit excessive. You can also hit a similar error even when using project envs when using a Manifest from 1.8 on 1.9:

julia> VERSION
v"1.9.0-beta2.8"

shell> cat Manifest.toml
# This file is machine-generated - editing it directly is not advised

julia_version = "1.8.4"
manifest_format = "2.0"
project_hash = "739f2f3af706c750957bd6ec39e0874ba8eb265d"

[[deps.DelimitedFiles]]
deps = ["Mmap"]
uuid = "8bb1440f-4735-579b-a4ab-409b98df4dab"

[[deps.Mmap]]
uuid = "a63ad114-7e13-5084-954f-fe012c677804"

julia> using DelimitedFiles
ERROR: ArgumentError: Package DelimitedFiles [8bb1440f-4735-579b-a4ab-409b98df4dab] is required but does not seem to be installed:
 - Run `Pkg.instantiate()` to install all recorded dependencies.
 ...

[LilithHafner]

A motivating use case for this is when I give someone a Julia script wrapped in a bash script that they can double-click to run (e.g. ManualImageCoding.jl via QuickDraw). It doesn't matter how explanatory the error message is because they don't speak Julia. It either works or it doesn't. Combining this with a Julia version shared across multiple users could lead to a situation where either user A's clickable application breaks inexplicably (to them) or other uses are pinned to a lower Julia version.

[Seelengrab]

If you're controlling the installation & execution of the script either way, I don't see an advantage over specifying --project in the shell script - something similar would be needed for general purpose standalone packages either way, since they presumably need to have a minimal launch_me.jl for starting in the first place.

For example, instead of activating the environment of the package, activating an environment for this installation seems more appropriate, to allow versioning of the package (irrespective of standalone packages not being affected by this, since they must already have DelimitedFiles in their dependencies).

[LilithHafner]

I don't see an advantage over specifying --project in the shell script

It is easier for non-technical users to handle a single file than a folder containing multiple files whose relationship must remain consistent.

they presumably need to have a minimal launch_me.jl for starting in the first place.

The shell script I distributed was a single file that is a cross platform relocatable executable (provided Julia is installed on that platform). I did not have any launch_me.jl nor did I have any user facing project or version management. The contents of the file were akin to:

julia -e 'import Pkg; Pkg.activate("ManualImageCoding", shared=true); Pkg.add(url="https://github.com/LilithHafner/ManualImageCoding.jl"); using ManualImageCoding; main()'

In my particular use case, my functionality was nontrivial so I put it all in a package. My dependance on DelimitedFiles is via a package, so my users would be fine either way. But if I had a simpler objective (e.g. horizontally concatenating delimited files) that might not have been the case.

I think that removing stdlibs without this feature is technically breaking. While IMO it would be an acceptable minor change, it's better to have this feature and avoid technical breakage altogether.

[brenhinkeller]

I think that removing stdlibs without this feature is technically breaking.

Yeah, it's absolutely breaking, and I think it's astounding that there is so much apparent opposition to a simple fix (just automatically install the ex-stdlib on using) that would make it entirely non-breaking

[giordano]

Yeah, it's absolutely breaking

The code still works as is. What changes is that you need to re-resolve the environment, which is something aside, and this is something you generally need to do anyway when changing versions of Julia.

[brenhinkeller]

No, it does not:

$ cat foo.jl
using DelimitedFiles
temp = tempname()
open(temp, "w") do io
    println(io, "1,2,3")
end
println(sum(readdlm(temp, ',', Int)))

$ julia-18 foo.jl
6
$ julia-19 foo.jl
ERROR: LoadError: ArgumentError: Package DelimitedFiles not found in current path.
- Run `import Pkg; Pkg.add("DelimitedFiles")` to install the DelimitedFiles package.
Stacktrace:
 [1] macro expansion
   @ ./loading.jl:1485 [inlined]
 [2] macro expansion
   @ ./lock.jl:267 [inlined]
 [3] require(into::Module, mod::Symbol)
   @ Base ./loading.jl:1466
in expression starting at /Users/cbkeller/foo.jl:1

[brenhinkeller]

Any way you try to spin it, it is a breaking change to the behavior of the Julia executable

And, incidentally, it's one that just impacted my entire class, thanks to the breakage it caused in the julia-vscode addon (c.f. https://julialang.slack.com/archives/C6LV2R5JB/p1672886310671379)