allow changing stdlib dir in context

[Roger-luo]

Tho in most cases we can assume Pkg ships with a complete installation of Julia, but when one generates a standalone app with PackageCompiler, this might not be true, and not all the files in stdlibs will be shipped with such kind of app. But there are cases that packages might depend on Pkg:

• they might be an enhancement tool on julia's package/project, e.g PkgTemplates
• the user may use Julia's Pkg system as a hot plugin loading system in their standalone application since Julia is dynamic and can actually load packages in runtime, and update methods table by requirements.

This PR make stdlib_dir available to change in Context, so this won't prevent developers to ship their own standalone app with certain stdlib files (or alternative stdlib files located other places).

[KristofferC]

What's the current behavior? Does it error? Perhaps Pkg should read and store the stdlib projects during compilation time of it.

[Roger-luo]

yeah, the simplest case is wrapping PkgTemplates in the app. Since plugins like Tests in PkgTemplates calls Pkg.update, and Pkg.update will use stdlib to resolve and update the Manifest.toml. But since these stdlib åctually locates in a different directory, Pkg.update gets nothing then it errors.

[Roger-luo]

Perhaps Pkg should read and store the stdlib projects during compilation time of it.

what does compilation mean here? the PackageCompiler compilation stage? e.g let PackageCompiler modify the stdlib path?

[Roger-luo]

Ah I think I get what @KristofferC mean now, do you mean Pkg should just read the Project.toml of stdlibs during compilation of julia's system image and store that as a constant instead of read it in runtime? Would this affect the ability of developing stdlibs without building julia etc. tho?

---

but I feel it's still necessary/beneficial to read stdlib etc. in runtime. I'm building a CLI binary for Pkg, and now since we have julia_version kwargs I can resolve package version directly from a Pkg shipped with julia 1.6, but I will need to set the stdlib path to user's stdlib path to resolve stdlibs. I don't think doing during compilation would solve this problem. Perhaps, Pkg should just provide this API, and let upstream tools to decide whether they want to fix the stdlib PackageSpec to a constant or not?

I just realize since there is julia_version already, now it makes sense to compile stdlib's deps and UUIDs into julia binary as constant, and filter them based on julia_version, which is probably simpler?

[KristofferC]

Ah I think I get what @KristofferC mean now, do you mean Pkg should just read the Project.toml of stdlibs during compilation of julia's system image and store that as a constant instead of read it in runtime?

Yes, any reading of stdlib files during runtime seems vulnerable to the issue here when Pkg is run in a way where these files are not accessible.

Would this affect the ability of developing stdlibs without building julia etc. tho?

I dunno, probably not? You would have to describe the exact workflow you mean for developing stdlibs.

[Roger-luo]

OK, so I go through the implementation related to this in Pkg, and tried to store stdlib's Project in compilation, here is what I found:

I dunno, probably not? You would have to describe the exact workflow you mean for developing stdlibs.

I was not sure, but I find the answer should be no since all stdlibs can be developed using a different UUID so there won't be anything to do with stored stdlib info.

for stdlibs in most cases, there are no version specifications except for some fake JLL packages. However, if we load stdlib's Project object to store all the information of stdlibs, currently this won't work since the fake LibUnwind_jll package has an invalid Project.toml. But I guess that was because JLLWrappers is not a stdlib yet, and current implementation of Pkg only loads stdlib's Project.toml when it's being resolved, which won't error. I guess we need to find a way to workaround this, or somehow add the valid Project.toml to LibUnwind_jll.

[KristofferC]

currently this won't work since the fake LibUnwind_jll package has an invalid Project.toml.

That should just be fixed there then (JuliaLang/julia#38743).