JULIA_LOAD_PATH replace load path instead of append to it

[zhouyan]

According to the documents, as well as the behavior of previous versions, JULIA_LOAD_PATH shall append to LOAD_PATH instead of replace it, however, with Julia 1.0.1, this is not the case. Below is a minimal example, (julia-1.0 is just an alias to my Juila 1.0.1 installation, I have multiple versions on the system).

$ julia-1.0 -e '@show LOAD_PATH'                                                                                                                                                                                                                                                           
LOAD_PATH = ["@", "@v#.#", "@stdlib"]

$ JULIA_LOAD_PATH=/usr/local/julia julia-1.0 -e '@show LOAD_PATH'
LOAD_PATH = ["/usr/local/julia"]

[fredrikekre]

End with a : (; on Windows) to append, so

JULIA_LOAD_PATH=/usr/local/julia:

Is there an error in the documentation?

[zhouyan]

I see, thanks. The doc did not mention this specifically.

[fredrikekre]

Seems to be documented: https://docs.julialang.org/en/v1/manual/environment-variables/#JULIA_LOAD_PATH-1

[zhouyan]

Yeah, the document only mentions the separator. I am not sure that the behavior that depends on the presence of a trailing separator is well understood. For example, in Unix usually programs does not change behavior when PATH has a trailing separator or not.

…

On Oct 4, 2018, at 17:14, Fredrik Ekre ***@***.***> wrote: Seems to be documented: https://docs.julialang.org/en/v1/manual/environment-variables/#JULIA_LOAD_PATH-1 — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub, or mute the thread.

[Liso77]

it is not that about append - it is about replacing first "empty" field in JULIA_LOAD_PATH . Try ":path" or "path1::path2" or "path:" Or look at https://github.com/JuliaLang/julia/blame/453a7ddcb6894cfcd8fc016634857b39416585cc/base/initdefs.jl#L113 (BTW env′ is "cool" variable name with '′': Unicode U+2032 (category Po: Punctuation, other) :/ )

[mfrigerio17]

I agree with @zhouyan ; the behaviour is NOT documented, and, in my opinion, it should be considered a small defect. Unix users used to env variables would not put a separator when there is only one element in fhe list.

[fredrikekre]

NOT documented

https://docs.julialang.org/en/v1/manual/environment-variables/#JULIA_LOAD_PATH-1

Unix users used to env variables would not put a separator when there is only one element in fhe list.

Well, if you only want one element in LOAD_PATH you don't need to have a separator as shown in #29513 (comment). The issue here seems to be missing docs that an empty path is replaced with the default load path.

[zhouyan]

Even if it is properly documented it is still a very unintuitive behavior IMHO. Unix users expect the PATH variable always be a list and an unset variable has no value at all. The behavior here is Unset JULIA_LAOD_PATH actually has an implicit default value, A set variable without colon replace that value, A set variable with a trailing common is the same as JULIA_LOAD_PATH=path:$JULIA_LOAD_PATH with the second taking that implicit value. I am sure this is not what really happens in the code, but it is how it behaves from outside. The env variable sometime replace the default some time prepend the default, depend on its value Technically it’s fine either way. But there’s wisdom in following convention. It’s it a variable to be set in a shell environment, most likely bashrc or zshrc or whatever rc file people use, so it shall behave like others and follow the convention of shell variables and how programs interpret environment variables Last but not least, I rarely use Windows, but if I recall correctly it is not so different from Unix on this regard.

…

On Jan 29, 2019, at 22:26, Fredrik Ekre ***@***.***> wrote: NOT documented https://docs.julialang.org/en/v1/manual/environment-variables/#JULIA_LOAD_PATH-1 Unix users used to env variables would not put a separator when there is only one element in fhe list. Well, if you only want one element in LOAD_PATH you don't need to have a separator as shown in #29513 (comment). The issue here seems to be missing docs that an empty path is replaced with the default load path. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

[mfrigerio17]

NOT documented

https://docs.julialang.org/en/v1/manual/environment-variables/#JULIA_LOAD_PATH-1

I know the link and the documentation. I meant that it only says that there are path separators, but it does not describe the requirement of having at least a separator for the appending to happen.

Well, if you only want one element in LOAD_PATH you don't need to have a separator

I have a single custom path element, that I want to append to the default LOAD_PATH. Without separator, my path replaces the content of LOAD_PATH, thus, for example, the standard library is no longer accessible (e.g. import LinearAlgebra fails)

The behaviour I observed:

JULIA_LOAD_PATH unset in the env ---> LOAD_PATH=<defaults> ("@" "@v#.#" "@stdlib")

JULIA_LOAD_PATH=<path> ---> LOAD_PATH=<path> (std lib lost)

JULIA_LOAD_PATH=:<path> ---> LOAD_PATH=<defaults> <path>

JULIA_LOAD_PATH=<path>: ---> LOAD_PATH=<path> <defaults>

[StefanKarpinski]

I agree that this should be better document. I'll make a PR to do so. The behavior is that an empty entry after splitting on the separator (: on UNIX, ; on Windows) is expanded to the default value of LOAD_PATH. If you set JULIA_LOAD_PATH to the empty string then the LOAD_PATH is empty. This allows you to do all of these things fairly easily:

• Replace entirely: export JULIA_LOAD_PATH="path1:path2"
• Prefix default with value: export JULIA_LOAD_PATH="path:"
• Postfix default with value: export JULIA_LOAD_PATH=":path"
• Pre and post fix default value: export JULIA_LOAD_PATH="path1::path2"
• Prepend before current value: export JULIA_LOAD_PATH="path:$JULIA_LOAD_PATH"
• Append after current value: export JULIA_LOAD_PATH="$JULIA_LOAD_PATH:path"
• Prepend & append current value: export JULIA_LOAD_PATH="path1:$JULIA_LOAD_PATH:path2"

You can even do subtle mixes of these easily, e.g.:

export JULIA_LOAD_PATH="path1:$JULIA_LOAD_PATH:path2::path3"

This will put path1 at the front before the current value of JULIA_LOAD_PATH, followed by path2, followed by the default LOAD_PATH, followed by path3. Any duplicates are removed and the fist instance of each value in the load path is kept.

[zhouyan]

I think the problem here is that this behavior itself is confusing, documents or not. It’s just not how people expect a path variable to work. Take the

JULIA_LOAD_PATH="path1:$JULIA_LOAD_PATH:path2"

Example, if it is PATH then it is equivalent to just path1:path2, without some hidden implicit value inserted in between. It is a shell variable, it shall behave like one. Of course Julia does not have to follow others. But I think breaking convention, confusing everyone apart from the guys invented this idea for the sake of some niche use such as prepend and append at the same time is not worth it. We can have it behaves as always prepend or append, which should satisfy 90% or more use cases. For the remaining 10% one can just manipulate the load path in startup.jl with the full power of Julia to do so. Better documentation is of course welcome, but not having things work in the unexpected way to begin with if possible is better.

…

Sent from my iPad

On Jan 30, 2019, at 00:07, Stefan Karpinski ***@***.***> wrote: I agree that this should be better document. I'll make a PR to do so. The behavior is that an empty entry after splitting on the separator (: on UNIX, ; on Windows) is expanded to the default value of LOAD_PATH. If you set JULIA_LOAD_PATH to the empty string then the LOAD_PATH is empty. This allows you to do all of these things fairly easily: Replace entirely: export JULIA_LOAD_PATH="path1:path2" Prefix default with value: export JULIA_LOAD_PATH="path:" Postfix default with value: export JULIA_LOAD_PATH=":path" Pre and post fix default value: export JULIA_LOAD_PATH="path1::path2" Prepend before current value: export JULIA_LOAD_PATH="path:$JULIA_LOAD_PATH" Append after current value: export JULIA_LOAD_PATH="$JULIA_LOAD_PATH:path" Prepend & append current value: export JULIA_LOAD_PATH="path1:$JULIA_LOAD_PATH:path2" You can even do subtle mixes of these easily. E.g. export JULIA_LOAD_PATH="path1:$JULIA_LOAD_PATH:path2::path3" This will put path1 at the front before the current value of JULIA_LOAD_PATH, followed by path2, followed by the default LOAD_PATH, followed by path3. Any duplicates are removed and the fist instance of each value in the load path is kept. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

[StefanKarpinski]

What do you think empty entries in PATH do?

[zhouyan]

It doesn’t do anything

[zhouyan]

Why does it have to do something?

[Liso77]

Empty entry usually represents current workdir. Which is useful! It is impossible to have empty entry without default which is IMHO issue. Instead of first empty I propose to replace something like @default in parse_load_path.

[zhouyan]

Empty entry usually represents current workdir. Which is useful! It is impossible to have empty entry without default which is IMHO issue. Instead of first empty I propose to replace something like @default in parse_load_path.

I think we are talking about two slightly different things. I agree that internally Julia should have a default LOAD_PATH even if the user does not do anything.

My point is about the behavior of manipulating that LOAD_PATH through a shell environment variable like JULIA_LOAD_PATH, which shall have consistent behavior, such as say always append, prepend, or replace.

One can argument that the current behavior is consistent, but I think it is just quite unconventional.

For example, if you set any one of the following,

unset PATH # no search path at all
export PATH=/foo
export PATH=:/foo
export PATH=:/foo:
export PATH=/foo:$PATH:/foo

they will leads to the same results, that search path being replaced by /foo.

But the same cannot be said fro JULIA_LOAD_PATH.

unset JULIA_LOAD_PATH # become default path
export JULIA_LOAD_PATH=/foo # /foo only
export JULIA_LOAD_PATH=:/foo # default_path then /foo
export JULIA_LOAD_PATH=:/foo: # /foo then default path
export PJULIA_LOAD_ATH=/foo:$JULIA_PATH:/foo # /foo default path, and /foo

Either has the behavior being JULIA_LOAD_PATH always replace the default LOAD_PATH or always append to it, or always prepend to it. An empty element inside the shell variable shall not have any special meaning.

[Liso77]

Try this:

echo 'echo A' > tst;chmod u+x tst
export PATH=/foo
tst
export PATH=/foo:
tst

EDIT:
Normally you have PATH without empty entry which prevent to hide system commands (instead of tst you could have for example ls in current directory). It is practical and more secure.

[StefanKarpinski]

@zhouyan: It doesn’t do anything

No—we should probably all be clear on what the shell does before insisting that Julia should copy it.

@Liso77: Empty entry usually represents current workdir. Which is useful!
It is impossible to have empty entry without default which is IMHO issue.

It is useful, but the same effect can be accomplished by explicitly using . in the PATH of JULIA_LOAD_PATH instead. Which is only one character and much more obvious—and far less dangerous for reasons that I'm about to explain...

Here are a few observations about the "empty entry means current directory" behavior of PATH. Putting the current directory in the PATH is a security hole. Similarly, putting the current directory in your Julia load path is also a security hole, which is why it's not in there by default. It's also really easy when doing shell programming to accidentally end up with an empty entry in a :-separated shell variable. For example, let's say you write some shell code that does this:

export PATH="foo:$PATH"

Seems fine, right? But what if PATH is unset? Then you end up with foo: in the PATH and you've accidentally opened up a security hole. Oops. Fortunately, the shell sets PATH for you so most of the time this trap doesn't get sprung. However that is very much not the case for JULIA_LOAD_PATH—it is usually not set. But sometimes it is. Let's suppose that I have this in my .bashrc:

# ~/.bashrc

export JULIA_LOAD_PATH=bar:baz

Now, what if I want to write some shell code that prepends a value to it, assuming it's already set to something like this? Then I might write this:

# some shell script that calls Julia code

export JULIA_LOAD_PATH="foo:$JULIA_LOAD_PATH"

Great. Assuming it has the value from the bashrc file, we end up with foo:bar:baz. But now what if I change my startup scripts and decide that I didn't actually need to override the default Julia load path after all and I delete the line that sets it to bar:baz. Now what effect does my shell script have? We end up with a JULIA_LOAD_PATH value of foo:. What does that do? If we used the "empty entry means current directory" convention then we would have have now opened up a big ol' security hole. What does it mean currently? It means exactly what we wanted it to mean: prepend foo to whatever value the load path has. So the current load path behavior not only avoided a security hole but also let us write naive code that continues to work as intended whether JULIA_LOAD_PATH is set or not.

Let's suppose we didn't do it this way and just ignored empty entries in JULIA_LOAD_PATH. Ok, so at least we avoid the security hole. But now let's suppose we want to generalize our shell script above so that it correctly prepends foo to the Julia load path regardless of whether JULIA_LOAD_PATH is set or not. Let's further suppose that we used the @default idea that @Liso77 proposed. In order to do this correctly, we would have to write this:

if [[ -z "$JULIA_LOAD_PATH" ]]
then
    export JULIA_LOAD_PATH="foo:@default"
else
    export JULIA_LOAD_PATH="foo:$JULIA_LOAD_PATH"
fi

That doesn't roll off the fingers so easily, does it? Worse though is that everyone will forget to do this correctly and only handle the case that they're currently in with respect to JULIA_LOAD_PATH being set. If JULIA_LOAD_PATH is not set, they'll write export JULIA_LOAD_PATH="foo:@default" unconditionally and their script will do the wrong thing when JULIA_LOAD_PATH is already set. If JULIA_LOAD_PATH is set, they'll export JULIA_LOAD_PATH="foo:$JULIA_LOAD_PATH" and their script will do the wrong thing when JULIA_LOAD_PATH isn't set. In the current scheme you can always just write export JULIA_LOAD_PATH="foo:$JULIA_LOAD_PATH" to prepend foo and it works in both cases, whetherJULIA_LOAD_PATH is set or not.

Moreover, the same exact mechanism works for any manipulation of the load path, not just for prepending. If you want to add stuff at the front, the back, or both ends of the load path you can do it just as easily and correctly handle the cases where JULIA_LOAD_PATH is set or not set.

The shell can't change the unfortunate and dangerous "empty entry means current directory" behavior because it is ancient and needs to remain backwards compatible instead of breaking everyone's shell scripts, but that does not mean that we should monkey-see-monkey-do copy bad choices that the shell made 50 years ago and is stuck with for legacy reasons.

[StefanKarpinski]

For example, if you set any one of the following,

unset PATH # no search path at all
export PATH=/foo
export PATH=:/foo
export PATH=:/foo:
export PATH=/foo:$PATH:/foo

they will leads to the same results, that search path being replaced by /foo.

Having lots of extraneously different ways to express the same thing is a hallmark of a bad design.

[Liso77]

But it leads to different results!!! (see simple counterexample I wrote above)

[StefanKarpinski]

But it leads to different results!!! (see simple counterexample I wrote above)

I have no idea what you're replying to or which of the many examples in this thread you mean.

[mgkuhn]

@Liso77 and @zhouyan: The behavior that an empty entry after splitting on the separator (: on UNIX, ; on Windows) is expanded to the default value is already well established in other classic tools, most notably TeX's TEXINPUT variable. As Stefan has pointed out above already, this well-established TEXINPUT-style behavior is vastly more useful than bash's interpretation of an empty entry in PATH as the current working directory, because the dot is already available to denote that.

Also note that the IEEE POSIX standard agrees: “A zero-length prefix is a legacy feature that indicates the current working directory. It appears as two adjacent characters ( "::" ), as an initial preceding the rest of the list, or as a trailing following the rest of the list. A strictly conforming application shall use an actual pathname (such as .) to represent the current working directory in PATH.”

I don't think Julia should copy from POSIX sh a “legacy feature” that the POSIX standard discourages being used, when TeX has long ago established a much more useful convention.

[zhouyan]

I agree that we shouldn’t copy the legacy behavior of interpreting it as the current directory. I also believe the current behavior is technically sound once it is understood, (that’s why I never brought it up again when the issue is closed until some one else also had the same confusion months later) My argument is more of should we be interpreting the empty path in the first place? My opinion is that it is is not very intuitive. The user get something (the default path) without explicitly specifying it and may remove it without explicitly specifying it either. I would think that a more explicit approach would be better: the default path is either always there or never there (I guess most people would prefer the former). In summary “being explicit” is my main concern. But I have to admit there’s no clear line between being explicit and being too verbose. If we want to keep the current behavior it need much better documents. I am aware the TEXINPUT behavior but I really don’t think it is well known or understood precisely by the majority. Most people only know more or less how PATH and alike work and they just works. I think this is where the initial confusion come from, at least for me. I am by no means the most knowledgeable person here but I am not entirely inexperienced in Unix environment either. If it confuses me I believe there are others as well.

…

On Feb 9, 2019, at 20:08, Markus Kuhn ***@***.***> wrote: @Liso77 and @zhouyan: The behavior that an empty entry after splitting on the separator (: on UNIX, ; on Windows) is expanded to the default value is already well established in other classic tools, most notably [TeX's TEXINPUT variable](TeX's TEXINPUT variable). As Stefan has pointed out above already, this well-established TEXINPUT-style behavior is vastly more useful than bash's interpretation of an empty entry in PATH as the current working directory, because the dot is already available to denote that. Also note that the IEEE POSIX standard agrees: “A zero-length prefix is a legacy feature that indicates the current working directory. It appears as two adjacent characters ( "::" ), as an initial preceding the rest of the list, or as a trailing following the rest of the list. A strictly conforming application shall use an actual pathname (such as .) to represent the current working directory in PATH.” I don't think Julia should copy from POSIX sh a “legacy feature” that the POSIX standard discourages being used, when TeX has long ago established a much more useful convention. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

[mgkuhn]

I think the confusion is best fixed (for Linux/Unix users at least) by adding a complete list of environment variable names and the specification of their value syntax to the man julia man page. That reference is where Unix users normally expect to look first to understand the semantics of environment variables, command-line options, configuration files and the return value of any Unix command-line tool. The current julia man page has already a list of command-line options, but it still lacks a list of environment variables, files and the return value, and that information is currently either missing or spread elsewhere across various sections of the Julia documentation at the moment, so fewer users are readily aware of it.

[StefanKarpinski]

We already document environment variables. I have now updated the docs for the JULIA_LOAD_PATH variable to thoroughly document how this works and why: #31010.

[Liso77]

It is good that problem with documentation is fixed! ✌️

Using . as current dir "solve" issue I was talking above. 🙇‍♂️

Last problem is with clarity, simplicity and intuitivness. I think that if we have @ with special meaning it is unnecessary to complicate this micro-language with another default meaning for empty string. But I understand that for backward compatibility this (IMHO) glitch remain unfixed. ¯_(ツ)_/¯

[jay0805]

my outputs are:
julia> LOAD_PATH
3-element Array{String,1}:
"@"
"@v#.#"
"@stdlib"

julia> Base.load_path()
2-element Array{String,1}:
"C:\Users\jay patel\.julia\environments\v1.5\Project.toml"
"C:\Users\jay patel\AppData\Local\Programs\Julia 1.5.3\share\julia\stdlib\v1.5"

and i cant run the command using pkg
julia> using pkg
ERROR: ArgumentError: Package pkg not found in current path:

• Run import Pkg; Pkg.add("pkg") to install the pkg package.

can please guide me

[StefanKarpinski]

Please post questions to the Julia discourse discussion forum.