realpath errors for shared network drives on Julia 1.3 on windows

[corakingdon]

I get the following error:

julia> realpath("C:/Users")
"C:\\Users"
julia> realpath("M:/")
ERROR: IOError: realpath: permission denied (EACCES)
Stacktrace:
 [1] uv_error at .\libuv.jl:97 [inlined]
 [2] realpath(::String) at .\path.jl:374
 [3] top-level scope at REPL[8]:1

This seems like this could be related to a previous issue on 1.2: #33127

This results in an error when trying to activate an environment on a network drive:

pkg> activate "M:/test/"
ERROR: IOError: realpath: permission denied (EACCES)
Stacktrace:
 [1] uv_error at .\libuv.jl:97 [inlined]
 [2] realpath(::String) at .\path.jl:374
 [3] safe_realpath(::String) at D:\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.3\Pkg\src\Types.jl:505
 [4] find_project_file(::Nothing) at D:\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.3\Pkg\src\Types.jl:248
 [5] Pkg.Types.EnvCache(::Nothing) at D:\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.3\Pkg\src\Types.jl:336 (repeats 2 times)
 [6] _activate_dep(::String) at D:\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.3\Pkg\src\API.jl:795
 [7] #activate#122(::Bool, ::typeof(Pkg.API.activate), ::String) at D:\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.3\Pkg\src\API.jl:817
 [8] activate at D:\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.3\Pkg\src\API.jl:809 [inlined]
 [9] do_activate!(::Array{String,1}, ::Dict{Symbol,Any}) at D:\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.3\Pkg\src\REPLMode.jl:454
 [10] #invokelatest#1(::Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}}, ::typeof(Base.invokelatest), ::Any, ::Any, ::Vararg{Any,N} where N) at .\essentials.jl:709
 [11] invokelatest(::Any, ::Any, ::Vararg{Any,N} where N) at .\essentials.jl:708
 [12] do_cmd!(::Pkg.REPLMode.Command, ::REPL.LineEditREPL) at D:\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.3\Pkg\src\REPLMode.jl:414
 [13] #do_cmd#23(::Bool, ::typeof(Pkg.REPLMode.do_cmd), ::REPL.LineEditREPL, ::String) at D:\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.3\Pkg\src\REPLMode.jl:391
 [14] do_cmd at D:\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.3\Pkg\src\REPLMode.jl:387 [inlined]
 [15] (::Pkg.REPLMode.var"#28#31"{REPL.LineEditREPL,REPL.LineEdit.Prompt})(::REPL.LineEdit.MIState, ::Base.GenericIOBuffer{Array{UInt8,1}}, ::Bool) at D:\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.3\Pkg\src\REPLMode.jl:619
 [16] #invokelatest#1 at .\essentials.jl:709 [inlined]
 [17] invokelatest at .\essentials.jl:708 [inlined]
 [18] run_interface(::REPL.Terminals.TextTerminal, ::REPL.LineEdit.ModalInterface, ::REPL.LineEdit.MIState) at D:\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.3\REPL\src\LineEdit.jl:2306
 [19] run_frontend(::REPL.LineEditREPL, ::REPL.REPLBackendRef) at D:\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.3\REPL\src\REPL.jl:1045
 [20] run_repl(::REPL.AbstractREPL, ::Any) at D:\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.3\REPL\src\REPL.jl:201
 [21] (::Base.var"#770#772"{Bool,Bool,Bool,Bool})(::Module) at .\client.jl:382
 [22] #invokelatest#1 at .\essentials.jl:709 [inlined]
 [23] invokelatest at .\essentials.jl:708 [inlined]
 [24] run_main_repl(::Bool, ::Bool, ::Bool, ::Bool, ::Bool) at .\client.jl:366
 [25] exec_options(::Base.JLOptions) at .\client.jl:304
 [26] _start() at .\client.jl:460

[musm]

Sounds like a permission error. Do you get the same error if you run julia as admin? Do you have access to M is the real question?

[davidanthoff]

Does gci M: in PowerShell work?

[corakingdon]

Yes gci M: works. I have read/write access to M

[corakingdon]

Hmm when I run Julia as an administrator, I get

julia> realpath("M:/")
ERROR: IOError: realpath: no such file or directory (ENOENT)
Stacktrace:
 [1] uv_error at .\libuv.jl:97 [inlined]
 [2] realpath(::String) at .\path.jl:374
 [3] top-level scope at REPL[5]:1

so I don't have access to M: as an administrator.

But in a regular Julia REPL, I can do cd("M:/") without error.

[musm]

We're using libuv for realpath so this is probably a libuv issue.

Here are the warnings:
http://docs.libuv.org/en/v1.x/fs.html#c.uv_fs_realpath

[musm]

For myself, I haven't encountered any issues with realpath on network drives or external drives.

[davidanthoff]

@ckingdon95 and I just looked into this a bit more, and we think we know what is going on: her drive M: is mapped to something like \\SERVERNAME\users\herusername. She has full read/write access to that path, but not to \\SERVERNAME\users.

The documentation for uv_fs_realpath mentions that on Windows it uses GetFinalPathNameByHandleA . The documentation for that mentions that for SMB file shares it will split a path into its components, query each component individually and that one needs permissions for each component. So I bet the problem here is that she doesn't have permissions to \\SERVERNAME\users, and therefor the call to GetFinalPathNameByHandleA fails.

I do think that the setup at her workplace is a common one, though, and that it would be good if one could activate an environment on such a network location with this permission structure. So maybe this is actually more a Pkg.jl issue, that the environment activation path should somehow avoid calling realpath if the path is on a network drive, or in some other way work around this situation.

[davidanthoff]

@StefanKarpinski and @KristofferC, do you think there is a way that Pkg.jl could work around these problems with realpath? I think it would be good if users could active projects on network shares that have this particular permission structure, it seems a very common one.

[StefanKarpinski]

Sure, although someone who has this setup should implement and test it. Perhaps a way to implement it is to have a safer_realpath function used in Pkg that wraps the realpath call in a try/catch and just returns the input path if realpath fails? I'm not sure what the call to realpath in Pkg is even for. Why do we care about the realpath?

[StefanKarpinski]

There's even already a safe_realpath function but it works around a different issue: calling realpath on a path that doesn't exist. So adding the functionality there and then using it wherever this is failing might do the trick.

[davidanthoff]

That sounds like a plan. @ckingdon95 do you want to take a crack at this? I think https://github.com/JuliaLang/Pkg.jl has the instructions on how to develop Pkg.jl, and maybe it would be enough to modify https://github.com/JuliaLang/Pkg.jl/blob/8322d21c11ddf721cc9d68580cef5e37b7cfdea6/src/utils.jl#L16 accordingly?

[StefanKarpinski]

That and you'll probably want to audit other calls to realpath in Pkg and see if any of them should be calls to safe_realpath instead. Like in particular, if they are causing something to fail on your network file system setup.

[corakingdon]

Thank you both! I will try to make those changes

[corakingdon]

I opened the draft pull request (linked above). I just have it return path if realpath(path) errors. Not sure if that's exactly what we want or not!

Regarding @StefanKarpinski's last comment, it looks like all calls to realpath in Pkg.jl (except for some tests) are called through safe_realpath.

[corakingdon]

I'm wondering if we should try to answer the original question about why we need realpath in the first place though.

[corakingdon]

Changing the safe_realpath function in Pkg allowed me to activate my environment on the shared network drive, but then when trying to actually use a package in the environment, realpath gets called with the project file in julia base here, which causes the same permission error I was getting before:
https://github.com/JuliaLang/julia/blob/master/base/loading.jl#L120

So I think this means we would also need to make changes there in order to fully resolve this problem?

[corakingdon]

@StefanKarpinski I was having trouble building julia from source (I'm on Windows and @davidanthoff said he's also had a hard time doing it on windows). Would it be possible for you (or someone else) to compile the modified version and then send it to me to test it out with my setup? I'd be happy to make the changes, as you've noted they are small, I just won't be able to test it out on my own.

[fredrikekre]

If you open a PR with the change you can download the built binary after CI has built it.

[davidanthoff]

If you open a PR with the change you can download the built binary after CI has built it.

That is super convenient, didn't know about that! That might be good information to put into some sort of README (if it is not already there)?

[StefanKarpinski]

I didn't know that either!

[corakingdon]

Thanks @fredrikekre! where on the PR can I find the option to download?