Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Pkg directory shared among users has several permission problems #12876

Closed
diegozea opened this Issue Aug 30, 2015 · 23 comments

Comments

@diegozea
Copy link
Member

diegozea commented Aug 30, 2015

In my computer, using a package gives me this SystemError (denied permission).
I need to run sudo julia in order to use packages... Is there a way to solve this? Can *.ji be created with less restrictions?

julia> using Compat
ERROR: SystemError: opening file /home/dzea/.julia/lib/v0.4/Compat.ji: Permiso denegado
 in open at ./iostream.jl:90
 in open at iostream.jl:102
 in stale_cachefile at loading.jl:383
 in recompile_stale at loading.jl:418
 in _require_from_serialized at loading.jl:83
 in _require_from_serialized at ./loading.jl:109
 in require at ./loading.jl:186

julia> versioninfo()
Julia Version 0.4.0-dev+7044
Commit 706cad1 (2015-08-27 21:32 UTC)
Platform Info:
  System: Linux (x86_64-linux-gnu)
  CPU: Intel(R) Core(TM) i7-3770 CPU @ 3.40GHz
  WORD_SIZE: 64
  BLAS: libopenblas (NO_LAPACK NO_LAPACKE DYNAMIC_ARCH NO_AFFINITY Sandybridge)
  LAPACK: liblapack.so.3
  LIBM: libopenlibm
  LLVM: libLLVM-3.3
@nalimilan

This comment has been minimized.

Copy link
Contributor

nalimilan commented Aug 30, 2015

Maybe you installed the package while running julia under sudo? Can you post the output of ls -l /home/dzea/.julia/lib/v0.4/Compat.ji?

@diegozea

This comment has been minimized.

Copy link
Member Author

diegozea commented Aug 30, 2015

I can't remember if I installed the package with sudo. The output is:

-rw------- 1 root root 30383 ago 29 00:51 /home/dzea/.julia/lib/v0.4/Compat.ji

All my *.jis have the same mode: -rw------- 1 root root

@yuyichao

This comment has been minimized.

Copy link
Member

yuyichao commented Aug 30, 2015

They cannot possibly have root as their owner if it is not created or changed ownership by root. This is a very typical issue with running command under sudo while maintaining the value of HOME.

@jakebolewski

This comment has been minimized.

Copy link
Member

jakebolewski commented Aug 30, 2015

That looks to be your problem.

@timholy timholy reopened this Aug 30, 2015

@timholy timholy changed the title using can't open *.ji (denied permission) Pkg directory shared among users has several permission problems Aug 30, 2015

@nalimilan

This comment has been minimized.

Copy link
Contributor

nalimilan commented Aug 30, 2015

I case you don't know how to fix it, chown -R dzea: /home/dzea/.julia should do the trick. Let us know if it ever comes back.

@timholy

This comment has been minimized.

Copy link
Member

timholy commented Aug 30, 2015

I'm reopening this to point out that using a system-wide shared Pkg folder has several issues:

  • Even if /usr/local/julia/packages is created with g+w permissions, any new folders get created with default permissions. If user1 adds CoolPackage.jl, user2 can't fix bugs in it even though they are in the same group.
  • The *.ji files get written with permissions like these:
user2@diva:/tmp/packages/lib/v0.4$ ls -l
total 484
-rw------- 1 user2 dummy 376701 Aug 30 11:27 ColorTypes.ji
-rw------- 1 user2 dummy  30165 Aug 30 11:27 Compat.ji
-rw------- 1 user2 dummy  82414 Aug 30 11:27 FixedPointNumbers.ji

What that means is that if one user has used a package, it prevents any other user from accessing it. As user1:

julia> using ColorTypes
ERROR: SystemError: opening file /tmp/packages/lib/v0.4/ColorTypes.ji: Permission denied
 in open at ./iostream.jl:90
 in open at iostream.jl:102
 in stale_cachefile at loading.jl:379
 in recompile_stale at loading.jl:414
 in _require_from_serialized at loading.jl:83
 in _require_from_serialized at ./loading.jl:105
 in require at ./loading.jl:182

julia> 

In this case user1 can read the source files, but because the *.ji file is present julia tries (and fails) to load it.

@timholy

This comment has been minimized.

Copy link
Member

timholy commented Aug 30, 2015

For the benefit of others, here's an easy way to do this kind of testing:

addgroup dummy
adduser --ingroup dummy user1
adduser --ingroup dummy user2

and then start julia with JULIA_PKGDIR=/tmp/packages /path/to/julia.

@nalimilan

This comment has been minimized.

Copy link
Contributor

nalimilan commented Aug 30, 2015

It would probably make sense to respect the permissions of the parent directory (.julia or a subfolder thereof) when installing packages.

@timholy

This comment has been minimized.

Copy link
Member

timholy commented Aug 30, 2015

That's my sense too.

@kshyatt kshyatt added the packages label Aug 30, 2015

@stevengj

This comment has been minimized.

Copy link
Member

stevengj commented Aug 31, 2015

The .ji permissions are the result of #12699, because mkstemp+rename uses mode 0600. I guess it could run chmod afterwards.

@adowling2

This comment has been minimized.

Copy link

adowling2 commented Jul 22, 2016

We have setup a shared Julia package folder on a shared computer, and have observed a few (possibly related) issues:

  • When a package is recompiled, the .ji permissions are set such that other users of the machine cannot access them. We have a script the fixes this via chown and chgrp and runs every 5 minutes via cron as a work around.
  • After one user recompiles a package, any other users with get "Recompiling stale cache file" message. Were constantly recompiling packages on a busy multiuser machine.
  • The git repos for the packages inevitably become dirty, and each require a "stash" command before Pkg.update() will do anything.
@tkelman

This comment has been minimized.

Copy link
Contributor

tkelman commented Jul 22, 2016

If you're sharing package installs, you should also share julia itself, or set the precompile cache to be user-specific.

Packages becoming dirty is either the fault of your script or a bug in the package.

@adowling2

This comment has been minimized.

Copy link

adowling2 commented Jul 22, 2016

We're sharing the Julia executable too. I'll look into using user-specific percompile caches.

@tkelman

This comment has been minimized.

Copy link
Contributor

tkelman commented Jul 22, 2016

if the julia executable is the same, then you shouldn't be seeing recompilation unless people are editing the package sources

@adowling2

This comment has been minimized.

Copy link

adowling2 commented Jul 22, 2016

People are not editing packages, so this is strange.

Let's take the JuMP package as an example. When I run

$ cd /opt/julia_packages/.julia/v0.4/JuMP
$ git status

I see many files have been modified. On closer inspection

$ git diff ./test/REQUIRE
diff --git a/test/REQUIRE b/test/REQUIRE
old mode 100644
new mode 100755

I see that only the file permissions have changed. I think this is also causing the dirty git directories. What file permissions are appropriate for /.julia/v0.4 and /.julia/lib/v0.4? All of the authorized users are members of a common group. I'm currently changing all of the files in this folder to said group.

Ideally, I'd like all members of the authorized group to be able to

  1. Run shared packages
  2. Run Pkg.add()
  3. Run Pkg.update()

Thanks in advance.

@tkelman

This comment has been minimized.

Copy link
Contributor

tkelman commented Jul 23, 2016

I'm not sure if git tracks group ownership, but it does track permissions, and you shouldn't need most git-tracked files in packages to be executable.

@adowling2

This comment has been minimized.

Copy link

adowling2 commented Jul 25, 2016

Below is the most recent version of our maintenance script:

chgrp ourgroup -R /opt/julia_packages/.julia
### chmod 774 -R /opt/julia_packages/.julia

chgrp ourgroup -R /opt/julia_packages/.julia/.cache
### chmod 774 -R /opt/julia_packages/.julia/.cache

# Added:
chmod 770 -R /opt/julia_packages/.julia/lib/v0.4

Removing the commented out chmod statements fixed the dirty git repo and constant recompiling issues.

The added chmod statement is necessary, as Julia by default compiles the .ji files to only be accessible by the file owner. Is there a way to change this?

We still need to test if permissions issues persist when one user adds or updates packages.

@vtjnash

This comment has been minimized.

Copy link
Member

vtjnash commented Jul 25, 2016

The .ji files were mostly intended to be user-private, to avoid the inherent race condition with multiple users trying to update the same file.

@adowling2

This comment has been minimized.

Copy link

adowling2 commented Jul 25, 2016

Is there a recommended setup for a shared package directory? Should I specify LOAD_CACHE_PATH for each user to avoid sharing .ji files and associated permissions issues?

@stevengj

This comment has been minimized.

Copy link
Member

stevengj commented Jul 25, 2016

@adowling2, normally you would have a shared package directory and per-user .julia directories, and have only one person in charge of updating the shared package directory. When other users do Pkg.add etcetera it would only affect their personal .julia directories.

@adowling2

This comment has been minimized.

Copy link

adowling2 commented Jul 25, 2016

@stevengj Can you recommend instructions on configuring Julia to use both a shared and per-user .julia directory?

@stevengj

This comment has been minimized.

Copy link
Member

stevengj commented Jul 25, 2016

@adowling, I think /usr/local/share/julia/site/v0.4 is already in the default LOAD_PATH for julia installed in /usr/local, so why not just install Julia packages there?

Otherwise, install wherever you want and then have users add that directory to their LOAD_PATH in their .juliarc files.

If you want to install shared cache files, too, you'll need to have users add that directory to their Base.LOAD_CACHE_PATH.

@simonbyrne

This comment has been minimized.

Copy link
Contributor

simonbyrne commented Aug 10, 2018

This has changed considerably with the new Pkg. If still an issue, please open at https://github.com/JuliaLang/Pkg.jl

@simonbyrne simonbyrne closed this Aug 10, 2018

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.
You signed in with another tab or window. Reload to refresh your session. You signed out in another tab or window. Reload to refresh your session.