permission funk w/ shiny-server docker reading files on mbon server

[bbest]

Was having a hell of a time getting the env shiny app to read the files on /mbon/data_big/satellite, eg /mbon/data_big/satellite/chlor_a/clim_27km

The code for the env app is at https://github.com/marinebon/sdg14-shiny/tree/master/env and on mbon server at /mbon/shiny/env.

There were all kinds of issues:

• the existing rstudio-shiny docker user shiny had uid 998, which was same as telegraf on host server mbon
• the rstudio-shiny docker user mhepner had a default gid of 528, which is the same gid as group users on mbon
• the rstudio-shiny docker user mbon has same uid 1000 as user ben on mbon

Below documents what I did since I'm now having a major issue with getting the shiny apps to even show, per #24.

Connect to rstudio-shiny docker instance on host mbon.marine.usf.edu

ssh ben@mbon.marine.usf
docker exec -it rstudio-shiny bash

Fix permissions on rstudio-shiny docker instance

On rstudio-shiny as root, so don't need to prefix commands with sudo....

# update groups for shiny
addgroup --gid 900 shiny
usermod -g shiny shiny

# setup default group to users for user shiny 
usermod -a -G users shiny

# update users group from mhepner to users
groupmod -g 529 mhepner
groupmod -g 528 users

# add ben, shiny users to group users
usermod -g users shiny
usermod -g users ben
usermod -g staff ben

# update shiny uid from 998 to 900 to not conflict w/ mbon user telegraf
usermod -u 900 shiny
groupmod -g 900 shiny

# change uid/gid on paths that should be owned by shiny
find / -user 998
function p {
  chown -R -h 900 $1
  chgrp -R -h 900 $1
}
p /home/shiny
p /mbon/shiny-log
p /var/log/shiny-server

Restart rstudio-shiny docker instance

On server mbon as user ben...

docker restart rstudio-shiny

Wish there was a better way to restart just shiny-server and not whole docker instance, but I can't make sense of how based on comments in Hint for making a docker image containing both rstudio-server and shiny-server? · Issue #200 · rocker-org/rocker which is apparently using the s6-svc for stop/start/restart control, but I can't fathom how to get that going.

Tried these configs without luck:

# s6-svc friendly?
wget https://raw.github.com/rstudio/shiny-server/master/config/upstart/shiny-server.conf -O /etc/init/shiny-server.conf

# revert to default
wget https://raw.githubusercontent.com/rstudio/shiny-server/master/config/default.config -O /etc/init/shiny-server.conf

Generic template for changing uid/gid and updating ownership

usermod -u <NEWUID> <LOGIN>
groupmod -g <NEWGID> <GROUP>
find / -user <OLDUID> -exec chown -h <NEWUID> {} \;
find / -group <OLDGID> -exec chgrp -h <NEWGID> {} \;
usermod -g <NEWGID> <LOGIN>

Other paths on rstudio-shiny related to old shiny uid/gid

On rstudio-shiny...

find / -user 998

/tmp/Rtmp*
/home/shiny
/mbon/shiny-log
/var/log/shiny-server
/var/lib/shiny-server/bookmarks/shiny
/srv/shiny-server/test_shiny.txt
/srv/shiny-server/test_users/test_shiny.txt
/proc/*

On mbon...

sudo find / -group 998:

/etc/telegraf/telegraf.conf
/etc/telegraf/telegraf.d
/proc/*
/mbon/data_big/satellite/chlor_a/clim_27km
/mbon/shiny/test_shiny.txt
/mbon/shiny/test_users/test_shiny.txt
/mbon/shiny-log/*
/var/log/telegraf
/var/lib/docker/aufs/mnt/*

/etc/passwd and /etc/group entries

rstudio-shiny /etc/passwd:

shiny:x:998:998::/home/shiny:/bin/sh
rstudio-server:x:999:999::/home/rstudio-server:/bin/sh

rstudio-shiny /etc/group:

shiny:x:998:
rstudio-server:x:999:
sudo:x:27:mbon
staff:x:50:mbon
mbon:x:1000:

mbon /etc/passwd:

telegraf:x:999:998::/etc/telegraf:/bin/false

mbon /etc/group:

telegraf:x:998:
docker:x:999:ben

Shiny configuration

• shiny-server/#default-configuration

On rstudio-shiny:

# shiny configuration
view /etc/shiny-server/shiny-server.conf

Shiny logs

On rstudio-shiny cd into log directory and list all in reverse time with timestamp:

cd /var/log/shiny-server
ls -latr

Same paths mapped to mbon and easier to access:

cd /mbon/shiny-logs
ls -latr

Related issues

• can't log in to rstudio · Issue #23 · marinebon/sdg14
• add users to mbon server · Issue #12 · marinebon/sdg14

[bbest]

Hi Dan and Tylar,

Sorry about the hassles with permissions. I've struggled with permissions managed separately across server filesystems corals, mbon and docker images (rstudio-shiny, www). I changed those permissions per permission funk w/ shiny-server docker reading files on mbon server · Issue #25 · marinebon/sdg14:

Was having a hell of a time getting the env shiny app to read the files on /mbon/data_big/satellite, eg /mbon/data_big/satellite/chlor_a/clim_27km ...

Really appreciate your help with this Tylar. Another approach is to give Dan sudo access for changing as needed, hopefully while still giving access to shiny app.

Documenting further in permission funk w/ shiny-server docker reading files on mbon server · Issue #25 · marinebon/sdg14. I get a 404 error at https://github.com/USF-IMARS/issues/issues/17, but maybe I don't have permissions to the USF-IMARS/issues repo.

Thanks, Ben

Ben Best, PhD
Environmental Data Scientist
EcoQuants.com
calendar: LA (PDT), NY (EDT)
+1.805.705.9770



On Mon, Sep 18, 2017 at 7:12 AM, Tylar Murray <tylarmurray@mail.usf.edu> wrote:
Hmm. Things look normal at that location, and I have had success with sshfs in the past. Glad to hear you've found a workaround; I've added this low-priority issue to the bottom of the queue at https://github.com/USF-IMARS/issues/issues/17 

tylarmurray@mbon:~$ groups dotis
dotis : dotis users

tylarmurray@mbon:~$ ls -lh /mbon/data_big/satellite/
total 12K
drwxrwxrw- 2 ben   users   64 Sep  5 10:53 adg_443_giop
drwxrwxrw- 2 ben   users   64 Sep  5 10:59 bbp_s_giop
drwxrwxrw- 2 ben   users 4.0K Sep  5 11:00 chlor_a
drwxrwxrw- 2 ben   users   64 Apr 11 19:40 ghrsst
drwxrwxrw- 2 ben   users   64 Sep  5 10:59 nflh
drwxrwxrw- 2 dotis dotis   64 Jul 18 17:39 nFLH_CHL_ratio
drwxrwxrw- 2 ben   users   64 Sep  5 10:57 par
drwxrwxrw- 2 ben   users   64 May 19 12:09 seascapes
drwxrwxrw- 2 ben   users   64 Jun  6 13:25 sst
drwxrwxrw- 2 ben   users 4.0K Sep  1 17:14 sst4


On Mon, Sep 18, 2017 at 1:55 PM, Daniel Otis <dotis@mail.usf.edu> wrote:
Tylar,

When I connect, I use the following:

ssh -Y -l dotis mbon.marine.usf.edu

Or, for the OSXFUSE (from Terminal on the Mac):
IMaRS_iMAC$ sshfs dotis@mbon.marine.usf.edu:/mbon/data_big/satellite ~/server/remote2 (basically, this mounts the server disk onto my machine - it's quite handy)

Using the OSXFUSE, I can get to mbon/data_big/satellite/, but I can't get into any further sub-directories (chlor_a or sst4 for example).



I have worked around this issue for now, so this is not a high priority. Thanks,



Dan


On Sat, Sep 16, 2017 at 9:52 PM, Tylar Murray <tylarmurray@mail.usf.edu> wrote:
Things do look a bit different than I remember but I'm not quite sure what has changed. Can you give me a specific file you're not able to access? Also: are you connecting to mbon.marine.usf.edu:/mbon or seashell.marine.usf.edu:/mbon ?

On Fri, Sep 15, 2017 at 2:43 PM, Daniel Otis <dotis@mail.usf.edu> wrote:
Ben, Tylar

I am encountering permission issues accessing the mbon server. Has anything changed? I don't have problems if I login to the server directly, but I also use a program called OSXFUSE that allows me to tunnel in from my Mac desktop, which is currently experiencing permission issues. Any ideas are appreciated. 

Dan