Unable to automate installation of bash_kernel (on top of jupyter/minimal-notebook)

[mjbright]

OK, so this is more of a question ... than an issue.

I wanted to add bash_kernel to jupyter/minimal-notebook.

Not quite sure about how start-notebook.sh is being used I decided to make a modified copy into which I added the following lines before the final exec line.

    CBIN=/opt/conda/bin                                                                                                                            

    conda install pip                                                                                                                              

    sudo -u $NB_USER $CBIN/pip install --user bash_kernel                                                                                          
    sudo -u $NB_USER $CBIN/python -m bash_kernel.install                                                                                           

However these commands don't seem to be executed when invoked by the CMD of the Dockerfile.
Connecting to the docker container and rerunning /start-notebook.sh installs the kernel as expected.

Why is this manual step necessary, and how what should I do so that it's not necessary?

[parente]

I haven't tried it, but I think I would opt to put all additional install steps in a new Dockerfile that derives FROM the minimal notebook like so:

FROM jupyter/minimal-notebook:4.0

/opt/conda/bin/pip install --user bash_kernel
/opt/conda/bin/python -m bash_kernel.install

I would docker build -t jupyter/bash-notebook . and spawn a container from that image.

The start-notebook.sh script is really about late-creation of the jovyan user and housekeeping to ensure permissions are correct for that user in the home directory and the conda directory.

[mjbright]

Yes, that was the initial approach I'd taken until I realized that this
wasn't working as bash_kernel was installing only for the current, i.e.
root user, so I moved it after the jovyan user creation (so out of the
Dockerfile into start-notebook.sh).

Whilst bash_kernel could install system-wide, that would solve the problem-
I think would be interesting to be able to do derivations after the
creation of the jovyan user.

I guess there were good reasons to move the jovyan user creation out of the
Dockerfile ...

On 10 September 2015 at 13:55, Peter Parente notifications@github.com
wrote:

I haven't tried it, but I think I would opt to put all additional install
steps in a new Dockerfile that derives FROM the minimal notebook like so:

FROM jupyter/docker-stacks:4.0

/opt/conda/bin/pip install --user bash_kernel
/opt/conda/bin/python -m bash_kernel.install

I would docker build -t jupyter/bash-notebook . and spawn a container
from that image.

The start-notebook.sh script is really about late-creation of the jovyan
user and housekeeping to ensure permissions are correct for that user in
the home directory and the conda directory.

—
Reply to this email directly or view it on GitHub
#25 (comment)
.

[parente]

Whilst bash_kernel could install system-wide, that would solve the problem-

Ah, yes, removing --user would do the trick. I see it now.

I think would be interesting to be able to do derivations after the
creation of the jovyan user.

Do you have a use case in mind? I'm trying to think of one and have come up short (but it's still early in the day ;)

I guess there were good reasons to move the jovyan user creation out of the
Dockerfile ...

Yes. See issue #16. I've heard rumors that Docker 1.9 might finally have host/container user mapping. If so, and if it works on reasonable kernel versions, we might be able to remove the bandaid.

[parente]

Another possibility.

1. Undo the user creation in script and move it back to Dockerfile so that the username / home direct exist for commands in the Dockerfile to work as expected.
2. Change the start-notebook script so that instead of late-creating the user there, it instead changes in the UID/GID of the jovyan user and fixes permissions (e.g., https://muffinresearch.co.uk/linux-changing-uids-and-gids-for-user/)

Pros

• User exists at docker build time
• UID can be configured to fix host/container permission problems

Cons

• Have to scan the entire container to fix perms at start time OR limit to specific directories to scan by default (e.g., /home/jovyan, /opt/conda)
• Bandaid for docker shortcoming that may get resolved in a real-soon-now version

[mjbright]

I'm not aware of the uid/permission problem, but I do like the sound of
doing things cleanly in the Dockerfile, even if it requires bandaid to
workaround some hopefully temporary Docker issues.

Although my 'bash_kernel' issue is probably a feature of that kernel -
which could be fixed to install system wide, I think it would be better to
be able to install things as the target user where possible.

On 10 September 2015 at 14:38, Peter Parente notifications@github.com
wrote:

Another possibility.

1. Undo the user creation in script and move it back to Dockerfile so
   that the username / home direct exist for commands in the Dockerfile to
   work as expected.
2. Change the start-notebook script so that instead of late-creating
   the user there, it instead changes in the UID/GID of the jovyan user and
   fixes permissions (e.g.,
   https://muffinresearch.co.uk/linux-changing-uids-and-gids-for-user/)

Pros

• User exists at docker build time
• UID can be configured to fix host/container permission problems

Cons

• Have to scan the entire container to fix perms at start time OR
  limit to specific directories to scan by default (e.g., /home/jovyan,
  /opt/conda)
• Bandaid for docker shortcoming that may get resolved in a
  real-soon-now version

—
Reply to this email directly or view it on GitHub
#25 (comment)
.

[parente]

I can give the create jovyan early / fix uid later approach a shot to see if it's terribly slow or if there are other surprises.

/cc @costerwi @rgbkrk for other perspectives

[parente]

But for the moment, the workaround of doing /opt/conda/bin/pip install bash_kernel to install it global is not that terrible. The container is for a single user anyway, imho.

[mjbright]

Sure ... but

• that has to be done after creation of that user
• for some reason this wasn't getting "done" on the first invocation of
  start-notebook.sh (?)
  I had to manually run my modified start-notebook.sh once the container
  was up.

Which I can keep doing, it's just ugly and not-automated.

Thanks for the suggestions.

On 10 September 2015 at 14:52, Peter Parente notifications@github.com
wrote:

But for the moment, the workaround of doing /opt/conda/bin/pip install
bash_kernel to install it global is not that terrible. The container is
for a single user anyway, imho.

—
Reply to this email directly or view it on GitHub
#25 (comment)
.

[parente]

• that has to be done after creation of that user

Even without the --user flag? I thought that would work fine without the user in existence and install to /usr/local/ or some other global PREFIX.

[mjbright]

Oops, sorry I'll have to try that (sorry didn't know the --user flag and
assumed it did something quite different !)

Will try tonight.

On 10 September 2015 at 14:58, Peter Parente notifications@github.com
wrote:

• that has to be done after creation of that user

Even without the --user flag? I thought that would work fine without the
user in existence and install to /usr/local/ or some other global PREFIX.

—
Reply to this email directly or view it on GitHub
#25 (comment)
.

[parente]

I just tried it and the kernel spec didn't install anywhere wtihout --user. And now I see why: system wide install is not yet supported takluyver/bash_kernel#31.

jovyan@c0e462d94b33:/$ jupyter --paths                                                                                             
config:                                                                                                                            
    /home/jovyan/.jupyter                                                                                                          
    /opt/conda/etc/jupyter                                                                                                         
    /usr/local/etc/jupyter                                                                                                         
    /etc/jupyter                                                                                                                   
data:                                                                                                                              
    /home/jovyan/.local/share/jupyter                                                                                              
    /opt/conda/share/jupyter                                                                                                       
    /usr/local/share/jupyter                                                                                                       
    /usr/share/jupyter                                                                                                             
runtime:                                                                                                                           
    /home/jovyan/.local/share/jupyter/runtime

[costerwi]

I'm not sure why the modified start-notebook.sh didn't work. Can you check docker logs for that image?

As a different ugly workaround, a derived container may be able to install or copy misbehaving packages like this into /etc/skel and allow the existing start-notebook.sh to move them into jovyan's home.

[parente]

With PR #31, this Dockerfile now works:

FROM jupyter/minimal-notebook

USER jovyan

# Make sure not to create a cache dir else NB_UID switching
# will hit issues.
RUN /opt/conda/bin/pip install --no-cache-dir bash_kernel
RUN /opt/conda/bin/python -m bash_kernel.install

USER root

[parente]

After the PR #31 merge,

docker pull jupyter/minimal-notebook:latest
# in a folder containing the Dockerfile from https://github.com/jupyter/docker-stacks/issues/25#issuecomment-139444841
docker build --rm -t jupyter/bash-notebook .
docker run --rm -it --p 8888:8888 jupyter/bash-notebook

Also works fine if I change the UID of jovyan (-e NB_UID=1005).

[mjbright]

Thanks a lot Peter, I really appreciate the work you put in on this!

[mjbright]

... but ... any idea why this worked the first time (first build and run) but then I keep getting the following errors on subsequent runs?

(My host is a Fedora 22, I'm running docker 1.8.1 by the way)

2015-09-13 07:42:50,019 CRIT Supervisor running as root (no user in config file)
2015-09-13 07:42:50,019 WARN Included extra file "/etc/supervisor/conf.d/notebook.conf" during parsing
2015-09-13 07:42:50,034 INFO RPC interface 'supervisor' initialized
2015-09-13 07:42:50,034 CRIT Server 'unix_http_server' running without any HTTP authentication checking
2015-09-13 07:42:50,034 INFO supervisord started with pid 1
2015-09-13 07:42:51,039 INFO spawned: 'notebook' with pid 9
2015-09-13 07:42:51,050 INFO exited: notebook (exit status 127; not expected)
2015-09-13 07:42:52,053 INFO spawned: 'notebook' with pid 10
2015-09-13 07:42:52,061 INFO exited: notebook (exit status 127; not expected)
2015-09-13 07:42:54,066 INFO spawned: 'notebook' with pid 11
2015-09-13 07:42:54,077 INFO exited: notebook (exit status 127; not expected)
2015-09-13 07:42:57,082 INFO spawned: 'notebook' with pid 12
2015-09-13 07:42:57,092 INFO exited: notebook (exit status 127; not expected)
2015-09-13 07:42:58,094 INFO gave up: notebook entered FATAL state, too many start retries too quickly

[mjbright]

OK, found the problem.

Sunday driver - I was doing a bad mount on my docker run ....

hey, it's Sunday morning ;-)

[rgbkrk]

😄