SAS kernel failing inside Docker container with Python 3.9

[hongooi73]

(Crossposted from Stack Overflow, at user Joe's suggestion)

For some reason I'm unable to run SAS notebooks inside a Jupyter Docker container. I can run SAS code inside Python notebooks via saspy, but the SAS kernel keeps throwing errors on me.

I'm using the image jupyter/pyspark-notebook:notebook-6.4.2 as the base, and connecting to a remote server via IOM. If I run a Python notebook, eg with

import saspy
%reload_ext saspy.sas_magic
ss = saspy.SASsession(cfgname="xxx")

then the connection works fine. Sometimes it times out, but that's probably a server issue.

However if I run a SAS notebook, when I try to submit a code chunk I get an error result:

[<class 'TypeError'>, TypeError("Frame 0 ({'shell': [b'f8fd30a1-60bc-4f25-...) does not support the buffer interface."), <traceback object at 0x7fad01e41100>]

There are also a bunch of error messages in the terminal window running the container:

notebook_1  | [I 17:14:56.181 NotebookApp] Kernel started: 58182c39-9df4-4922-b068-1d6bde76c31a, name: sas
notebook_1  | [IPKernelApp] ERROR | Exception in message handler:
notebook_1  | Traceback (most recent call last):
notebook_1  |   File "/opt/conda/lib/python3.9/site-packages/ipykernel/kernelbase.py", line 352, in dispatch_shell
notebook_1  |     await result
notebook_1  |   File "/opt/conda/lib/python3.9/site-packages/ipykernel/kernelbase.py", line 642, in execute_request
notebook_1  |     reply_content = self.do_execute(
notebook_1  |   File "/opt/conda/lib/python3.9/site-packages/metakernel/_metakernel.py", line 397, in do_execute
notebook_1  |     retval = self.do_execute_direct(code)
notebook_1  |   File "/opt/conda/lib/python3.9/site-packages/sas_kernel/kernel.py", line 213, in do_execute_direct
notebook_1  |     res = self.mva.submit(code, prompt=self.promptDict)
notebook_1  | AttributeError: 'NoneType' object has no attribute 'submit'

The dockerfile is intended for use behind a corp firewall so contains some confidential entries, making it hard to give a reproducible example. But the gist of it is:

FROM jupyter/pyspark-notebook:notebook-6.4.2

RUN mamba install -c conda-forge -y nodejs && \
    conda clean --all -f -y && \
    fix-permissions $CONDA_DIR && \
    fix-permissions /home/$NB_USER

RUN pip install jupyterlab_templates &&\
    jupyter labextension install jupyterlab_templates && \
    jupyter serverextension enable --py jupyterlab_templates

RUN pip install pytest sas_kernel

# Adding SAS config
ADD saspy/*.jar /opt/conda/lib/python3.9/site-packages/saspy/java/
ADD saspy/sascfg_personal.py /opt/conda/lib/python3.9/site-packages/saspy/

Some specific versions:

Python 3.9.6

jupyter core     : 4.7.1
jupyter-notebook : 6.4.2
qtconsole        : not installed
ipython          : 7.26.0
ipykernel        : 6.0.3
jupyter client   : 6.1.12
jupyter lab      : 3.1.4
nbconvert        : 6.1.0
ipywidgets       : 7.6.3
nbformat         : 5.1.3
traitlets        : 5.0.5

sas-kernel                2.4.11
saspy                     3.7.5  

An earlier build of the image using Python 3.7 worked, so I'm guessing it's a Python 3.9 compatibility issue.

[tomweber-sas]

I'll have to let Jared look at this from the SAS_Kernel side of the fence, but since you're specifying a cfgname= on SASsession(), I'm guessing you have more than one config def available? Do you? If that's so, then when you try to start a SAS notebook, and you submit the first thing, are you getting prompted for which config to use, or is it failing w/out even getting that far? It would have to prompt you, if you have more than one, since there's no way for you to provide it ahead of time. Just trying to get a little more info.

Thanks,
Tom

[hongooi73]

On the older, working image, it would show a prompt asking for the config, followed by the username and password. With the new image, it never gets that far; it crashes immediately with the above error.

[tomweber-sas]

Ok, good (well, for understanding), as that means it's something w/ the kernel or the install or some requirement. I'll have to let Jared look into this. I'm not sure about the kernel requirements or install or diagnosing what this might be. Thanks for the info!
Tom

[snoopy369]

I replicated this:

• Created new python 39 environment
• Installed sas_kernel and jupyter notebook
• Running saspy in python works fine (with a sascfg_personal) inside this jupyter notebook
• Running sas_kernel fails, sasme as Hongooi

I see this:

    We failed in getConnection
    An exception was thrown during the encryption key exchange.

Is sas_kernel using a different method than SASPY for handling the key exchange? Does it need its own version of the TK files separate from the SASPY ones, or something along those lines?

edit: Looks like hongooi is'nt getting that error, so perhaps this is an entirely different issue.

[hongooi73]

On a kind-of related note: when I do conda install sas_kernel, I invariably end up with version 2.3.1, even though the Anaconda website says it has 2.4.11. I have to use pip install to get the latest version. This happens even in a brand-new empty environment.

[tomweber-sas]

The SAS_Kernel, uses SASPy under the covers to interface to SAS. It's just a 'thin client' on top of SASPy. So, any connection configuration and all of that is just saspy. So, since you had saspy connecting, but failing in the SAS notebook, even before asking which configuration definition to use, it seems like something specific to the kernel or environment or something.
@snoopy369 for your case, does the python notebook work (saspy itself)?

[snoopy369]

@tomweber-sas Yes, SASPy works fine. IE, if I take the same Jupyter Notebook instance and start a Python3, and then do the basic

import saspy
sas = saspy.SASsession

Then I get a SAS session established and can run code normally.

I'm not totally sure my issue isn't on my end though with the environments - I'm not too handy with environments and since it doesn't look like it's exactly the same issue Hongooi is having, it may be me doing something wrong (I'm running Jupyter inside a 3.9 environment installed inside my 3.8 base...). In my case this isn't critical so don't worry about it unless it happens to be useful for Hongooi's issue, I was just trying to replicate.

[jld23]

@hongooi73 thanks for the details on this issue. I've been able to reproduce the issue locally but I haven't completed the fix yet. I hope to have it done this week. There are a couple of issues at play so I'm working through those interactions.

[jld23]

Just an update. I'm still trying to resolve why this fails inside a docker image. It is working on my Ubuntu system with python 3.9. As a short-term workaround, you can build the container with just one valid configuration -- there is no prompting in that case it automatically uses the only configuration option. You don't need to remove the other entries just change the python list (example below).

I'm still looking at the issue and little baffled at the moment why it isn't working.

# SAS_config_names   = ['iomviya', 'httpsviya','viya4_trial', 'viya4', 'saspc']
SAS_config_names   = ['httpsviya']

[hongooi73]

Thanks but I'm still getting the same error. 😕

I should also mention: I'm running Jupyter Lab using docker-compose, not the OG notebook interface. Here's the docker-compose yml, stripped down a little bit:

version: '3'
services:
  notebook:
    image: mr
    hostname: notebook
    cap_add:
      - SYS_ADMIN
      - DAC_READ_SEARCH
    ports:
      - 8888:8888
    environment:
      - GRANT_SUDO=yes
    user: root
    command: start.sh jupyter lab --ServerApp.token='' --ServerApp.root_dir='/'

And then I connect to http://localhost:8888/lab. Sorry for leaving this out originally.

[hongooi73]

Some progress! Connecting with IOM is still broken, but if I switch to the ssh config, I get a different error:

The server isn't setup for an ssh connection, and the path is wrong, so I wasn't expecting it to connect. But it looks like the original error is specific to IOM.

[hongooi73]

Here's the IOM config:

grid01 = {'java'      : '/usr/bin/java',
            'iomhost'   : 'hostname',
            'iomport'   : 18592
            }

The JDK version is openjdk-11-jre-headless.

[hongooi73]

Hi, just wondering if there's been any progress on this lately?

[jld23]

@hongooi73 ,
I've been looking into it but no progress to share. I'm struggling to debug since it only appears inside of docker and not on either of my systems (ubuntu & osx).

[jld23]

@hongooi73,

Still working on this...

The issue is not docker.
Nor is it solely a python 3.9 issue.
Jupyter Lab and Jupyter Notebook are not factors.

The issue appears, from my testing, only python 3.9 and docker. I'm not sure what you're loosing by reverting to python 3.8+ but I don't have a better work around currently.

This code has been unchanged for 5+ years and I'm still digging through dependency changes to see what might be the issue. Debugging a Jupyter Kernel is more painful than anyone would like.

Here is a Dockerfile similar to what you provided:

# FROM jupyter/pyspark-notebook:notebook-6.4.2
FROM jupyter/pyspark-notebook:python-3.8.8

RUN mamba install -c conda-forge -y nodejs && \
    conda clean --all -f -y && \
    fix-permissions $CONDA_DIR && \
    fix-permissions /home/$NB_USER

RUN pip install jupyterlab_templates &&\
    jupyter labextension install jupyterlab_templates && \
    jupyter serverextension enable --py jupyterlab_templates 

RUN pip install saspy sas_kernel

USER jovyan
RUN mkdir -p /home/jovyan/.config/saspy
COPY --chown=$NB_USER:users sascfg_personal_issue83.py /home/jovyan/.config/saspy/sascfg_personal.py
COPY --chown=$NB_USER:users saspy.ipynb .
COPY --chown=$NB_USER:users sas_kernel.ipynb .

# docker run -it -P sas_kernel:issue83

and here is the screen shot from the container prompting the user for which configuration in both SASPy and the SAS Kernel.

[hongooi73]

No worries, thanks for the update @jld23. We're using saspy in a Python notebook for now, which doesn't seem to have any problems.

[tomweber-sas]

Hey, so Jared isn't at SAS anymore, and I've started looking into issues for this repo. I've fixed the other 2 that had been outstanding on here; never fixed. This one I've looked into today and found the problem to be that prompting in the kernel interface has changed at some point and that's what is broke. So, I've made a fix to account for that, which is actually in SASPy, since that's where the processing is happening. I've pushed that out and when I build then next version, it will then fix thins problem for the SAS_kernel. I also just built a new version of the kernel and build the doc with a number of doc fixes. So, once I publish saspy V4.4.1 (tomorrow), getting both new releases will address this problem and the other one, 45 I also just fixed. I'll close this once saspy is published, again, I plan on doing that tomorrow.
Thanks,
om

[tomweber-sas]

nvm, I went ahead and built it out now. So, if you get the current production version of saspy, V4.4.1 (or higher, later), then this bug will go away and prompting will again work in the SAS_kernel in the newer versions of jupyter. Closing this out. If you have any issues though, just reopen or open a fresh issue.
Thanks,
Tom