Installed Python Modules Not Accessible

[TuckAndRoll]

I am very new to Che and I'm trying to get my own python environment set up. I have created my own python docker image that has a 3rd party python module (Pyomo) installed. When I create a workspace and pull this image in via the devfile I can't use the module. I can open a terminal to that image and see that it is installed, but when I run my python script in Theia it doesn't recognize it as being installed. When I try to run

import pyomo

it fails because it can't find it.

My devfile:

metadata:
  name: Pyomo
projects:
  - name: python-hello-world
    source:
      location: 'https://github.com/che-samples/python-hello-world.git'
      type: git
      branch: master
components:
  - id: ms-python/python/latest
    memoryLimit: 512Mi
    type: chePlugin
  - mountSources: true
    memoryLimit: 512Mi
    type: dockerimage
    alias: pyomo
    image: '<my-python-image>'
apiVersion: 1.0.0
commands:
  - name: run
    actions:
      - workdir: '${CHE_PROJECTS_ROOT}/python-hello-world'
        type: exec
        command: python hello-world.py
        component: python

My docker file for the custom python image:

FROM    192.168.1.8:5000/<ubuntu-image-base>

RUN     sudo apt-get purge -y python.* && \
        sudo apt autoremove -y && \
        sudo apt-get update && \
        sudo apt-get install -y --no-install-recommends \
        autoconf \
        automake \
        bzip2 \
        file \
        g++ \
        gcc \
        imagemagick \
        libbz2-dev \
        libc6-dev \
        libcurl4-openssl-dev libdb-dev libevent-dev libffi-dev libgdbm-dev libgeoip-dev libglib2.0-dev libjpeg-dev \
        libkrb5-dev liblzma-dev libmagickcore-dev libmagickwand-dev libmysqlclient-dev libncurses-dev libpng-dev \
        libpq-dev libreadline-dev libsqlite3-dev libssl-dev libtool libwebp-dev libxml2-dev libxslt-dev libyaml-dev make patch xz-utils zlib1g-dev
ENV     LANG=C.UTF-8
ENV     PYTHON_VERSION=3.7.4
ENV     PYTHON_PIP_VERSION=19.2.3
RUN     set -ex && \
        sudo curl -fSL "https://www.python.org/ftp/python/${PYTHON_VERSION%%[a-z]*}/Python-$PYTHON_VERSION.tar.xz" -o python.tar.xz && \
        sudo curl -fSL "https://www.python.org/ftp/python/${PYTHON_VERSION%%[a-z]*}/Python-$PYTHON_VERSION.tar.xz.asc" -o python.tar.xz.asc && \
        export GNUPGHOME="$(mktemp -d)" && \
        sudo rm -r "$GNUPGHOME" python.tar.xz.asc && \
        sudo mkdir -p /usr/src/python && \
        sudo tar -xJC /usr/src/python --strip-components=1 -f python.tar.xz && \
        sudo rm python.tar.xz && \
        cd /usr/src/python && \
        sudo ./configure --enable-shared --enable-unicode=ucs4 && \
        sudo make -j$(nproc) && \
        sudo make install && \
        sudo ldconfig && \
        sudo pip3 install --upgrade --ignore-installed pip==$PYTHON_PIP_VERSION && \
        sudo pip3 install pyomo && \
        sudo find /usr/local \( -type d -a -name test -o -name tests \) -o \( -type f -a -name '*.pyc' -o -name '*.pyo' \) -exec rm -rf '{}' + && \
        sudo rm -rf /usr/src/python
RUN     cd /usr/local/bin && \ 
        sudo ln -s easy_install-3.7 easy_install && \
        sudo ln -s idle3 idle && \
        sudo ln -s pydoc3 pydoc && \
        sudo ln -s python3 python && \
        sudo ln -s python-config3 python-config
RUN     sudo pip3 install --upgrade pip && \
        sudo pip3 install --no-cache-dir virtualenv && \
        sudo pip3 install --upgrade setuptools && \
        sudo pip3 install 'python-language-server[all]'
EXPOSE  8080

Any help would be greatly appreciated.

[tolusha]

@TuckAndRoll
That's a good question. You installed the pyomo package in the dev container of your workspace but the python plugin works in a sidecar container.

There is no a general way to fix the issue.

1. It requires to build a docker image for the python plugin. Something like that:

FROM eclipse/che-remote-plugin-python-3.7.3:next
pip3 install pyomo

2. Create a new plugin to use the image above
3. Create a new devfile to use the new plugin

[TuckAndRoll]

@tolusha
Thank you for your response! This confirms what I thought I was seeing, in that there were two separate environments that I was working with between the python runtime environment and the image that I had a terminal open into.

Do you have any resources that you would recommend for building a che plugin like this?

[tolusha]

By "creating a pluging" a mean just create a new devfile [1] and replace underlying image [2] whith pyomo package installed.

[1] https://github.com/eclipse/che-plugin-registry/blob/master/v3/plugins/ms-python/python/latest/meta.yaml
[2] https://github.com/eclipse/che-plugin-registry/blob/master/v3/plugins/ms-python/python/latest/meta.yaml#L16

[tsmaeder]

@tolusha can we configure the Python extension to pick up the installed python module if it's accessible on the file system?

[TuckAndRoll]

@tolusha - Thank you very much for the direction. I have created a custom Python plugin using the microsoft base plugin that is included with Che and successfully deployed it into my environment.

@tsmaeder - I don't fully understand how everything works together between the plugin and the development image, but if it were possible to have Python plugin pull the installed modules on the development image (like in the site-packages folder) into the plugin environment that would be ideal. It would also make it easier to support more custom Python environments out of the box.

[TuckAndRoll]

@tolusha @tsmaeder

On a related note, I have created the image and have also installed COIN-OR's CBC solver to use with the referenced package Pyomo. What's interesting is that I have installed CBC in the image for both the plugin and also the dev image in the exact same way, but running the python script on the dev image via the terminal succeeds, while trying to debug the same file on the plugin fails. I'm currently trying to track down any possible differences between the runtimes, but as of right now I'm a bit stumped. As always, your insight is valuable and appreciated.

[TuckAndRoll]

For reference, here are my two docker files for the plugin and the dev file.

Pyomo Plugin:

FROM    eclipse/che-remote-plugin-python-3.7.3:next

RUN     pip3 install pyomo

COPY    COIN-OR/build/bin/ /src/Cbc/build/bin

COPY    COIN-OR/build/lib/ /lib/

COPY    COIN-OR/neos2.mps /src

ENV     PATH="/src/Cbc/build/bin:${PATH}" COMPILER_VERSION=$COMPILER_VERSION COMPILER=$COMPILER

Python Dev Image:

FROM    <myRepo>/atlas-python3.7-base:latest

RUN     sudo apt-get -y update && \
        sudo apt-get -y upgrade && \
        sudo apt-get -y install gfortran

RUN     sudo pip3 install pyomo

COPY    COIN-OR/build/bin/ /src/Cbc/build/bin

COPY    COIN-OR/build/lib/ /lib/

ENV     PATH="/src/Cbc/build/bin:${PATH}" COMPILER_VERSION=$COMPILER_VERSION COMPILER=$COMPILER

WORKDIR /src

[tolusha]

@TuckAndRoll

while trying to debug the same file on the plugin fails

Do you have any errors that might be helpful?

We faced some issues with debugging python applications recently
#13816

The idea is that we have to run the application in a dev container and use a remote connection to debug.

[TuckAndRoll]

@tolusha
I think I understand what you are saying. Currently, my launch.json file is set to "internal console" for my debug configuration:

{
  "version": "0.2.0",
  "configurations": [
    {
        "name": "Debug",
        "type": "python",
        "request": "launch",
        "program": "${file}",
        "console": "internalConsole"
    }
]
}

I also have tasks.json that has a run task that will execute the python code in my dev environment, but it's obviously not with a remote debugger attached:

{
    "tasks": [
        {
            "type": "che",
            "label": "run",
            "command": "python helloPyomoWorld.py",
            "target": {
                "workingDir": "${CHE_PROJECTS_ROOT}/PyomoExamples/HelloPyomoWorld",
                "containerName": "pyomo"
            }
        }
    ]
}

I'm not exactly sure how to set it up for remote debugging. Would I configure that in the launch.json and tasks.json files? Is there an example of how to do that specifically for Python? Do I need to modify the development container image in some way to prepare for this type of debugging?

I apologize for my limited knowledge of this so far. Your help is greatly appreciated and I am becoming more and more familiar with the whole system.

[tolusha]

Have a look at my example [1]

1. Modify a source file (manage.py in my case) and add such code:

        import os
        if os.environ.get("DEBUG_MODE", "False") == "True":
            import ptvsd
            ptvsd.enable_attach()
            ptvsd.wait_for_attach()

2. Install ptsvd in a dev container
   pip install ptvsd

3. Start application in a debug mode
   export DEBUG_MODE=True && python SOURCE_FILE

4. Wait until message is appeared

A new process is now listening on port 5678 but this port is not exposed in the workspace as a server. Would you want to add a redirect for this port so it becomes available ?

5. Start debugging using the configuration below:

 {
      "name": "Python: Remote Attach",
      "type": "python",
      "request": "attach",
      "port": 5678,
      "host": "localhost",
      "pathMappings": [
        {
          "localRoot": "${workspaceFolder}",
          "remoteRoot": "${workspaceFolder}"
        }
      ]
    }

[1] https://github.com/che-samples/django-realworld-example-app

[tolusha]

Probably it is possible to debug without modifying sources [1] but I haven't tried yet.
[1] https://github.com/microsoft/ptvsd#ptvsd-cli-usage

[che-bot]

Issues go stale after 180 days of inactivity. lifecycle/stale issues rot after an additional 7 days of inactivity and eventually close.

Mark the issue as fresh with /remove-lifecycle stale in a new comment.

If this issue is safe to close now please do so.

Moderators: Add lifecycle/frozen label to avoid stale mode.

[tsmaeder]

/remove-lifecycle stale

[che-bot]

Issues go stale after 180 days of inactivity. lifecycle/stale issues rot after an additional 7 days of inactivity and eventually close.

Mark the issue as fresh with /remove-lifecycle stale in a new comment.

If this issue is safe to close now please do so.

Moderators: Add lifecycle/frozen label to avoid stale mode.

[ericwill]

Our sample devfiles come with virtual environment tasks: the Python Django devfile is a good example to use. It will create a virtual environment and install modules from a requirements.txt. You can adapt this devfile to your needs.