Gitpod break the behavior of pre-commit of Python

[jefer94]

Bug description

[INFO] Restored changes from /home/gitpod/.cache/pre-commit/patch1626779319-12649.
An unexpected error has occurred: CalledProcessError: command: ('/home/gitpod/.cache/pre-commit/reponkw94t64/py_env-python3.9/bin/python', '-mpip', 'install', '.')
return code: 1
expected return code: 0
stdout: (none)
stderr:
    ERROR: Can not perform a '--user' install. User site-packages are not visible in this virtualenv.
    
Check the log at /home/gitpod/.cache/pre-commit/pre-commit.log

version information

pre-commit version: 2.13.0
sys.version:
    3.9.6 (default, Jul  3 2021, 16:40:50) 
    [GCC 9.3.0]
sys.executable: /workspace/apiv2/.venv/bin/python
os.name: posix
sys.platform: linux

### error information

An unexpected error has occurred: CalledProcessError: command: ('/home/gitpod/.cache/pre-commit/reponkw94t64/py_env-python3.9/bin/python', '-mpip', 'install', '.')
return code: 1
expected return code: 0
stdout: (none)
stderr:
    ERROR: Can not perform a '--user' install. User site-packages are not visible in this virtualenv.
    
Traceback (most recent call last):
  File "/workspace/apiv2/.venv/lib/python3.9/site-packages/pre_commit/error_handler.py", line 65, in error_handler
    yield
  File "/workspace/apiv2/.venv/lib/python3.9/site-packages/pre_commit/main.py", line 385, in main
    return run(args.config, store, args)
  File "/workspace/apiv2/.venv/lib/python3.9/site-packages/pre_commit/commands/run.py", line 408, in run
    install_hook_envs(to_install, store)
  File "/workspace/apiv2/.venv/lib/python3.9/site-packages/pre_commit/repository.py", line 224, in install_hook_envs
    _hook_install(hook)
  File "/workspace/apiv2/.venv/lib/python3.9/site-packages/pre_commit/repository.py", line 82, in _hook_install
    lang.install_environment(
  File "/workspace/apiv2/.venv/lib/python3.9/site-packages/pre_commit/languages/python.py", line 204, in install_environment
    helpers.run_setup_cmd(prefix, install_cmd)
  File "/workspace/apiv2/.venv/lib/python3.9/site-packages/pre_commit/languages/helpers.py", line 52, in run_setup_cmd
    cmd_output_b(*cmd, cwd=prefix.prefix_dir)
  File "/workspace/apiv2/.venv/lib/python3.9/site-packages/pre_commit/util.py", line 154, in cmd_output_b
    raise CalledProcessError(returncode, cmd, retcode, stdout_b, stderr_b)
pre_commit.util.CalledProcessError: command: ('/home/gitpod/.cache/pre-commit/reponkw94t64/py_env-python3.9/bin/python', '-mpip', 'install', '.')
return code: 1
expected return code: 0
stdout: (none)
stderr:
    ERROR: Can not perform a '--user' install. User site-packages are not visible in this virtualenv.

Steps to reproduce

Select a workspace with Django, install pipenv, install pre-commit, do anything that use pre-commit

Expected behavior

That pre-commit run in Gitpod without modify the .bashrc also is common break the docker image if we trying to mount other .bashrc

Example repository

No response

Anything else?

Here is my solution pre-commit/pre-commit#1985

[csweichel]

Sounds like we should not be setting PIP_USER=1 in workspace-full. Would that solve the issue?

[jefer94]

Really yes, the team of pre-commit perform a pip install ... to install the hooks dependencies from .pre-commit-config.yaml that normally work because Python by default not set PIP_USER=1 but in Gitpod this environment is set, this configuration should still generating problems with more packages that should work locally but don't in Gitpod and this for any newbie is weird

[billsioros]

Hello everybody!

I'm facing a similar (maybe even the exact same, but I'm not quite sure as I'm pretty new to Gitpod 😄) issue. Given the following .gitpod.Dockerfile:

FROM gitpod/workspace-full

SHELL ["/bin/bash", "-o", "pipefail", "-c"]

ENV PATH="${HOME}/.poetry/bin:${PATH}"
ENV PIP_USER="no"

RUN curl -sSL https://raw.githubusercontent.com/python-poetry/poetry/master/get-poetry.py | python3 - \
    && poetry config virtualenvs.create "true" \
    && poetry config virtualenvs.in-project "true"

PIP_USER remains set to yes. I have tried various values just in case (false, 0, no), as well as not using Environment replacement, with no results whatsoever.

[jefer94]

Should be great that .bashrc was generated getting the environment

@billsioros this was my solution

FROM gitpod/workspace-postgres

RUN sudo apt-get update \
    && sudo apt-get update \
    && sudo apt-get install -y redis-server \
    && sudo apt-get clean \
    && sudo rm -rf /var/cache/apt/* /var/lib/apt/lists/* /tmp/*

RUN pyenv update && pyenv install 3.9.6 && pyenv global 3.9.6
RUN pip install pipenv

# remove PIP_USER environment
USER gitpod
RUN if ! grep -q "export PIP_USER=no" "$HOME/.bashrc"; then printf '%s\n' "export PIP_USER=no" >> "$HOME/.bashrc"; fi

[billsioros]

Hi @jefer94, thank you very much! Sadly my issue has not been resolved. Shouldn't PIP_USER (and possibly other environment variables set by the Dockerfile ) be respected, in any case?

Edit: I'm not quite sure, but could it be due to spawning a python virtual environment and afterwards installing the dependencies?

[jefer94]

@billsioros no necesaryly, you have the PIP_USER=yes because the gitpod image has your own .bashrc file, if you run gp open ~/.bashrc you should see PIP_USER is being setting, and this file is static

[billsioros]

Hello again @jefer94. PIP_USER was being set to no, but after instnatiating a new virtual environment, attempting to install the dependencies resulted in nothing. I was able to resolve my issue by not instantiating a new virtual environment.

If I'm correct, Gitpod does not proceed to execute commands specified by tasks if a new shell is instantiated. But should this be considered an issue? I'm not sure about any security complications it might introduce, but I think something like this should be at least documented (I'm sorry if it is already documented, I search for information on the matter but couldn't come up with anything 😢)

[jefer94]

In Gitpod I think is not necesary use virtualenv, but for you resolve you issue you need to set the PIP_USER before Gitpod open the custom docker image and for me it's safe, venv just prevent have conflicts dependencies, in you local machine is not recommended install each dependency of each project in the system or user packages

[csweichel]

This is an issue with workspace-full itself. I've raised a PR in workspace-images to set PIP_USER=no

[jankeromnes]

This is an issue with workspace-full itself.

Isn't this rather an issue with the gitpod-layer?

gitpod/components/image-builder/workspace-image-layer/gitpod-layer/debian/gitpod/.bashrc-prepend

Lines 39 to 44 in af51739

	export PIPENV_VENV_IN_PROJECT=true
	export PIP_USER=yes
	export PYTHONUSERBASE=/workspace/.pip-modules
	export PATH=$PYTHONUSERBASE/bin:$PATH
	unset PIP_TARGET
	unset PYTHONPATH

[da-moon]

@billsioros I got that problem fixed in a hacky way by setting env vars in files under /etc/profile.d/ and having /etc/bash.bashrc to source all files under /etc/profile.d/ and some sed statements to update local ~/.bashrc and ~/profile file.

# ────────────────────────────────────────────────────────────────────────────────
USER root
ENV EDITOR="nvim"
ENV VISUAL="nvim"
ENV PATH="${PATH}:/usr/local/bin"
WORKDIR "/etc/profile.d"
RUN \
  [ -d "${HOME}/.bashrc.d" ] && while read i; do ln -sf "$i" "/etc/profile.d/$(basename $i).sh"; done < <(find ${HOME}/.bashrc.d -type f ) || true ; \
  curl \
  -fsSl https://raw.githubusercontent.com/3hhh/fzfuncs/master/bashrc_fzf \
  -o /etc/profile.d/bashrc_fzf.sh \
  && ( \
  echo "export EDITOR=\"${EDITOR}\"" ; \
  echo "export VISUAL=\"${VISUAL}\"" ; \
  echo 'export DOCKER_BUILDKIT="0"' ; \
  echo 'export BUILDKIT_PROGRESS="auto"' ; \
  echo 'export COMPOSE_DOCKER_CLI_BUILD="0"' ; \
  echo 'export GEM_HOME=/workspace/.rvm' ; \
  echo 'export GEM_PATH=$GEM_HOME:$GEM_PATH' ; \
  echo 'export GOROOT="${HOME}/go" ;' ; \
  echo 'export GOPATH="/workspace/go-packages" ;' ; \
  echo 'export PYTHONUSERBASE=/workspace/.pip-modules ;' ; \
  echo 'export PATH="${PATH}:/usr/local/bin"' ; \
  echo '[ -d ${GEM_HOME} ] && export PATH="${GEM_HOME}/bin:$PATH" ;' ; \
  echo '[ -d ${GOROOT} ] && export PATH="${GO_ROOT}/bin:$PATH" ;' ; \
  echo '[ -d ${GOPATH} ] && export PATH="${GOPATH}/bin:$PATH" ;' ; \
  echo '[ -d ${HOME}/.local/bin ] && export PATH="${HOME}/.local/bin:${PATH}"' ; \
  echo '[ -d ${HOME}/.cargo/bin ] && export PATH="${HOME}/.cargo/bin:${PATH}";';  \
  echo '[ -d ${PYTHONUSERBASE} ] && export PATH="${PYTHONUSERBASE}/bin:$PATH" ;' ; \
  echo '[ -d ${HOME}/.pyenv/bin ] && export PATH="${HOME}/.pyenv/bin:${PATH}";' ; \
  echo '[ -d ${HOME}/.pyenv/shims ] && export PATH="${HOME}/.pyenv/shims:${PATH}" ;' ; \
  echo '[ -d ${HOME}/bin ] && export PATH="${HOME}/bin":${PATH}' ; \
  echo 'npm --version >/dev/null 2>&1 && export PATH="$(npm -g bin):${PATH}" ;' ; \
  echo 'yarn --version >/dev/null 2>&1 && export PATH="$(yarn global bin):${PATH}" ;' ; \
  echo 'gp --help >/dev/null 2>&1 && (eval $(gp env | sed -e "s/\(=[[:blank:]]*\)\(.*\)/\1\"\2\" ;/" -e "s/^/export /g") ; source <(gp completion) ;)' ; \
  echo 'starship --version >/dev/null 2>&1 && eval "$(starship init bash)" ;' ; \
  echo 'just --version >/dev/null 2>&1 && eval "$(just --completions bash)" ;' ; \
  echo 'rustup --version >/dev/null 2>&1 && eval "$(rustup completions bash rustup)" && eval "$(rustup completions bash cargo)";' ; \
  echo 'curl --version >/dev/null 2>&1 && source <(curl -sSL git.io/forgit) ;' ; \
  ) | tee 00-env.sh > /dev/null \
  && ( \
  echo "alias apt='sudo apt'" ; \
  echo "alias g='git'" ; \
  echo 'exa --version >/dev/null 2>&1 && alias la="exa -A";alias ll="exa -alhF";alias ls="exa";' ; \
  echo 'bat --version >/dev/null 2>&1 && alias cat="bat -pp";' ; \
  echo 'just --version >/dev/null 2>&1 && alias j="just";' ; \
  echo "alias nvim='nvim --headless -c \"call dein#recache_runtimepath()\" -c \"qall\" && nvim'" ; \
  ) | tee 01-alias.sh > /dev/null \
  && ( \
  echo '(rg -h >/dev/null 2>&1 && bat -h >/dev/null 2>&1 && sk -h >/dev/null 2>&1 ) && function fp { rg --files $1 | sk --preview="bat {} --color=always --style=grid,numbers" ; }' ; \
  echo '(rg -h >/dev/null 2>&1 && sk -h >/dev/null 2>&1 ) && function fs { sk -i -c "rg {} --color=always $1" --ansi ; }' ; \
  ) | tee 02-func.sh > /dev/null \
  && [ -r /etc/bash.bashrc ] && (\
    echo 'while read i; do source "$i"; done < <(find /etc/profile.d/ -name "*.sh") ;' ; \
  ) | tee -a /etc/bash.bashrc
# ────────────────────────────────────────────────────────────────────────────────
USER gitpod
WORKDIR "${HOME}"
RUN \
  [ -r "${HOME}/.bashrc.d/60-python" ] \
  && sed -i -e 's/pyenv/${HOME}\/.pyenv\/bin\/pyenv/g'  "${HOME}/.bashrc.d/60-python" \
  && sed -i \
    -e '/\$HOME\/\.bashrc\.d/d' \
    -e '/PIP/d' \
    -e '/PYTHONUSERBASE/d' \
    -e '/gp completion/d' \
    -e '/\.cargo\/env/d' \
    -e '/rvm/d' \
    -e '/GEM/d' \
    "${HOME}/.bashrc" \
  && ( \
  echo "[ -r '\${HOME}/.bashrc' ] && sed -i \\" ; \
  echo "-e '/\\\$HOME\/\.bashrc\.d/d' \\" ; \
  echo "-e '/cargo/d' \\" ; \
  echo "-e '/PIP/d' \\" ; \
  echo "-e '/PYTHONUSERBASE/d' \\" ; \
  echo "-e '/gp completion/d' \\" ; \
  echo "-e '/rvm/d' \\" ; \
  echo "-e '/GEM/d' \\" ; \
  echo "'\${HOME}/.bashrc' \\" ; \
  echo '&& . ${HOME}/.bashrc' ; \
  ) | tee "${HOME}/.profile" > /dev/null ;
# ────────────────────────────────────────────────────────────────────────────────

there are more steps involved so I would recommend you to study my Gitpod docker image here.

To test and see if your repo and local config works with my changes, you can open my gitpod-workspace repo in Gitpod, clone your repo under workspace/ and see if everything works for you.

[billsioros]

Hello @da-moon,

I have already adopted the workaround described by @jefer94, but I appreciate the advice! Thanks very much 😄

Should be great that .bashrc was generated getting the environment

@billsioros this was my solution

FROM gitpod/workspace-postgres

RUN sudo apt-get update \
    && sudo apt-get update \
    && sudo apt-get install -y redis-server \
    && sudo apt-get clean \
    && sudo rm -rf /var/cache/apt/* /var/lib/apt/lists/* /tmp/*

RUN pyenv update && pyenv install 3.9.6 && pyenv global 3.9.6
RUN pip install pipenv

# remove PIP_USER environment
USER gitpod
RUN if ! grep -q "export PIP_USER=no" "$HOME/.bashrc"; then printf '%s\n' "export PIP_USER=no" >> "$HOME/.bashrc"; fi

[jab]

I see this was closed a few months ago, but I just hit this issue. Has the fix been released yet?

In case it helps anyone else, I am working around this for now by prepending env PIP_USER=false to the pre-commit install --install-hooks command in my init section (see https://github.com/jab/bidict/pull/210/files).

[ghuntley]

Have been ablet to reproduce this problem in workspace-images/

gitpod /workspace/workspace-images $ env |grep PIP
PIPENV_VENV_IN_PROJECT=true
PIP_USER=yes

[jefer94]

I think that's not the right approach set PIP_USER=false in .gitpod.yml, this should be in the official docker image

[heitorlessa]

Came here to say that apparently this has been fixed in the latest Gitpod version -- I've just launched a project managed by Poetry + pre-commit that didn't work before.... and setup worked in the first try.

env | grep "PIP"
PIPENV_VENV_IN_PROJECT=true
PIP_USER=no

[svenefftinge]

Awesome, thanks for verifying.

[jab]

I just tried removing the export PIP_USER=false workaround from my .gitpod.yml and now I'm once again getting errors like:

ERROR: Could not install packages due to an OSError: [Errno 13] Permission denied: '__init__.py'
Consider using the `--user` option or check the permissions.

from the GitPod init task showing up in the terminal. See jab/bidict#238.

Are you sure this is fixed?

[heitorlessa]

I tried with a repo without a .gitpod.yml and worked consistently on three attempts - PIP_USER was false by default as comment above. Perhaps it's using a newer image as I haven't used for months?

Run make dev on https://gitpod.io/#https://github.com/awslabs/aws-lambda-powertools-python

[jab]

@svenefftinge, when you have a moment, could you please take a peek at https://gitpod.io/#https://github.com/jab/bidict/pull/238 (which shows that when I remove the workaround for this issue I get the Permission denied...Consider using the --user option error from my previous message), and possibly reopen this issue?

[svenefftinge]

I can reopen the issue, of course. We could certainly need some help from Python experts to figure out what's going on here.

[jab]

Thanks!

[utam0k]

@jab Maybe it has something to do with this issue.
gitpod-io/workspace-images#664

[jab]

Thanks @utam0k, I noticed that $PATH bug too! But it looks like I'm hitting something far weirder. Just left a comment in gitpod-io/workspace-images#664 (comment) in case anyone has any ideas. Thank you!

[utam0k]

Thanks @utam0k, I noticed that $PATH bug too! But it looks like I'm hitting something far weirder. Just left a comment in gitpod-io/workspace-images#664 (comment) in case anyone has any ideas. Thank you!

Thanks for your response. I'll check these comments.

[utam0k]

@jab I will close this issue in order to consolidate the issues. gitpod-io/workspace-images#664 If you still have any questions, please reopen this issue or create a new issue.　

@svenefftinge
Thank you for reopening. Thanks to you I was able to catch the problem.