Pyenv-Init Broken for SSH w/ Bash

[kopfjager007]

System

• Platform: Ubuntu Server 20.04.4 LTS (headless)
  • Note: I built a fresh VM to confirm this issue I saw on one of my prod servers.
• OS Arch: x86_64
• Pyenv Version: 2.3.0
• Python Version: 3.8.10
• C Compiler: 9.4.0 (Ubuntu 9.4.0-1ubuntu1~20.04.1)

Pyenv was Installed via Pyenv Installer. Per the Pyenv Readme, I added the following lines to my ~/.bashrc, ~/.bash_profile, and ~/.profile files:

export PYENV_ROOT="$HOME/.pyenv"
command -v pyenv >/dev/null || export PATH="$PYENV_ROOT/bin:$PATH"
eval "$(pyenv init -)"

Running pyenv doctor shows no errors and states pyenv was installed successfully. I have no issue running any pyenv commands at this stage and can install and use python versions through pyenv without issue.

Issue

The issue I'm experiencing is when accessing my server via SSH with the default login shell set to /bin/bash, the login prompt seems to hang indefinitely. This does not happen when the default shell is set to /bin/zsh, nor does it happen when I login at the server using a keyboard/monitor. This seems to be unique to SSH w/ Bash.

While troubleshooting the issue I noticed several hundred, and eventually thousands, of new Bash processes (ran ps aux):

While the terminal appears to be hung, new bash processes spawn at a rate of about 1000 every ~30 seconds.

While at the apparently hung shell, I ^C and am dropped in at the bash prompt, but there are obvious other issues since it seems when eval "$(pyenv init -)" runs in ~/.bash_profile, bash never sources .bashrc or .profile. Wherever that eval is is where this issue begins.

Given the process listing and some debugging, I believe this is related to Line 151 in pyenv/libexec/pyenv-init in the latest 2.3.0 release; commit 8439f8e18714c2aec0399e9cea022467dc4511ff. If i comment this line out and initiate a new SSH connection, I don't have any issue and don't get thousands of bash processes (which I assume will eventually exhaust resources).

Dropping into /bin/zsh it's clear that the install was successful and Pyenv works.

I did add the recommended debug envvar for pyenv (trace.log) and I also did some shell debugging (debug.log.1) that I will drop here if that will help.
Logs
trace.log
debug.log

Note: debug.log line 338 is where the hang was observed. I then ^C'd .

[native-api]

I cannot reproduce this in a fresh Ubuntu Server 20.04 or 22.04 VM.

Do you have BASH_ENV set in your SSH environment? See the warning at the end of https://github.com/pyenv/pyenv#set-up-your-shell-environment-for-pyenv and #2354.

[kopfjager007]

My prod servers do not have $BASH_ENV set, nor does the vanilla server I had created. Prior to version 2.3.0 I had the pyenv shell environment stuff in .bashrc only, and this worked just fine. Reading the setup link you posted, I removed those lines from .bashrc and .profile and have them only in .bash_profile and that seems to have done the trick. It certainly is odd since my instances don't seem to meet the criteria for that warning to apply, but it seems to have remedied the situation.

Thanks!

[native-api]

From what you and @stevemuskiewicz reported, it looks like something is causing one of the Bash startup files to be sourced in the bash -ec invocation.

Prepend all the Bash startup files with the following code to see which of them is being sourced then search for any references to that file across the server to find out what can be causing that.

export PS4='+(${BASH_SOURCE:-}:${LINENO}): ${FUNCNAME[0]:+${FUNCNAME[0]}(): }'
set -x

[native-api]

then search for any references to that file across the server to find out what can be causing that

That said, if your SSH server is set up to allow injecting environment from the client (the default is not IIRC), your SSH client configuration may be causing that, too.

[native-api]

nor does the vanilla server I had created

@kopfjager007 , if you upload the image of that test VM to some cloud storage, I may be able to reproduce the issue with it.

Make sure to include reproduction steps, too.

[kopfjager007]

From what you and @stevemuskiewicz reported, it looks like something is causing one of the Bash startup files to be sourced in the bash -ec invocation.

Prepend all the Bash startup files with the following code to see which of them is being sourced then search for any references to that file across the server to find out what can be causing that.

export PS4='+(${BASH_SOURCE:-}:${LINENO}): ${FUNCNAME[0]:+${FUNCNAME[0]}(): }'
set -x

That is the output of the debug.log I had attached.

For the VM, it's literally just a vanilla Ubuntu 20.04 server; no special setup or files outside of creating a .bash_profile file that had the pyenv initialization in it. This is the contents of my .bash_profile file

export PS4='+(${BASH_SOURCE:-}:${LINENO}): ${FUNCNAME[0]:+${FUNCNAME[0]}(): }'
set -xv

export PYENV_ROOT="$HOME/.pyenv"
command -v pyenv >/dev/null || export PATH="$PYENV_ROOT/bin:$PATH"
eval "$(pyenv init -)"

# Load the default .profile
[[ -s "$HOME/.profile" ]] && source "$HOME/.profile"

I built the VM from the latest Ubuntu 20.04LTS server ISO image, installed ssh and pyenv, and that's literally all I did. I don't have anything special on my end with SSH; all I'm doing is ssh user@<IP>. I decided to build a basic VM since I wasn't sure if there was something unique to my servers that was causing this, but it also happened with this basic server build too.

Setup:
I downloaded the latest Ubuntu 20.04LTS server ISO from the Ubuntu site and installed. Basically next, next, next... no special configs and I didn't install anything extra (i.e. Apache, SSH, etc) during OS installation).

I installed SSH with:

apt install ssh

Then i installed Pyenv:

curl --silent --location https://raw.githubusercontent.com/pyenv/pyenv-installer/master/bin/pyenv-installer --output pyenv-installer
export PATH=$PATH:/root/.pyenv/bin
/bin/bash pyenv-installer

Confirmed Pyenv is working:

pyenv --version

I initially added this to ~/.bashrc only as I have in the past:

export PYENV_ROOT="/root/.pyenv"
command -v pyenv >/dev/null || export PATH="$PYENV_ROOT/bin:$PATH"
eval "$(pyenv init -)"

Now I leave VMware and access the VM via SSH and notice the issue. I added those lines to .bash_profile and .profile as well per the README and the issue persists.

[leepa]

I have reproduced this issue on a freshly installed Fedora 36 installation.

[leepa]

As an additional - commenting out line 151 in pyenv-init also fixed the problem for me. That line clearly doesn't work over ssh - thank you @kopfjager007! I guess I will wait to see what the actual fix is as my Shell-fu isn't good enough to understand the line.

[houmie]

I have the same problem on a fresh installation of Debian 11.
The only way to resolve this for the time being is to fall back to version 2.2.5

git clone -b v2.2.5 https://github.com/yyuu/pyenv.git ~/.pyenv

Thank you

[ah2501]

I also have this issue on Ubuntu. I reproduced on Ubuntu 20.04. Is the problem line 151 even necessary if it's causing an infinite loop of bash processes? A lot of people access servers over SSH and use bash. If i comment it out like @leepa mentioned everything seems fine.

[unashamedgeek]

Came here in search of a solution to this as well. I am experiencing this across my systems, all which are Ubuntu 20.04 that I ssh into. I see some workarounds, but I can confirm seeing line 151 in pyenv-init causing the issue. I can provide details if necessary, but @kopfjager007 fully detailed exactly what I am seeing.

[native-api]

Okay, thanks to @martindorey's #264 (comment), I found out why this happens -- and why I couldn't reproduce it.

---

Some distros, notably Debian (and hence its derivatives), uncomment a non-default option in Bash's source to source ~/.bashrc when running under SSH, even noninteractively (https://github.com/linuxmint/bash/blob/98aa991a2f5259aff948270d5f06b2ee9a0ccc6c/debian/patches/deb-bash-config.diff#L49, https://github.com/linuxmint/bash/blob/98aa991a2f5259aff948270d5f06b2ee9a0ccc6c/shell.c#L1039-L1053).

However, Debian's stock ~/.bashrc has this guard at the start of it:

case $- in
    *i*) ;;
      *) return;;
esac

Which negates that behavior when Bash is run noninteractively.

---

Looks like some people are fond of removing that guard or running stuff before it...

[native-api]

Though there are valid cases to want to source some standard shell startup file when running noninteractive commands via SSH. So I'm fine with adding --norc to the nested Bash invocation as @martindorey suggested.

[stevemuskiewicz]

Thanks @native-api and @kopfjager007 for chasing this one down. Looks like Fedora/RH doesn't have the bash "guard" in their default ~/.bashrc but I can confirm that after adding it this resolves my hang issue for any ssh sessions when using pyenv 2.3.x.

[houmie]

@native-api You are correct, part of my automated installation script up until Pyenv version 2.2.5 I could do this:

sed -i '1 i\export PYENV_ROOT="$HOME/.pyenv"' ~/.profile
sed -i '2 i\export PATH="$PYENV_ROOT/bin:$PATH"' ~/.profile
sed -i '3 i\eval "$(pyenv init --path)"' ~/.profile

sed -i '1 i\eval "$(pyenv init -)"' ~/.bashrc
sed -i '2 i\eval "$(pyenv virtualenv-init -)"' ~/.bashrc

The second part inserts those two lines above the guard:

case $- in
    *i*) ;;
      *) return;;
esac

I'm not quite following why this was working fine up until Pyenv 2.2.5 , and now it suddenly doesn't anymore. But the solution is that I need to find a way to insert those two lines after the guard. Is this correct?

Because looking at your commit that you merged against this issue, it seems you only added documentation and there is no change to the actual code.

Thanks for clarifying

[martindorey]

only added documentation and there is no change to the actual code

It's a big comment, because this was really quite non-obvious, but I see --norc being added to the code.