Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Issue with environmental variables & PATH on remote server (not sourcing .bashrc) #1519

Closed
alexvicegrab opened this issue Oct 9, 2016 · 21 comments

Comments

Projects
None yet
4 participants
@alexvicegrab
Copy link

commented Oct 9, 2016

Hi,

I'm trying to get the .bashrc file to be loaded (i.e. source /home/ubuntu/.bashrc) when running fab's run command to add some environmental variables and expand the path variable:

run('source /home/ubuntu/.bashrc && echo $PATH')

This shows me only:

[ubuntu@ecX-XX-XXX-XXX-XXX.compute-1.amazonaws.com] out: /usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games

instead of a much longer list of paths that I see when I log into the remote server manually.

How do I get fab to correctly import the .bashrc file in my remote home directory?

Thank you!

@ploxiln

This comment has been minimized.

Copy link

commented Oct 11, 2016

This is a bash thing, not a fab thing.

What files bash decides to source at startup can be complicated http://blog.flowblok.id.au/2013-02/shell-startup-scripts.html and debian/ubuntu (and most distros) have some customization in /etc/profile and in the default ~/.bashrc.

The default shell fab uses is /bin/bash -l -c, and the -l makes it a "login" shell. Without the debian/ubuntu customization, it's possible for a bash "login" shell to source ~/.bash_profile but not ~/.bashrc.

But on ubuntu 16.04, it does seem to source .bashrc by default even for login-shell. But lines added at the bottom of a default .bashrc are not processed, because it bails out near the top if it detects running non-interactively.

Here I've added two lines to the ubuntu-16.04 default user .bashrc

# ~/.bashrc: executed by bash(1) for non-login shells.
# see /usr/share/doc/bash/examples/startup-files (in the package bash-doc)
# for examples

export VAR1=val1

# If not running interactively, don't do anything
case $- in
    *i*) ;;
      *) return;;
esac

export VAR2=val2

# don't put duplicate lines or lines starting with space in the history.
# See bash(1) for more options
HISTCONTROL=ignoreboth
...

Here's a fabfile I'm testing with:

from fabric.api import run, env, task

@task
def get_myvars():
    run("echo VAR1=$VAR1 VAR2=$VAR2")

The results:

$ fab -H testpy05.ec2.st-av.net get_myvars
[testpy05.ec2.st-av.net] Executing task 'get_myvars'
[testpy05.ec2.st-av.net] run: echo VAR1=$VAR1 VAR2=$VAR2
[testpy05.ec2.st-av.net] out: VAR1=val1 VAR2=
[testpy05.ec2.st-av.net] out: 
...

@bitprophet bitprophet closed this Oct 11, 2016

@bitprophet

This comment has been minimized.

Copy link
Member

commented Oct 11, 2016

Sounds legit at a glance, thanks @ploxiln!

@alexvicegrab

This comment has been minimized.

Copy link
Author

commented Oct 11, 2016

Thank you for the detailed answer!

@code-tree

This comment has been minimized.

Copy link

commented Jul 27, 2018

This seems to differ from standard ssh behaviour and was hard to debug. When I manually ssh'd in all worked fine, where as fabric had a different PATH value due to not running .bash_profile in OSX.

@code-tree

This comment has been minimized.

Copy link

commented Jul 27, 2018

@bitprophet this may be an OSX bug??

@bitprophet

This comment has been minimized.

Copy link
Member

commented Jul 31, 2018

@code-tree Are you using Fabric 1 or 2? 1.x uses explicit shell wrappers which would be why it'd differ from standard OpenSSH client; version 2 doesn't do any wrapping and should behave a lot more like OpenSSH client in terms of what the sshd is running on your behalf.

Version 1 has some env config value options for changing that shell wrapper, so you may want to try adding flags to it, like -i which IIRC is bash for "run interactive mode and source some extra files".

@code-tree

This comment has been minimized.

Copy link

commented Jul 31, 2018

That's strange because I'm using version 2.2.1. I currently have to do the following to get my app working:
c.run('bash -l -c "python3 ./configure.py"')
As when I installed Python 3 it added itself to PATH via my .bash_profile, which is not being run by Fabric. I haven't made any modifications to the standard builtin sshd config either.

@bitprophet

This comment has been minimized.

Copy link
Member

commented Aug 3, 2018

That is strange then, I'll have to see if I can replicate. Just did a quick sanity trace to prove what I mean re: Fabric 2 not doing anything 'special' re: sshd driven execution:

@bitprophet

This comment has been minimized.

Copy link
Member

commented Aug 3, 2018

@code-tree also, if you can, please post some more details about exactly what you're running & other environmental details (client and server OS/version, etc etc).

@code-tree

This comment has been minimized.

Copy link

commented Aug 4, 2018

Sure, I'm running an old OSX version as server. If you can confirm all works fine on a newer MacOS then oh well. But if not then maybe an issue with MacOS as server in general?

  • Client: Ubuntu 16.04, Python 3.6.6, OpenSSH 7.2p2, Bash 4.3.48, Fabric 2.2.1
  • Server: OSX 10.11, Python 3.6.6, OpenSSH 6.9p1, Bash 3.2.57
@bitprophet

This comment has been minimized.

Copy link
Member

commented Aug 7, 2018

Side note, I'm wondering if this is actually a case of misdiagnosis:

fabric had a different PATH value due to not running .bash_profile in OSX

Could this instead be an example of #1744, where Fabric 2 is differing from ssh in terms of shunting local env vars down the pipe? (Depends on the exact env vars in question and whether they might be similar locally and remotely; @code-tree could easily disprove this theory by double checking the exact values in play re: their local and remote bash_profile... 😁)


But I'm going to troubleshoot this straight anyway, just so I know I'm not crazy re: my understanding of what the sshd is doing in terms of shells and sourced files.

@bitprophet

This comment has been minimized.

Copy link
Member

commented Aug 8, 2018

Also I wonder if this is related to the scenario/intelligence from #1816, which is also about shells and startup files. It's probably not directly relevant because of the assertion here about ssh behaving one way and Fabric behaving another way (versus it being "just how the sshd is invoking the shell normally"), but worth linking anyways.

Found that trying to stumble on a bunch of deep sshd code diving I remember doing in the last few months, related to something along these lines. I can't find it now though which is irritating. EDIT: ah I think it was an unrelated Paramiko issue (not linking because no point confusing things) about executing commands before authentication. OK.

@bitprophet

This comment has been minimized.

Copy link
Member

commented Aug 8, 2018

All right, so what's really happening when we request command exec?

  • SSH-the-protocol wants you to either make a shell channel request (which loads up the user's defined login shell program and does not take any command) or an exec channel request (which just says "executes given command string, which may contain a path" and doesn't specify more)
  • Paramiko does the latter, as we found above.
  • So what's OpenSSH itself actually doing? I'll be looking at my local checkout of openssh-portable (at 775f8a23f2353f5869003c57a213d14b28e0736e)

Well, first I'll goof around a little. Running against an arbitrary Debian 8.10 container (which is running OpenSSH 6.7!) I see this in the DEBUG3-level logs upon doing ssh localhost whoami:

debug1: server_input_channel_req: channel 0 request exec reply 1
debug1: session_by_channel: session 0 channel 0
debug1: session_input_channel_req: session 0 req exec
Starting session: command for root from 172.17.0.1 port 42598

Doing fab -H localhost -- whoami:

debug1: server_input_channel_req: channel 0 request exec reply 1
debug1: session_by_channel: session 0 channel 0
debug1: session_input_channel_req: session 0 req exec
Starting session: command for root from 172.17.0.1 port 42610

OK, they're both doing exec...that's to be expected. Anything different in the process tree? shouldn't be, and...there is not. both look like this:

sshd,1 -D -e
  └─sshd,1535
      └─pstree,1537 -alpU

So, definitely no shell in play? What's it even doing? Can I use shelly things like &&?

I can! eg whoami && id works fine. So that's a bit weird given my understanding of Unix process control; it seems unlikely the sshd is just literally doing exec(string), but if it were using a shell to interpret, wouldn't we see that in pstree?

I think it's time to really dig into openssh-portable and see what's what.


@bitprophet

This comment has been minimized.

Copy link
Member

commented Aug 8, 2018

Also, huh, I did do part of this tracing (but for a different reason/focus) in the past, as in, Dec 2016: paramiko/paramiko#398 (comment). Had totally forgotten...


The reason why I'm not seeing the shell in pstree is because of the use of execve, which does a replacement of parent process instead of spawning a child. Good to know. (EDIT: wrong, that just replaces the sshd child proc with the shell; the shell itself may then also be doing a replace-me style exec call; see below comments.)


Anyway, so! This still should mean Fabric and OpenSSH's clients are doing the same exact thing in terms of shell execution; nothing either of them is doing would be modifying these parts of OpenSSH's codebase. It should always be something like bash -c "python3 ./configure.py", with the actual value of bash depending on @code-tree's macOS level user and their configured shell.

Makes me wonder if my hunch about env var transmission being the differentiator is accurate, since that's the only other thing I can think of re: how the two execution environments could differ. FWIW in my CLI-focused testing, env reports same on both systems, and regardless of those other env related issues, the default behavior (due to SSH security policy) should always be that no local env "leaks" to the other side.

@ploxiln

This comment has been minimized.

Copy link

commented Aug 8, 2018

Yeah, I'm pretty sure sshd always runs the command string in the user's shell (configured in /etc/passwd). However it prepends the command string with "exec" if it can parse the command string and determine that it's a single command.

$ ssh testdeploy02.ec2.st-av.net pstree -a
...
  |-sshd -D
  |   `-sshd 
  |       `-sshd  
  |           `-pstree -a
$ ssh testdeploy02.ec2.st-av.net 'pstree -a && sleep 1'
...
  |-sshd -D
  |   `-sshd 
  |       `-sshd  
  |           `-bash -c pstree -a && sleep 1
  |               `-pstree -a
$ ssh testdeploy02.ec2.st-av.net 'VAR=-a sh -c "exec pstree \$VAR"'
...
  |-sshd -D
  |   `-sshd 
  |       `-sshd  
  |           `-pstree -a

As for environment variables, these are mostly controlled by the ssh client and sshd server configs:

$ grep Env /etc/ssh/*_config
/etc/ssh/ssh_config:    SendEnv LANG LC_*
/etc/ssh/sshd_config:AcceptEnv LANG LC_*

So the ssh client does explicitly send some variables (outside of the command-string itself) and they are filtered by sshd. But there are apparently some exceptions:

$ ssh testdeploy02.ec2.st-av.net env | grep TERM
$ ssh -t testdeploy02.ec2.st-av.net env | grep TERM
TERM=xterm-256color
@ploxiln

This comment has been minimized.

Copy link

commented Aug 8, 2018

Actually I think the "automatic exec to replace shell with single command" behavior is a feature of bash:

$ dash -c "pstree -a"
...
  ├─sshd -D
  │   └─sshd 
  │       └─sshd  
  │           └─bash
  │               └─dash -c pstree -a
  │                   └─pstree -a
$ bash -c "pstree -a"
  ├─sshd -D
  │   └─sshd 
  │       └─sshd  
  │           └─bash
  │               └─pstree -a

EDIT: For completeness:

$ bash -c "pstree -a && sleep 1"
  ├─sshd -D
  │   └─sshd 
  │       └─sshd  
  │           └─bash
  │               └─bash -c pstree -a && sleep 1
  │                   └─pstree -a
@bitprophet

This comment has been minimized.

Copy link
Member

commented Aug 8, 2018

Ah yea, I was mistaken, execve just means the sshd child proc is what gets replaced, so what happens after that is up to the shell itself and what it does for -c xxx. My tests were in zsh, fwiw.

@bitprophet

This comment has been minimized.

Copy link
Member

commented Aug 8, 2018

Also @ploxiln yea the thing with env vars is referencing other tickets linked above, though it remains to be seen if @code-tree is truly experiencing a different symptom to those.

@code-tree

This comment has been minimized.

Copy link

commented Aug 9, 2018

Sorry, I think this is a false alarm. I wasn't aware that there was any difference between executing a command via ssh inline and executing after logging in via ssh. When I do ssh host 'echo $PATH' it also has not updated PATH where as echo $PATH after logging in works fine.

I still don't really get why ssh does this (seems like I have to indeed do ssh host 'bash -l -c "python3 ./configure.py"'). Regardless, it is safe to say this isn't a Fabric issue after all. Sorry for the confusion.

@bitprophet

This comment has been minimized.

Copy link
Member

commented Aug 9, 2018

@code-tree it's related to what was mentioned earlier - shells have a few different modes they operate in, frequently called 'login' and 'interactive' (and usually those are on top of a base mode which is considered neither) and which mode the shell is being run in, changes which set of rc-files it loads.

The salient point here is that bash -c xxx isn't considered "interactive" and thus skips loading certain files - like .bash_profile - but just running bash without -c is usually interactive, and does load profile files.

As you saw above - sshd always runs bash -c <command send down the pipe> when you ask it to run a single command (as Fabric does, or as ssh host command does), and thus, is always in non-interactive mode. (Unless you do what Fabric 1 does and run your own, nested, shell with -l...but then you're running bash -c "bash -l -c \"oh god escaping is hard help\"" and I'm going to go have some 🥃 now.

@bitprophet

This comment has been minimized.

Copy link
Member

commented Aug 9, 2018

Anyway, not all is lost, I need to refresh my memory of this part of things every few years apparently so uh...now I'm refreshed 😆

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.