remote execution: bash: guild: command not found

[KimBue]

I try to train on remote via ssh. guildai is installed in both local and remote environments (venv). Remote environment is specified in the config.yml
On guild run train --remote remote-gpu I get the following error:

...
Starting train on remote-gpu as 6f95...
bash: guild: command not found

Same error occures when executing other commands on remote, e.g. guild check --remote remote-gpu, but when logged in on remote I can do guild check without issue.
I tried to follow the documentation. Any idea what the issue could be?

[gar1t]

Are you able to run ssh <remote host> guild check?

The issue here is likely the different env that is setup when you run ssh over a tty connection vs not. This is related to how the env is setup when you login. This post explains the issue to some extent.

[KimBue]

ssh <remote host> guild check results in the same error. I will investigate further...

[gar1t]

I would typically look to add this to the end of ~/.bash_profile on the remote system:

# ~/.bash_profile ...

if [ -f ~/.bashrc ]; then
	. ~/.bashrc
fi

This ensures that the env used for non-interactive ssh sessions is initialized using the same script that's used for interactive (tty present).

[KimBue]

unfortunately this does not resolve the problem.

[gar1t]

You can compare the two PATH environment variables based on how you connect via ssh.

Show the path used for non-interactive commands (this is what Guild will use with remote commands):

$ ssh <host> 'echo $PATH'  # be sure to use single quotes here

Show the path used for interactive commands:

$ ssh <host>
<host> $ echo $PATH

You want to make sure that Guild's location in the first path. You can either alter that path to be the same as the second (this is the tactic of sourcing ~/.bashrc above. Or you can install Guild somewhere in the existing path.

The file that's sourced for non-iteractive shells in your case might be ~/.profile or ~/.zprofile. This thread answer does a better job explaining the situation than the other link I give above.

[gar1t]

I just went through this issue myself on a system. For Ubuntu based distros, ~/.bashrc is used for both non-interactive and interactive ssh sessions. In my cases there's a check for interactive session with an early exit - along this line:

# If not running interactively, don't do anything                                                   
case $- in                                                                                          
    *i*) ;;                                                                                         
      *) return;;                                                                                   
esac 

If your init script has something like this, make sure you set PATH before that section is run.