SSH Config Issues

[kwrobert]

GC3Pie makes a confusing assumption when the "frontend" entry is missing from the gc3pie.conf file.

If there is no “frontend” entry in the configuration section for a resource, gc3pie assumes the hostname is ‘localhost’. I think this makes the error message a bit confusing. Here is an example configuration:

[auth/workstations]
type=ssh
username=a_person

[resource/dell]
enabled=yes
override=true
auth=workstations
type=shellcmd
transport=ssh
max_cores=4
max_cores_per_job=4
ssh_config=~/.ssh/config
architecture=x86_64
max_memory_per_core=4 GB
max_walltime=48 hours
override=true

If I try running `gservers -vv’ with this config I get the following traceback:

No handlers could be found for logger "gc3.gc3libs"
gservers: [2017-03-31 08:57:25] INFO : Starting gservers at Fri Mar 31 08:57:25 2017; invoked as '/Users/Kyle/gc3pie/bin/gservers -vv'
gservers: [2017-03-31 08:57:25] DEBUG : Configuration.load(): File '/etc/gc3/gc3pie.conf' does not exist, ignoring.
gservers: [2017-03-31 08:57:25] DEBUG : Configuration.load(): File '/Users/Kyle/gc3pie/etc/gc3/gc3pie.conf' does not exist, ignoring.
gservers: [2017-03-31 08:57:25] DEBUG : Configuration.merge_file(): Reading file '/Users/Kyle/.gc3/gc3pie.conf' ...
gservers: [2017-03-31 08:57:25] DEBUG : Config._parse(): Read configuration stanza for auth 'none'
gservers: [2017-03-31 08:57:25] DEBUG : Config._parse(): Read configuration stanza for auth 'workstations'
gservers: [2017-03-31 08:57:25] DEBUG : Config._parse(): Read configuration stanza for resource 'localhost'.
gservers: [2017-03-31 08:57:25] DEBUG : Config._parse(): Resource 'localhost' defined by: architecture=set(['x86_64']), auth='none', enabled=True, max_cores=4, max_cores_per_job=4, max_memory_per_core=Memory(4, unit=GB), max_walltime=Duration(4, unit=hour), name='localhost', override='true', time_cmd='/usr/local/bin/time', transport='local', type='shellcmd'.
gservers: [2017-03-31 08:57:25] DEBUG : Config._parse(): Read configuration stanza for resource 'dell'.
gservers: [2017-03-31 08:57:25] DEBUG : Config._parse(): Resource 'dell' defined by: architecture=set(['x86_64']), auth='workstations', enabled=True, max_cores=4, max_cores_per_job=4, max_memory_per_core=Memory(4, unit=GB), max_walltime=Duration(48, unit=hour), name='dell', override='true', ssh_config='/.ssh/config', transport='ssh', type='shellcmd'.
gservers: [2017-03-31 08:57:25] DEBUG : Creating resource 'dell' defined by: architecture=set(['x86_64']), auth='workstations', enabled=True, max_cores=4, max_cores_per_job=4, max_memory_per_core=Memory(4, unit=GB), max_walltime=Duration(48, unit=hour), name='dell', override='true', ssh_config='/.ssh/config', transport='ssh', type='shellcmd'.
gservers: [2017-03-31 08:57:25] DEBUG : Using class <class 'gc3libs.backends.shellcmd.ShellcmdLrms'> from module <module 'gc3libs.backends.shellcmd' from '/Users/Kyle/gc3pie/src/gc3libs/backends/shellcmd.pyc'> to instanciate resources of type shellcmd
gservers: [2017-03-31 08:57:25] INFO : Computational resource 'dell' initialized successfully.
gservers: [2017-03-31 08:57:25] DEBUG : Creating resource 'localhost' defined by: architecture=set(['x86_64']), auth='none', enabled=True, max_cores=4, max_cores_per_job=4, max_memory_per_core=Memory(4, unit=GB), max_walltime=Duration(4, unit=hour), name='localhost', override='true', time_cmd='/usr/local/bin/time', transport='local', type='shellcmd'.
gservers: [2017-03-31 08:57:25] INFO : Computational resource 'localhost' initialized successfully.
gservers: [2017-03-31 08:57:25] DEBUG : Opening SshTransport...
gservers: [2017-03-31 08:57:25] DEBUG : Using no ProxyCommand for SSH connections.
gservers: [2017-03-31 08:57:25] DEBUG : Connecting to host 'localhost' (port 22) as user 'None' via SSH (timeout 30s)...
gservers: [2017-03-31 08:57:25] ERROR : Could not create ssh connection to localhost: AuthenticationException: Authentication failed.
gservers: [2017-03-31 08:57:25] ERROR : ssh-agent is running but no key has been added. Please add a key with ssh-add command.
gservers: [2017-03-31 08:57:25] ERROR : Ignoring error updating resource 'dell': Failed connecting to remote host 'localhost': Authentication failed..
gservers: [2017-03-31 08:57:25] DEBUG : Got error 'TransportError' in updating resource 'dell'; printing full traceback.
Traceback (most recent call last):
File "/Users/Kyle/gc3pie/src/gc3libs/core.py", line 831, in update_resources
lrms.get_resource_status()
File "/Users/Kyle/gc3pie/src/gc3libs/backends/shellcmd.py", line 700, in get_resource_status
self._gather_machine_specs()
File "/Users/Kyle/gc3pie/src/gc3libs/backends/shellcmd.py", line 499, in _gather_machine_specs
self.transport.connect()
File "/Users/Kyle/gc3pie/src/gc3libs/backends/transport.py", line 627, in connect
.format(hostname=self.remote_frontend, msg=ex))
TransportError: Failed connecting to remote host 'localhost': Authentication failed.

So its attempting to connect the the resource dell, but using the hostname localhost. I think assuming the hostname ‘localhost’ when ‘frontend’ is not present and trying to connect via ssh using that hostname seems odd. At the very least, if you think ‘frontend’ should be a requirement for ssh transports, gc3pie should throw an error saying so explicitly when it is not present. My opinion on the matter is as follows:

1. If the “frontend” entry is missing in a resource section of gc3pie.conf where transport=ssh, attempt to look up the name of the resource in ~/.ssh/config (in the traceback above the name would be dell) and use all information present there. 
2. If the resource doesn’t exist in ~/.ssh/config or there is insufficient information there to establish a connection, throw an error saying as such.

Another interesting issue, if I remove the ‘username’ entry in the ‘auth’ section that some ssh connection refers to. If the config looks like this:

[auth/workstations]
type=ssh

[resource/dell]
enabled=yes
override=true
auth=workstations
frontend=dell
type=shellcmd
transport=ssh
max_cores=4
max_cores_per_job=4
ssh_config=~/.ssh/config
architecture=x86_64
max_memory_per_core=4 GB
max_walltime=48 hours
override=true

I get the following traceback:

gservers: [2017-03-31 09:06:36] INFO : Starting gservers at Fri Mar 31 09:06:36 2017; invoked as '/Users/Kyle/gc3pie/bin/gservers -vv'
gservers: [2017-03-31 09:06:36] DEBUG : Configuration.load(): File '/etc/gc3/gc3pie.conf' does not exist, ignoring.
gservers: [2017-03-31 09:06:36] DEBUG : Configuration.load(): File '/Users/Kyle/gc3pie/etc/gc3/gc3pie.conf' does not exist, ignoring.
gservers: [2017-03-31 09:06:36] DEBUG : Configuration.merge_file(): Reading file '/Users/Kyle/.gc3/gc3pie.conf' ...
gservers: [2017-03-31 09:06:36] DEBUG : Config._parse(): Read configuration stanza for auth 'none'
gservers: [2017-03-31 09:06:36] DEBUG : Config._parse(): Read configuration stanza for auth 'workstations'
gservers: [2017-03-31 09:06:36] DEBUG : Config._parse(): Read configuration stanza for resource 'localhost'.
gservers: [2017-03-31 09:06:36] DEBUG : Config._parse(): Resource 'localhost' defined by: architecture=set(['x86_64']), auth='none', enabled=True, max_cores=4, max_cores_per_job=4, max_memory_per_core=Memory(4, unit=GB), max_walltime=Duration(4, unit=hour), name='localhost', override='true', time_cmd='/usr/local/bin/time', transport='local', type='shellcmd'.
gservers: [2017-03-31 09:06:36] DEBUG : Config._parse(): Read configuration stanza for resource 'dell'.
gservers: [2017-03-31 09:06:36] DEBUG : Config._parse(): Resource 'dell' defined by: architecture=set(['x86_64']), auth='workstations', enabled=True, frontend='dell', max_cores=4, max_cores_per_job=4, max_memory_per_core=Memory(4, unit=GB), max_walltime=Duration(48, unit=hour), name='dell', override='true', ssh_config='/.ssh/config', transport='ssh', type='shellcmd'.
gservers: [2017-03-31 09:06:36] DEBUG : Creating resource 'dell' defined by: architecture=set(['x86_64']), auth='workstations', enabled=True, frontend='dell', max_cores=4, max_cores_per_job=4, max_memory_per_core=Memory(4, unit=GB), max_walltime=Duration(48, unit=hour), name='dell', override='true', ssh_config='/.ssh/config', transport='ssh', type='shellcmd'.
gservers: [2017-03-31 09:06:36] DEBUG : Using class <class 'gc3libs.backends.shellcmd.ShellcmdLrms'> from module <module 'gc3libs.backends.shellcmd' from '/Users/Kyle/gc3pie/src/gc3libs/backends/shellcmd.pyc'> to instanciate resources of type shellcmd
gservers: [2017-03-31 09:06:36] INFO : Computational resource 'dell' initialized successfully.
gservers: [2017-03-31 09:06:36] ERROR : Could not create resource 'dell': init() takes at least 3 arguments (2 given). Configuration file problem?
gservers: [2017-03-31 09:06:36] WARNING : Failed creating backend for resource 'dell' of type 'shellcmd': TypeError: init() takes at least 3 arguments (2 given)
Traceback (most recent call last):
File "/Users/Kyle/gc3pie/src/gc3libs/config.py", line 647, in make_resources
backend = self._make_resource(resdict)
File "/Users/Kyle/gc3pie/src/gc3libs/config.py", line 759, in _make_resource
return cls(**dict(resdict))
File "/Users/Kyle/gc3pie/src/gc3libs/backends/shellcmd.py", line 329, in init
auth = self._auth_fn()
File "/Users/Kyle/gc3pie/src/gc3libs/config.py", line 573, in
return (lambda **extra_args: self.auth_factory.get(name, **extra_args))
File "/Users/Kyle/gc3pie/src/gc3libs/authentication/init.py", line 109, in get
a = self._ctorsauth_name
TypeError: init() takes at least 3 arguments (2 given)

Again, totally reasonable to make username a requirement in the auth section. But, this is a piece of information that might also be present in ~/.ssh/config. It seems reasonable to at least attempt looking there for a username before failing.