Error while submitting flock job

[jasiedu]

I created the /data2/run directory by hand and launched a flock job from the UI. Here is the stack trace.

[ec2-54-90-73-70.compute-1.amazonaws.com] run: mkdir -p /data2/runs/20150308-152432
[ec2-54-90-73-70.compute-1.amazonaws.com] put: /tmp/tmp_XqrFi -> /data2/runs/20150308-152432/config
ERROR:paramiko.transport:Secsh channel 0 open FAILED: Connection refused: Connect failed
Got exception, reconnecting and retrying call
Traceback (most recent call last):
File "/home/vagrant/flock/ui/sshxmlrpc.py", line 87, in call
result = self._exec_call(args, kwargs)
File "/home/vagrant/flock/ui/sshxmlrpc.py", line 80, in _exec_call
result = xmlrpcclient.getattr(self.method_name)(_args, *_kwargs)
File "/usr/lib/python2.7/xmlrpclib.py", line 1224, in call
return self.*send(self.__name, args)
File "/usr/lib/python2.7/xmlrpclib.py", line 1578, in __request
verbose=self.__verbose
File "/usr/lib/python2.7/xmlrpclib.py", line 1264, in request
return self.single_request(host, handler, request_body, verbose)
File "/usr/lib/python2.7/xmlrpclib.py", line 1292, in single_request
self.send_content(h, request_body)
File "/usr/lib/python2.7/xmlrpclib.py", line 1439, in send_content
connection.endheaders(request_body)
File "/usr/lib/python2.7/httplib.py", line 969, in endheaders
self._send_output(message_body)
File "/usr/lib/python2.7/httplib.py", line 829, in _send_output
self.send(msg)
File "/usr/lib/python2.7/httplib.py", line 791, in send
self.connect()
File "/home/vagrant/flock/ui/sshxmlrpc.py", line 18, in connect
self.sock = self.ssh_transport._open_ssh_channel(dest_address, source_address)
File "/home/vagrant/flock/ui/sshxmlrpc.py", line 47, in _open_ssh_channel
source_address)
File "build/bdist.linux-x86_64/egg/paramiko/transport.py", line 740, in open_channel
raise e
ChannelException: (2, 'Connect failed')
ERROR:paramiko.transport:Secsh channel 0 open FAILED: Connection refused: Connect failed
Disconnecting from ubuntu@ec2-54-90-73-70.compute-1.amazonaws.com... done.
Traceback (most recent call last):
File "remoteExec.py", line 120, in
transfer_config_and_submit(sys.argv[2:])
File "remoteExec.py", line 92, in transfer_config_and_submit
submit_to_wingman(host, key_filename, target_dir, timestamp, remote_config, params)
File "remoteExec.py", line 96, in submit_to_wingman
service.run_submitted(target_dir+"/files", timestamp, remote_config, json.dumps(params))
File "/home/vagrant/flock/ui/sshxmlrpc.py", line 92, in __call
result = self._exec_call(args, kwargs)
File "/home/vagrant/flock/ui/sshxmlrpc.py", line 80, in _exec_call
result = xmlrpcclient.getattr(self.method_name)(_args, *_kwargs)
File "/usr/lib/python2.7/xmlrpclib.py", line 1224, in call
return self.__send(self.__name, args)
File "/usr/lib/python2.7/xmlrpclib.py", line 1578, in __request
verbose=self.__verbose
File "/usr/lib/python2.7/xmlrpclib.py", line 1264, in request
return self.single_request(host, handler, request_body, verbose)
File "/usr/lib/python2.7/xmlrpclib.py", line 1292, in single_request
self.send_content(h, request_body)
File "/usr/lib/python2.7/xmlrpclib.py", line 1439, in send_content
connection.endheaders(request_body)
File "/usr/lib/python2.7/httplib.py", line 969, in endheaders
self._send_output(message_body)
File "/usr/lib/python2.7/httplib.py", line 829, in _send_output
self.send(msg)
File "/usr/lib/python2.7/httplib.py", line 791, in send
self.connect()
File "/home/vagrant/flock/ui/sshxmlrpc.py", line 18, in connect
self.sock = self.ssh_transport._open_ssh_channel(dest_address, source_address)
File "/home/vagrant/flock/ui/sshxmlrpc.py", line 47, in _open_ssh_channel
source_address)
File "build/bdist.linux-x86_64/egg/paramiko/transport.py", line 740, in open_channel
paramiko.ssh_exception.ChannelException: (2, 'Connect failed')

[pgm]

This error sounds like the wingman service is not running on your host.
(It tried to tell the "wingman' service there was a new job but could not
connect.)

Things I would check:

There should be a file: /etc/init/wingman.conf
telling upstart how to start wingman

There should be a log at /var/log/upstart/wingman.log which might say why
it can't start.

It should run a script called /tmp/cluster_scripts/start_wingman.sh
You can try running that file by hand to see if you get an error why it
can't start.

On Sun, Mar 8, 2015 at 11:30 AM, jasiedu notifications@github.com wrote:

I created the /data2/run directory by hand and launched a flock job from
the UI. Here is the stack trace.

[ec2-54-90-73-70.compute-1.amazonaws.com] run: mkdir -p
/data2/runs/20150308-152432
[ec2-54-90-73-70.compute-1.amazonaws.com] put: /tmp/tmp_XqrFi ->
/data2/runs/20150308-152432/config
ERROR:paramiko.transport:Secsh channel 0 open FAILED: Connection refused:
Connect failed
Got exception, reconnecting and retrying call
Traceback (most recent call last):
File "/home/vagrant/flock/ui/sshxmlrpc.py", line 87, in call
result = self.

_exec_call(args, kwargs) File "/home/vagrant/flock/ui/sshxmlrpc.py", line
80, in exec_call result = xmlrpcclient._getattr(self.method_name)(_args,
*_kwargs)
File "/usr/lib/python2.7/xmlrpclib.py", line 1224, in call
return self.

_send(self.__name, args) File "/usr/lib/python2.7/xmlrpclib.py", line
1578, in __request verbose=self.__verbose File
"/usr/lib/python2.7/xmlrpclib.py", line 1264, in request return
self.single_request(host, handler, request_body, verbose) File
"/usr/lib/python2.7/xmlrpclib.py", line 1292, in single_request
self.send_content(h, request_body) File "/usr/lib/python2.7/xmlrpclib.py",
line 1439, in send_content connection.endheaders(request_body) File
"/usr/lib/python2.7/httplib.py", line 969, in endheaders
self._send_output(message_body) File "/usr/lib/python2.7/httplib.py", line
829, in _send_output self.send(msg) File "/usr/lib/python2.7/httplib.py",
line 791, in send self.connect() File
"/home/vagrant/flock/ui/sshxmlrpc.py", line 18, in connect self.sock =
self.ssh_transport._open_ssh_channel(dest_address, source_address) File
"/home/vagrant/flock/ui/sshxmlrpc.py", line 47, in _open_ssh_channel
source_address) File "build/bdist.linux-x86_64/egg/paramiko/transport.py",
line 740, in open_channel raise e ChannelException: (2, 'Connect failed')
ERROR:paramiko.transport:Secsh channel 0 open FAILED: Connection refused:
Connect failed Disconnecting from
ubuntu@ec2-54-90-73-70.compute-1.amazonaws.com
ubuntu@ec2-54-90-73-70.compute-1.amazonaws.com... done. Traceback (most
recent call last): File "remoteExec.py", line 120, in
transfer_config_and_submit(_sys.argv[2:]) File "remoteExec.py", line 92, in
transfer_config_and_submit submit_to_wingman(host, key_filename,
target_dir, timestamp, remote_config, params) File "remoteExec.py", line
96, in submit_to_wingman service.run_submitted(target_dir+"/files",
timestamp, remote_config, json.dumps(params)) File
"/home/vagrant/flock/ui/sshxmlrpc.py", line 92, in __call*
result = self.

_exec_call(args, kwargs) File "/home/vagrant/flock/ui/sshxmlrpc.py", line
80, in exec_call result = xmlrpcclient._getattr(self.method_name)(_args,
*_kwargs)
File "/usr/lib/python2.7/xmlrpclib.py", line 1224, in call
return self.__send(self.__name, args)
File "/usr/lib/python2.7/xmlrpclib.py", line 1578, in __request
verbose=self.__verbose
File "/usr/lib/python2.7/xmlrpclib.py", line 1264, in request
return self.single_request(host, handler, request_body, verbose)
File "/usr/lib/python2.7/xmlrpclib.py", line 1292, in single_request
self.send_content(h, request_body)
File "/usr/lib/python2.7/xmlrpclib.py", line 1439, in send_content
connection.endheaders(request_body)
File "/usr/lib/python2.7/httplib.py", line 969, in endheaders
self._send_output(message_body)
File "/usr/lib/python2.7/httplib.py", line 829, in _send_output
self.send(msg)
File "/usr/lib/python2.7/httplib.py", line 791, in send
self.connect()
File "/home/vagrant/flock/ui/sshxmlrpc.py", line 18, in connect
self.sock = self.ssh_transport._open_ssh_channel(dest_address,
source_address)
File "/home/vagrant/flock/ui/sshxmlrpc.py", line 47, in _open_ssh_channel
source_address)
File "build/bdist.linux-x86_64/egg/paramiko/transport.py", line 740, in
open_channel
paramiko.ssh_exception.ChannelException: (2, 'Connect failed')

—
Reply to this email directly or view it on GitHub
#5.