Add --debug option

[martjushev]

Feature request

clearml-session CLI should have a --debug argument, which would increase verbosity of logging.

Motivation:

There is a workstation behind NAT which has clearml-agent running. Invoking clearml-session on another machine fails:

Remote machine is ready
Setting up connection to remote session
Starting SSH tunnel
SSH tunneling failed, retrying in 3 seconds

it would be beneficial to understand why exactly tunneling fails. Currently there is no logging for that

clearml-session/clearml_session/__main__.py

Lines 563 to 565 in bf1851c

	except Exception as ex:
	child.terminate(force=True)
	child = None

Probably there are also other parts of code where extra debug logging could be handy

[bmartinn]

Thanks @martjushev , I think you have a valid point here, and we should definitely add the ability to better debug the SSH connection.
Now we just need to figure out if we can get the pexpect to dump the console stdout it processed for us...
Let me check if we can somehow access this data

[bmartinn]

@martjushev just updating, --debug will now print all conda outputs during the venv setup :)

[norrishd]

@martjushev did you find a solution to this specific error message? I'm also getting it, and am a bit stuck as the relevant port is open, and the logs look normal on the agent side.

[martjushev]

@martjushev did you find a solution to this specific error message? I'm also getting it, and am a bit stuck as the relevant port is open, and the logs look normal on the agent side.

No solution, only a workaround: I have set up my own profile in ~/.ssh/config for the server which runs clearml-agent. I run clearml-session until the point where it unsuccessfully tries to SSH into the server. I then Ctrl+C this process (which doesn't kill the session on the remote server) and then open my own SSH tunnel

[bmartinn]

Hi @martjushev
Is this dues to having to use SSH keys to login into the container, or is it due to the double tunneling ?
What would we need to add to the session to do the tunneling for you?
(Basically I'm assuming you tunnel into the host machine, then you SSH into the container, is this correct?)

[martjushev]

I'm not trying to SSH into the container, only into the host machine (which is enough to give me access to jupyter server). SSH connection to host machine initiated by clearml-session succeeds if I'm physically located in the same network as the host machine. Otherwise, I use ngrok to access the host machine. I assume that if I would be able to customize the args to ssh executable, then it would work (I would add the address + port provided by ngrok and IdentityFile + UseKeychain )

clearml-session/clearml_session/__main__.py

Line 518 in bf1851c

args = ['-N', '-C',

[bmartinn]

SSH connection to host machine initiated by clearml-session succeeds if I'm physically located in the same network as the host machine

Yep I think this is the main issue,

Otherwise, I use ngrok to access the host machine.

ngrok basically does reverse proxy for you...

I assume that if I would be able to customize the args to ssh executable, then it would work (I would add the address + port provided by ngrok and IdentityFile + UseKeychain )

That is a great idea! what did you have in mind ?

[qwaxys]

Closing this issue for now, the extra arguments will be added to the documentation. If you still have problems related to SSH or other things, feel free to open another issue and link to this one.