Issues in SSH communication between two cuquantum containers

[intelligi123]

Hi,

Question related to this and #36 and #37 (initiated by me but couldn't continue).

I am trying to execute a quantum circuit on two nodes having cuquantum (23.10) installed for which I need SSH communication.
I installed cuquantum on both master and worker nodes.

1. master node -> host : "master" hostname : "192.168.0.30"
2. worker node-> host : "worker_1" hostname: "192.168.0.29"

Currently working on master node:

sudo docker run -it --detach --name master --gpus all --network=host --uts=host --ipc=host nvcr.io/nvidia/cuquantum-appliance:23.10

Following mentioned links, I created config and hostfile files

This is my config file for master node which contains aliases of worker_1 node

Host worker_1
     Hostname 192.168.0.29
     Port 22
     IdentityFile ~/.ssh
     StrictHostKeyChecking no
     UserKnownHostsFile=/dev/null 

Here is hostfile content, not sure need to add ip address in it but as in openmpi I use to add.
worker_1 192.168.0.29

I am executing rest of the commands as it is

mkdir -p ./.sshc && ssh-keygen -t rsa -C '' -N '' -f ./.sshc/id_rsa && cat ./config > ./.sshc/config && cat ./hostfile > ./.sshc/hostfile 
docker cp ./sshc master:/home/cuquantm

and then

sudo docker exec -i master bash -c 'rm -rf /home/cuquantum/.ssh'  
sudo docker exec -i master bash -c 'mv /home/cuquantum/.sshc /home/cuquantum/.ssh'
sudo docker exec -i master bash -c 'chown -R cuquantum /home/cuquantum/.ssh' 
sudo docker exec -i master bash -c 'chmod 700 /home/cuquantum/.ssh'  
sudo docker exec -i master bash -c 'chmod -R 600 /home/cuquantum/.ssh'   # stuck here

I am getting access error even cuquantum user is owner of .ssh directory.

chmod: cannot access '/home/cuquantum/.ssh/id_rsa.pub': Permission denied
chmod: cannot access '/home/cuquantum/.ssh/id_rsa': Permission denied
chmod: cannot access '/home/cuquantum/.ssh/hostfile': Permission denied
chmod: cannot access '/home/cuquantum/.ssh/config': Permission denied

I open docker terminal and found permissions:

drwx------ 2 cuquantum cuquantum 4096 Apr 22 06:39 .
drwxr-x--- 1 cuquantum cuquantum 4096 Apr 22 08:00 ..
-rw------- 1 cuquantum cuquantum  143 Apr 22 06:39 config
-rw------- 1 cuquantum cuquantum   22 Apr 22 06:39 hostfile
-rw------- 1 cuquantum cuquantum 2578 Apr 22 06:39 id_rsa
-rw------- 1 cuquantum cuquantum  554 Apr 22 06:39 id_rsa.pub

permissions are same 600. I however continued, there are some errors in sed commands mentioned in #37 answer

sudo docker exec -i master bash -c \"sed -i '/#PermitRootLogin/c\PermitRootLogin yes\' /etc/ssh/sshd_config\"
-i: -c: line 1: unexpected EOF while looking for matching `"'
-i: -c: line 2: syntax error: unexpected end of file

I made changes with interactive session in sshd_config file and additionally uncommented HostKey configuration. Following changes have been done here

HostKey /home/cuquantum/.ssh/id_rsa
Port 22
PermitRootLogin yes

Now at the last step of ssh configuration, when I execute restart command, it gives me following error:

 * Starting OpenBSD Secure Shell server sshd
start-stop-daemon: unable to set gid to 0 (Operation not permitted)
   ...fail!

I tried to start the service with sudo interactively, it didn't throw error but service still doesn't start.

sudo docker exec -it master bash
sudo service ssh start
[sudo] password for cuquantum: 
 * Starting OpenBSD Secure Shell server sshd                                                                             [ OK ] 
service ssh status
 * sshd is not running

Any suggestions?

[intelligi123]

The links I was using had some minor mistakes regarding permission settings:

I corrected .ssh directory permission with this line

sudo docker exec -i master bash -c 'chmod -R 600 /home/cuquantum/.ssh/*'

In ./config file, made following changes:

*no underscores in name
*port different than 22 to avoid conflict with host ssh
*path of private key in IdentityFile

Host worker
     Hostname 192.168.0.29
     Port 2223
     IdentityFile ~/.ssh/id_rsa
     StrictHostKeyChecking no
     UserKnownHostsFile=/dev/null 

To start ssh, I use

/usr/sbin/sshd

Strange thing is when I check status of sshd its still say not running but ssh connection works fine between both dockers.