Routing Between VMs (Skywire)

[0pcom]

Qubes OS version:

4.0

Affected component(s):

networking / routing between VMs

---

Steps to reproduce the behavior:

I am attempting to replicate the skywire apparatus setup using QubesOS.

1. Create and name a VM

2. Install Dependencies if needed: curl git mercurial make binutils gcc bison screen bash

3. Install GO from source

cd ~
export GOV=1.9
curl -sS https://storage.googleapis.com/golang/go$GOV.linux-amd64.tar.gz > go$GOV.linux-amd64.tar.gz
tar xvf go$GOV.linux-amd64.tar.gz
rm go$GOV.linux-amd64.tar.gz
sudo mv go /usr/local/go
sudo ln -s /usr/local/go/bin/go /usr/local/bin/go
sudo ln -s /usr/local/go/bin/godoc /usr/local/bin/godoc
sudo ln -s /usr/local/go/bin/gofmt /usr/local/bin/gofmt
mkdir -p $HOME/go
mkdir -p $HOME/go/bin
mkdir -p $HOME/go/src
mkdir -p $HOME/go/pkg

3.1) Setup $GOPATH variable by appending the following to .bashrc

export GOROOT=/usr/local/go
export GOPATH=$HOME/go
export GOBIN=$GOPATH/bin
export PATH=$PATH:$GOBIN

3.2) append the following to /etc/profile

# include .bashrc if it exists
if [ -f "$HOME/.bashrc" ]; then
   . "$HOME/.bashrc"
fi

3.3) source ~/.bashrc

4. Test GO and InstallSkywire

go get github.com/skycoin/skycoin/...
cd $GOPATH/src/github.com/skycoin
git clone https://github.com/skycoin/skywire.git
cd $GOPATH/src/github.com/skycoin/skywire/cmd
go install ./...

5. Run Skywire manager & node

cd $GOPATH/bin
nohup ./manager -web-dir ${GOPATH}/src/github.com/skycoin/skywire/static/skywire-manager > /dev/null 2>&1 & echo $! > manager.pid
nohup ./node -connect-manager -manager-address 127.0.0.1:5998 -manager-web 127.0.0.1:8000 > /dev/null 2>&1 & echo $! > node.pid

6. Go to '127.0.0.1:8000' with a web browser and verify that the node appears (may take a minute to show up)

7. REPEAT STEPS 1-4

8. Launch the node and tell it to connect to the IP address of the other VM

cd $GOPATH/bin
./node -connect-manager -manager-address "manager_IP":5998 -manager-web "manager_IP":8000

9. Check 127.0.0.1:8000 with a web browser in the first VM to see if the second node appears

Expected behavior:

Nodes running in other VMs are able to connect to the manager

Actual behavior:

user@skytest-2:~/go/bin$ ./node -connect-manager -manager-address 10.137.0.7:5998 -manager-web 10.137.0.7:8000
DEBU[0000] writeOP &factory.regWithKey{PublicKey:cipher.PubKey{0x3, 0xc4, 0x5a, 0x8, 0xa, 0x5, 0x77, 0x28, 0xd0, 0xe, 0x44, 0xab, 0x58, 0x59, 0xec, 0xfe, 0xca, 0x89, 0x93, 0xf5, 0xf9, 0x8f, 0xb8, 0x86, 0x5f, 0xb2, 0xd8, 0x1c, 0x88, 0x79, 0xf5, 0xbf, 0x4f}, Context:map[string]string(nil), Version:1}  ctxId=1 dir=out type=tcp
DEBU[0000] executing op &factory.regWithKeyResp{Num:[]uint8{0x27, 0x98, 0xee, 0xba, 0x93, 0x25, 0x8f, 0xc8, 0x46, 0xf5, 0x24, 0xb9, 0x98, 0x9f, 0x9b, 0x3e}, Hash:cipher.SHA256{0xad, 0xa, 0x20, 0xe0, 0x3f, 0x27, 0xa, 0x66, 0x0, 0x1c, 0x7f, 0x21, 0x2e, 0xb1, 0x74, 0x0, 0x9, 0xe1, 0x92, 0xb6, 0x8, 0x6e, 0xad, 0xc0, 0xc5, 0x7f, 0x93, 0x58, 0x8c, 0x2c, 0x7f, 0x9}, PublicKey:cipher.PubKey{0x3, 0x26, 0x43, 0x65, 0x13, 0x6a, 0x15, 0x87, 0xf1, 0xdf, 0x42, 0xaa, 0xd0, 0x33, 0x9a, 0x3c, 0x16, 0xd4, 0xff, 0xa6, 0x21, 0xd5, 0xb1, 0x89, 0x2c, 0xea, 0xe3, 0xab, 0x64, 0x64, 0x82, 0xc0, 0xa}, Version:1}  ctxId=1 dir=out type=tcp
DEBU[0000] writeOP &factory.regCheckSig{Sig:cipher.Sig{0x84, 0xc6, 0x61, 0x5e, 0x61, 0x86, 0x68, 0x90, 0xda, 0x3e, 0x3d, 0x5c, 0x11, 0x35, 0x86, 0xce, 0xa5, 0x97, 0xc2, 0xbd, 0xe1, 0x6d, 0x2b, 0xcf, 0xd8, 0xa3, 0xc3, 0x61, 0xb1, 0x72, 0x5, 0xaf, 0x7d, 0xa9, 0x23, 0xcc, 0xc3, 0x7e, 0xe3, 0xf5, 0x9f, 0x6b, 0x28, 0x2e, 0xfb, 0xbb, 0x67, 0x38, 0xaa, 0x94, 0xb9, 0xe, 0x2d, 0x77, 0xeb, 0x63, 0x82, 0xd1, 0x9e, 0x5f, 0xa6, 0x53, 0x99, 0xce, 0x0}, Version:1}  ctxId=1 dir=out type=tcp
DEBU[0000] executed op &factory.regWithKeyResp{Num:[]uint8{0x27, 0x98, 0xee, 0xba, 0x93, 0x25, 0x8f, 0xc8, 0x46, 0xf5, 0x24, 0xb9, 0x98, 0x9f, 0x9b, 0x3e}, Hash:cipher.SHA256{0xad, 0xa, 0x20, 0xe0, 0x3f, 0x27, 0xa, 0x66, 0x0, 0x1c, 0x7f, 0x21, 0x2e, 0xb1, 0x74, 0x0, 0x9, 0xe1, 0x92, 0xb6, 0x8, 0x6e, 0xad, 0xc0, 0xc5, 0x7f, 0x93, 0x58, 0x8c, 0x2c, 0x7f, 0x9}, PublicKey:cipher.PubKey{0x3, 0x26, 0x43, 0x65, 0x13, 0x6a, 0x15, 0x87, 0xf1, 0xdf, 0x42, 0xaa, 0xd0, 0x33, 0x9a, 0x3c, 0x16, 0xd4, 0xff, 0xa6, 0x21, 0xd5, 0xb1, 0x89, 0x2c, 0xea, 0xe3, 0xab, 0x64, 0x64, 0x82, 0xc0, 0xa}, Version:1} err <nil>  ctxId=1 dir=out type=tcp
ERRO[0131] failed to connect addr(18.219.138.23:5999-029aa3b3adc1b3454304696aa8dd22bef3f59af07f75f948383fad365fe29b4053) err dial tcp 18.219.138.23:5999: getsockopt: connection timed out
ERRO[0131] dial tcp 18.219.138.23:5999: getsockopt: connection timed out
DEBU[0131] listen on :5000                              
ERRO[0131] failed to connect Manager addr(10.137.0.7:5998) err dial tcp 10.137.0.7:5998: getsockopt: no route to host
ERRO[0131] dial tcp 10.137.0.7:5998: getsockopt: no route to host
ERRO[0131] Get http://10.137.0.7:8000/getToken: dial tcp 10.137.0.7:8000: getsockopt: no route to host
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x40 pc=0x787361]

General notes:

Even if the manager VM is set as the netVM of the node VMs this does not work, which is weird because the node VM connects to the internet through the manager VM(!)

---

Other Anomalies:

Time can be saved if you clone the system AFTER you have set up GO and downloaded skywire BUT BEFORE installing or running skywire (avoids multiple nodes with the same key)

When I clone the VM at that point, the configuration sometimes does and sometimes does not appear in the clone.

Also, it seems that when I delete a VM and create another one with the same name there are sometimes remnants of the old VM terminal history in the newly created VM.

[marmarek]

See here: https://www.qubes-os.org/doc/firewall/#enabling-networking-between-two-qubes

[marmarek]

See here: https://www.qubes-os.org/doc/firewall/#enabling-networking-between-two-qubes

[0pcom]

Thank you marmarek I will try that

[0pcom]

Thank you marmarek I will try that

[0pcom]

works

[0pcom]

works