no eth0 in sys-net // no network traffic gets routed at all to the outside.

[yeoldegrove]

Qubes OS version:

4.0-rc3

Affected TemplateVMs:

sys-net

---

Steps to reproduce the behavior:

Install 4.0-rc3 on my hardware configuration.

[user@sys-net ~]$ lspci | grep -i eth
00:05.0 Ethernet controller: Broadcom Limited NetLink BCM57781 Gigabit Ethernet PCIe (rev 10)

Expected behavior:

There should be a eth0 present in sys-net VM.

Actual behavior:

There is no eth0 present in sys-net VM.

General notes:

The script /usr/lib/qubes/init/network-proxy-setup.sh which is triggered by qubes-networl.service does not find eth0.
Therefore no network traffic gets routed at all to the outside.

tg3 network module is loaded but eth0 still does not appear.

[user@sys-net ~]$ ip ad
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
3: vif3.0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 32
    link/ether fe:ff:ff:ff:ff:ff brd ff:ff:ff:ff:ff:ff
    inet 10.137.0.5/32 scope global vif3.0
       valid_lft forever preferred_lft forever
    inet6 fe80::fcff:ffff:feff:ffff/64 scope link 
       valid_lft forever preferred_lft forever

[user@sys-net ~]$ lspci | grep -i eth
00:05.0 Ethernet controller: Broadcom Limited NetLink BCM57781 Gigabit Ethernet PCIe (rev 10)

[user@sys-net ~]$ sudo bash -x /usr/lib/qubes/init/network-proxy-setup.sh
++ qubesdb-read /qubes-netvm-network
+ network=10.137.0.5
+ '[' x10.137.0.5 '!=' x ']'
+ '[' -e /proc/sys/kernel ']'
+ '[' -e /proc/sys/kernel/modules_disabled ']'
+ readonly modprobe_fail_cmd=false
+ modprobe_fail_cmd=false
++ qubesdb-read /qubes-netvm-gateway
+ gateway=10.137.0.5
++ qubesdb-read /qubes-netvm-primary-dns
+ primary_dns=10.139.1.1
++ qubesdb-read /qubes-netvm-secondary-dns
+ secondary_dns=10.139.1.2
+ modprobe netbk
+ modprobe xen-netback
+ echo NS1=10.139.1.1
+ echo NS2=10.139.1.2
+ /usr/lib/qubes/qubes-setup-dnat-to-ns
+ echo 1
+ /sbin/ethtool -K eth0 sg off
Cannot get device feature names: No such device
+ true

---

Related issues:

[hast0011]

Maybe the same reason is true for me with Release 4.0 rc2:

Ethernet controller: Broadcom Limited NetXtreme BCM5761 Gigabit Ethernet PCIe (rev 10)

In release 3.2 the following helped me out but now it won't work anymore:

[Unit]
Description=Netvm fixup
Before=qubes-netvm.service

[Service]
ExecStart=/bin/sh -c 'echo 0000:04:00.0 > /sys/bus/pci/drivers/pciback/permissive'
Type=oneshot
RemainAfterExit=yes

[Install]
WantedBy=multi-user.target

Then enable it with "systemctl enable qubes-pre-netvm.service".

[hast0011]

Maybe the same reason is true for me with Release 4.0 rc2:

Ethernet controller: Broadcom Limited NetXtreme BCM5761 Gigabit Ethernet PCIe (rev 10)

In release 3.2 the following helped me out but now it won't work anymore:

[Unit]
Description=Netvm fixup
Before=qubes-netvm.service

[Service]
ExecStart=/bin/sh -c 'echo 0000:04:00.0 > /sys/bus/pci/drivers/pciback/permissive'
Type=oneshot
RemainAfterExit=yes

[Install]
WantedBy=multi-user.target

Then enable it with "systemctl enable qubes-pre-netvm.service".

[marmarek]

Check kernel messages (sudo dmesg) - tg3 driver probably put some messages there why it doesn't work

[marmarek]

Check kernel messages (sudo dmesg) - tg3 driver probably put some messages there why it doesn't work

[yeoldegrove]

[user@sys-net ~]$ sudo dmesg |grep tg3
[   12.826961] tg3.c:v3.137 (May 11, 2014)
[   24.912236] tg3 0000:00:05.0: tg3_test_dma: Buffer write failed. err = -19
[   24.912295] tg3 0000:00:05.0: DMA engine test failed, aborting

[yeoldegrove]

[user@sys-net ~]$ sudo dmesg |grep tg3
[   12.826961] tg3.c:v3.137 (May 11, 2014)
[   24.912236] tg3 0000:00:05.0: tg3_test_dma: Buffer write failed. err = -19
[   24.912295] tg3 0000:00:05.0: DMA engine test failed, aborting

[hast0011]

For me the output of dmesg in sys-net was like (I cant remember the exact output):

tg3: Problem fetching invariants of chip, aborting.

[hast0011]

For me the output of dmesg in sys-net was like (I cant remember the exact output):

tg3: Problem fetching invariants of chip, aborting.

[marmarek]

Looks like some problem with accessing that device by the driver. Either DMA or config space. See xl dmesg in dom0 if you have some VT-d related errors. Also check sudo dmesg if you have Driver tried to write to a read-only configuration space field message about this device. If any of those yields anything, try permissive mode: https://www.qubes-os.org/doc/assigning-devices/#pci-passthrough-issues

[marmarek]

Looks like some problem with accessing that device by the driver. Either DMA or config space. See xl dmesg in dom0 if you have some VT-d related errors. Also check sudo dmesg if you have Driver tried to write to a read-only configuration space field message about this device. If any of those yields anything, try permissive mode: https://www.qubes-os.org/doc/assigning-devices/#pci-passthrough-issues

[hast0011]

Locked myself out of he system by having additional pci devices assgned to sys-net. This autostarts and then… I will reinstall and then come back. Maybw tomorrow. Thank you in the meantime Von: Marek Marczykowski-Górecki Gesendet: Freitag, 15. Dezember 2017 20:56 An: QubesOS/qubes-issues Cc: hast0011; Comment Betreff: Re: [QubesOS/qubes-issues] no eth0 in sys-net // no network trafficgets routed at all to the outside. (#3349) Looks like some problem with accessing that device by the driver. Either DMA or config space. See xl dmesg in dom0 if you have some VT-d related errors. Also check sudo dmesg if you have Driver tried to write to a read-only configuration space field message about this device. If any of those yields anything, try permissive mode: https://www.qubes-os.org/doc/assigning-devices/#pci-passthrough-issues — You are receiving this because you commented. Reply to this email directly, view it on GitHub, or mute the thread.

[hast0011]

Locked myself out of he system by having additional pci devices assgned to sys-net. This autostarts and then… I will reinstall and then come back. Maybw tomorrow. Thank you in the meantime Von: Marek Marczykowski-Górecki Gesendet: Freitag, 15. Dezember 2017 20:56 An: QubesOS/qubes-issues Cc: hast0011; Comment Betreff: Re: [QubesOS/qubes-issues] no eth0 in sys-net // no network trafficgets routed at all to the outside. (#3349) Looks like some problem with accessing that device by the driver. Either DMA or config space. See xl dmesg in dom0 if you have some VT-d related errors. Also check sudo dmesg if you have Driver tried to write to a read-only configuration space field message about this device. If any of those yields anything, try permissive mode: https://www.qubes-os.org/doc/assigning-devices/#pci-passthrough-issues — You are receiving this because you commented. Reply to this email directly, view it on GitHub, or mute the thread.

[hast0011]

I found the following output in xl dmesg:
intel vt-d dom0 dma passthrough not enabled

Vt-d is definitly switched on in bios settings. Maybe vt-d is more than just yes or no?
Is that the reason?

[hast0011]

I found the following output in xl dmesg:
intel vt-d dom0 dma passthrough not enabled

Vt-d is definitly switched on in bios settings. Maybe vt-d is more than just yes or no?
Is that the reason?

[marmarek]

This message is ok. If you didn't have VT-d enabled, VM wouldn't start at all.

Maybe vt-d is more than just yes or no?

Well, there are two versions: with and without interrupt remapping, but I believe that any not 3+ years old hardware if have VT-d at all, then it comes with interrupt remapping. You should see a message about it few lines before the one you've found.

So, you haven't found any lines clearly looking like errors (with words like "fail", "error" etc), with [VT-d] or similar tag there?

[marmarek]

This message is ok. If you didn't have VT-d enabled, VM wouldn't start at all.

Maybe vt-d is more than just yes or no?

Well, there are two versions: with and without interrupt remapping, but I believe that any not 3+ years old hardware if have VT-d at all, then it comes with interrupt remapping. You should see a message about it few lines before the one you've found.

So, you haven't found any lines clearly looking like errors (with words like "fail", "error" etc), with [VT-d] or similar tag there?

[hast0011]

Managed to successfully install RC4 and network is up:-) I needed to switch off vt-d and having sys-net in PV Mode. Additionally had to set permissive for the network card to 1 manually.

Two points came up during my struggle:

1. Settings in qubes manager always shows "default(HVM)" instead of "PV".
   However qvm-prefs --get sys-net shows "virt_mode - pv" as expected and it works.
2. There exists description about getting the parameter "class" with ist in fact named "klass"
   so qmv-prefs --get sys-net klass gives AppVM. The article is about win7 installation in https://www.qubes-os.org/doc/hvm/

[hast0011]

Managed to successfully install RC4 and network is up:-) I needed to switch off vt-d and having sys-net in PV Mode. Additionally had to set permissive for the network card to 1 manually.

Two points came up during my struggle:

1. Settings in qubes manager always shows "default(HVM)" instead of "PV".
   However qvm-prefs --get sys-net shows "virt_mode - pv" as expected and it works.
2. There exists description about getting the parameter "class" with ist in fact named "klass"
   so qmv-prefs --get sys-net klass gives AppVM. The article is about win7 installation in https://www.qubes-os.org/doc/hvm/

[marmarek]

Settings in qubes manager always shows "default(HVM)" instead of "PV".

Already tracked here: QubesOS/qubes-issues#3517

There exists description about getting the parameter "class" with ist in fact named "klass"
so qmv-prefs --get sys-net klass gives AppVM. The article is about win7 installation in https://www.qubes-os.org/doc/hvm/

Yes, qvm-prefs list its as klass, but with qvm-create it is --class...

Additionally had to set permissive for the network card to 1 manually.

Did you have any problems with that? It should be possible to set it per device (qvm-pci attach ... -o permissive=true).

What happened if you didn't disabled vt-d?

[marmarek]

Settings in qubes manager always shows "default(HVM)" instead of "PV".

Already tracked here: QubesOS/qubes-issues#3517

There exists description about getting the parameter "class" with ist in fact named "klass"
so qmv-prefs --get sys-net klass gives AppVM. The article is about win7 installation in https://www.qubes-os.org/doc/hvm/

Yes, qvm-prefs list its as klass, but with qvm-create it is --class...

Additionally had to set permissive for the network card to 1 manually.

Did you have any problems with that? It should be possible to set it per device (qvm-pci attach ... -o permissive=true).

What happened if you didn't disabled vt-d?

[hast0011]

With vt-d switched on my computer hangs frequently and I thought vt-d it is also related to the network card not beeing recognized fully. Now I leave it off, maybe unrelated to the network card thing.

[hast0011]

With vt-d switched on my computer hangs frequently and I thought vt-d it is also related to the network card not beeing recognized fully. Now I leave it off, maybe unrelated to the network card thing.