Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Bridged Networking or User mode? Replace ebtables & iptables with iptables-nft to allow bridged networking. #72

Open
sickcodes opened this issue Jul 25, 2020 · 13 comments
Labels
documentation Improvements or additions to documentation enhancement New feature or request

Comments

@sickcodes
Copy link
Owner

sickcodes commented Jul 25, 2020

Here are the two types of networking


# usermode
-netdev user,id=net0,hostfwd=tcp::10022-:22,
-device e1000-82545em,netdev=net0,id=net0,mac=52:54:00:09:49:17

# bridged
-netdev tap,id=net0,ifname=tap0,script=no,downscript=no
-device vmxnet3,netdev=net0,id=net0,mac=52:54:00:AB:F8:B7

user mode

The image has the same IP as your Docker IP ip n to see neighbors
QEMU cannot be reached unless you add ports in the QEMU args during "docker run"
Can't expose Docker ports on a running container (AFAIK), you have to start a new image and move the installation around.
You also have to edit the Launch.sh or add in arguments

bridged mode

The image will run on the container's own internal network, like 10.0.2.15 or something or 192.168.122.11x

After that, you can ssh OR docker exec into the Docker and then ssh into the QEMU image.

which is better?

Realistically, they both appear exactly the same.
Bridge mode would let people forward all connections to the bridge IP.

# usermode
# you have to add
docker run ... -p 50922:10022 -e EXTRA='-netdev user,hostfwd=tcp::10022-:22'

# bridge-mode
docker run ... -p 50922:10022 -ip p.ubl.ic.ip 

If anyone wants to test it out:

sudo tee -a /etc/sysctl.conf <<< 'net.ipv4.ip_forward=1'
sudo sysctl -p
sudo pacman -S iptables-nft

sudo systemctl enable --now libvirtd.service
sudo systemctl enable --now virtlogd.service

sudo ip tuntap add dev tap0 mode tap
sudo ip link set tap0 up promisc on 

sudo virsh net-autostart default
sudo virsh net-start default

sudo ip link set dev virbr0 up

sudo ip link set dev tap0 master virbr0


# use the  networking from above in the QEMU args


The actual bug is ebtalbes + dnsmasq doesn't work virsh networking.
You can test it out:

sudo pacman -S libvirtd
virsh net-info --domain default
# works

Bugged:

sudo pacman -S ebtables dnsmasq

virsh net-info --domain default
# fails 

Fix:

sudo pacman -S iptables-nft

virsh net-info --domain default
# works

It's a strange bug, I though related to polkit, however, I found it incompatible with ebtables.

Possibly just with Arch, that the default network doesn't get installed.

Easy fix: replace iptables and ebtables with iptables-nft. If not, virsh hangs, sudo virsh hangs.

On the other hand, QEMU and Docker networking is really, really easy to debug. If bridged network is added, you'd have to start the bridge every time, I also think you need --net host

@sickcodes sickcodes added bug Something isn't working enhancement New feature or request labels Jul 25, 2020
@Julioevm
Copy link
Contributor

Julioevm commented Aug 4, 2020

I was having issues with other machines communicating with a program running inside the macos docker, this might have something to do with it. If I get some free time ill try some of this stuff.

@sickcodes
Copy link
Owner Author

sickcodes commented Aug 4, 2020

I was having issues with other machines communicating with a program running inside the macos docker, this might have something to do with it. If I get some free time ill try some of this stuff.

If you use the two -net command strings above in the xml file while using virt-manager you can switch between bridged and userMode networking. Haven't done bridged in the dockerfile but it wouldn't be much more than whats already here.

In summary:

Use virt-manager without docker
Load the xml file from OSX-KVM

Try the networking commands above

Otherwise you can just add more ports to the QEMU command using the -e EXTRA env variable and also forwarding it again as a normal -p 111:2222 to the docker command.

@sickcodes sickcodes added documentation Improvements or additions to documentation and removed bug Something isn't working labels Sep 29, 2020
@sickcodes
Copy link
Owner Author

sickcodes commented Jan 8, 2021

Replaced iptables with iptables-nft

8d6d035

@sickcodes sickcodes reopened this Feb 8, 2021
@sickcodes
Copy link
Owner Author

sickcodes commented Feb 8, 2021

e1000-82545em is preventing iMessage from working on Catalina and above

https://forums.unraid.net/topic/84288-catalina-network-bridging-weirdness/?do=findComment&comment=781612

However, a fix is available here: https://github.com/chris1111/AppleIntelE1000e

@mikob
Copy link
Contributor

mikob commented Mar 2, 2021

Couldn't get this working. Trying to have a port shared so that the host can access a server running on the mac on port 8080.

-e EXTRA='-netdev user,hostfwd=tcp::8080-:8080'
This produces: qemu-system-x86_64: Parameter 'id' is missing

-e EXTRA='-netdev user,id=net0,hostfwd=tcp::8080-:8080'
This produces: Duplicate ID 'net0' for netdev

-p 8080:8080 -e EXTRA='-netdev user,id=net1,hostfwd=tcp::8080-:8080'
(qemu) qemu-system-x86_64: warning: netdev net1 has no peer
let's the container run... but can't access the port on the host :/

@sickcodes
Copy link
Owner Author

sickcodes commented Mar 2, 2021

I switched master to vmxnet3 in #160 as it did not affect iMessage and it massively increases speed.

@mikob
Copy link
Contributor

mikob commented Mar 2, 2021

@sickcodes Just unsure how to get port forwarding working. I suppose worst case scenario I can just forward via an ssh tunnel.

@sickcodes
Copy link
Owner Author

sickcodes commented Mar 2, 2021

Maybe we can add another ${ADDITIONAL_PORTS} at the end of the line that has 5900 in it too for any other networking arguments for that interface

@sickcodes sickcodes reopened this Mar 2, 2021
@sickcodes
Copy link
Owner Author

sickcodes commented Mar 2, 2021

At the end of this line: https://github.com/sickcodes/Docker-OSX/blob/master/Dockerfile#L228

I'll add it in shortly

@sickcodes
Copy link
Owner Author

sickcodes commented Mar 2, 2021

@mikob Added #162 let me test and then merge

@mikob
Copy link
Contributor

mikob commented Mar 3, 2021

@sickcodes just tested it out, it works! Thanks for your really speedy response!

@sickcodes
Copy link
Owner Author

sickcodes commented Mar 3, 2021

@mikob No problem!

Here is a worked example for anyone else reading, as written here: #162 (comment)

On the host

docker run -it \
    --device /dev/kvm \
    -p 50922:10022 \
    -e ADDITIONAL_PORTS='hostfwd=tcp::10023-:80,' \
    -p 10023:10023 \
    sickcodes/docker-osx:auto

Inside the container:

/bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)"

brew install nginx
sudo sed -i -e 's/8080/80/' /usr/local/etc/nginx/nginx.conf
# sudo nginx -s stop
sudo nginx

nginx should appear on the host at port 10023

@mikob
Copy link
Contributor

mikob commented Mar 3, 2021

P.S. you don't need to install nginx for a test server, you can just do eg. python3 -m http.server 8080 to spin up a static test server that serves the CWD and it ships with python by default, which ships with osx by default.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
documentation Improvements or additions to documentation enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

3 participants