Shadow allows overlapping local addresses

[stevenengler]

On Linux, sockets generally should not have (local addr, remote addr) 4-tuples that are ambiguous (ignoring options like SO_REUSEADDR and SO_REUSEPORT). For example if a listening TCP socket is bound to 127.0.0.1:80, you should not be able to create any TCP sockets that have that same local address (ignoring IP_BIND_ADDRESS_NO_PORT and connections with more-specific 4-tuples such as those created from server.accept()). The confusing rules are somewhat explored in:

• https://blog.cloudflare.com/the-quantum-state-of-a-tcp-port/
• https://blog.cloudflare.com/how-to-stop-running-out-of-ephemeral-ports-and-start-to-love-long-lived-connections/
• https://blog.cloudflare.com/everything-you-ever-wanted-to-know-about-udp-sockets-but-were-afraid-to-ask-part-1/

The rules on Linux are different between TCP and UDP, but Shadow generally doesn't make a distinction between the two.

Since Shadow's socket association table is indexed by the 5-tuple (protocol, local addr, remote addr), to prevent overlapping local addresses Shadow can only check for the existence of some concrete 5-tuple in the table. For example it can check if (tcp, 127.0.0.1:1000, 127.0.0.1:80) exists in the table, or if (tcp, 127.0.0.1:1000, 0.0.0.0:0) exists in the table, but not queries like (tcp, 127.0.0.1:1000, IP:PORT) for any concrete IP and PORT.

Shadow does contain some checks, such as the following in get_random_free_port():

shadow/src/main/host/network/namespace.rs

Lines 227 to 244 in 66b0712

	// `is_addr_in_use` will check all interfaces in the case of INADDR_ANY
	let specific_in_use = self
	.is_addr_in_use(
	protocol_type,
	SocketAddrV4::new(interface_ip, random_port),
	peer,
	)
	.unwrap_or(true);
	let generic_in_use = self
	.is_addr_in_use(
	protocol_type,
	SocketAddrV4::new(interface_ip, random_port),
	SocketAddrV4::new(Ipv4Addr::UNSPECIFIED, 0),
	)
	.unwrap_or(true);
	if !specific_in_use && !generic_in_use {
	return Some(random_port);
	}

Which when deciding if a port is free will check if (proto, local_ip:local_port, remote_ip:remote_port) or (proto, local_ip:local_port, 0.0.0.0:0) already exists. In the case of TCP this prevents binding a socket to the same local IP and port as a listening socket. For example if there exists a listening socket bound to 127.0.0.1:80, it will be in the table as (tcp, 127.0.0.1:80, 0.0.0.0:0). So if connect() is called on a new socket, this check prevents us from choosing port 80 since we'll check both (tcp, 127.0.0.1:80, peer_ip:peer_port) as well as (tcp, 127.0.0.1:80, 0.0.0.0:0), and this second check will fail since that 5-tuple is already in use.

This check doesn't help us in the reverse case though. If we connect() a socket to a peer and then bind() another socket to the same local IP and port, Shadow will allow this even though it's not allowed on Linux. For example on Linux:

>>> import socket
>>>
>>> server = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
>>> server.bind(('127.0.0.1', 8080))
>>> server.listen()
>>>
>>> s1 = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
>>> s1.connect(('127.0.0.1', 8080))
>>>
>>> print("s1 4-tuple:", s1.getsockname(), s1.getpeername())
s1 4-tuple: ('127.0.0.1', 50720) ('127.0.0.1', 8080)
>>>
>>> s2 = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
>>> s2.bind(s1.getsockname())
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
OSError: [Errno 98] Address already in use

Whereas in Shadow:

import socket

server = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
server.bind(('127.0.0.1', 8080))
server.listen()

s1 = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s1.connect(('127.0.0.1', 8080))

print("s1 4-tuple:", s1.getsockname(), s1.getpeername())

s2 = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s2.bind(s1.getsockname())

print("s2 4-tuple:", s2.getsockname(), ('0.0.0.0', 0))

exits successfully with the log:

s1 4-tuple: ('127.0.0.1', 30976) ('127.0.0.1', 8080)
s2 4-tuple: ('127.0.0.1', 30976) ('0.0.0.0', 0)

There are likely other cases that don't work correctly in Shadow, and if we wanted to support options like SO_REUSEADDR, SO_REUSEPORT, and IP_BIND_ADDRESS_NO_PORT it would complicate things further. The behaviour is also different between TCP and UDP, and on Linux these associations are stored in two separate tables: tcp_hashinfo and udp_table.