FIN is sent before earlier data

[stevenengler]

If an application does a large write and then a shutdown(SHUT_WR), the FIN packet seems to be sent before the data (and even before acknowledging the peer's SYN). For example with the client:

s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.connect(('server', 8080))
s.send(b'1'*1000)
s.shutdown(socket.SHUT_WR)
assert b'' == s.recv(1)
s.close()

Shadow gives the pcap:

We can see that the 1000 bytes are sent after the FIN packet, which leads to bugs.

@robgjansen This might be related to the tgen stream issue, where we can close the socket while there is still data in the send buffer, and it seems like the FIN might be sent out-of-order.

Server code

import asyncio
import socket
import sys

async def handle_client(client, addr):
    loop = asyncio.get_event_loop()

    while True:
        buf = await loop.sock_recv(client, 1024)
        if len(buf) == 0:
            break

    print("Connection closed:", addr)
    client.close()

async def run_server(port):
    server = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
    server.bind(('0.0.0.0', port))
    server.listen(100)
    server.setblocking(False)

    loop = asyncio.get_event_loop()

    while True:
        client, addr = await loop.sock_accept(server)
        print("New connection:", addr)
        loop.create_task(handle_client(client, addr))

asyncio.run(run_server(int(sys.argv[1])))

[stevenengler]

Also worth noting that the sequence numbers seem to be ordered correctly (the data has SEQ=2 and the FIN has SEQ=3), even though they're sent out-of-order.

I'm not sure exactly what's happening within shadow's TCP code. It seems like the server is dropping the FIN, receiving the data, and then the client is never retransmitting the FIN.