Packet does not reach network adapter

[shit-lord]

Describe the bug

Similar to #4592

Of this particular client machine(and perhaps several others exhibiting similar symptom), sometimes it can connect to server for an interval, then it fails for another interval of like 10min.

During unconnectable periods, procmon can see client process sending data, but the packets do not appear in wireshark.

I wrote a testing python script that sends 1 udp packet per second in a loop, running alongside the msquic client. It never fails, even when the quic program is unable to send packets.

I wonder how to debug this problem. What can cause packets to disappear between the point seen in procmon(tcpip.sys?) and nic driver?

Affected OS

• Windows
• Linux
• macOS
• Other (specify below)

Additional OS information

No response

MsQuic version

main

Steps taken to reproduce bug

Run msquic client, which attemps to establish a connection with server

Expected behavior

procmon, wireshark, server all being able to see the data

Actual outcome

only procmon sees the sending activity

Additional details

No response

[anrossi]

You can use pktmon to collect drop events to see if TCPIP, or some other filter driver, is dropping the packet before it reaches the NIC.

[shit-lord]

Same thing happened with pktmon.
During working periods, pktmon can capture the packets. During nonworking periods, pktmon can't
procmon is able to see the sends in all cases.
I guess the packets are dropped silently by tcpip.sys, before the earliest pktmon checkpoint, but after procmon's.
Perhaps it is related to the socket options set by msquic?

[nibanks]

If pktmon doesn't show anything, I'd recommend looking at MsQuic logs too to see if things are dropping before it sends.

[shit-lord]

After some testing, the cause is found out to be IP_DONTFRAGMENT.
Sometimes this machine is unable to send udp packets >1200 bytes without fragmentation. Removing setsockopt(IP_DONTFRAGMENT) solves this problem.

[nibanks]

The QUIC protocol requires disabling IP fragmentation.

[shit-lord]

The program is not required to be interoperable with 3rd party programs. I choose msquic as a convenient "reliable udp with encryption", so using a "nonstandard QUIC" shouldn't be an issue.

Also I dont quite understand how this "UDP datagrams MUST NOT be fragmented at the IP layer" affects msquic that is built upon UDP. Shouldnt IP fragmentation be transparent to udp(except performance), hence transparent to any quic implementation?

[nibanks]

IP fragmentation complicates many things and breaks a few as well in QUIC (PMTUD). If you want to fork MsQuic, you can obviously do what you wish with it, but as it stands, fragmentation must stay off in MsQuic.

[shit-lord]

By the way. The error code caused by IP_DONTFRAGMENT is kind of weird.
In the default overlapped send mode, IoResult of CxPlatSendDataComplete is ERROR_INVALID_USER_BUFFER.
However,
SendData->WsaBufferCount == 1
SendData->WsaBuffers[0].len == 1220
IsBadReadPtr(SendData->WsaBuffers[0].buf, SendData->WsaBuffers[0].len) == 0
This is the initial connection establishing packet from ConnectionStart, so it cant be a bad buffer given to StreamSend.

After changing the &SendData->Sqe.Overlapped argument for WSASendMsg to 0, the error code became WSAEMSGSIZE.
But getsockopt(SO_MAX_MSG_SIZE) was about 64KB, then I began to suspect IP_DONTFRAGMENT.

[nibanks]

Closing this since we don't have any logs to investigate. As mentioned, QUIC requires Don't Fragment bits to be set. We'll be happy to reopen if you have more information to go on.