Sending on pipe rings

[larseggert]

I am trying to implement the suggestion by @vmaffione in #322 (comment), to use pipes instead of loopback.

Is there a difference between sending on pipe rings vs. regular interface rings?

It seems that when I place a buffer into a slot and call NIOCTXSYNC, that buffer is gone from the slot after the tail has moved forward?

[vmaffione]

Transmission on VALE ports and pipes is synchronous, which means that transmission happens in the context of the NIOCTXSYNC call. So on return from the ioctl call tail moves forward of N units, where N is the number of packets that were transmitted. On the contrary, physical netmap ports are asynchronous, so the NIOCTXSYNC just pushes the slots to the hardware rings. Transmissions are done in parallel by the hardware and completion is signaled by interrupts.

[vmaffione]

Sorry, I think I misunderstood your question. You are right, transmission within pipes is implemented through zerocopy, so buffers in TX rings are swapped with the ones on the other side. The current netmap API does not guarantee that the TX slots that you submit for transmission are still there after transmission is complete (synchronously or asynchronously). In practice buffers will stay there for all netmap ports except for pipes, and provided zerocopy monitors are not used on your netmap ports.

For your tests you can use VALE ports, which are not zerocopy (and are also non-blocking) on transmission.

If there is any interest we could extend the netmap API so that the user can specify it doesn't want zerocopy to happen under the hood for a certain netmap port (e.g. NS_BUF_CHANGED can't be accepted).
We could also extend pipes with a "copy" version.

[larseggert]

Right. On interface rings, I can retrieve the sent data from the TX ring afterwards. On pipe rings, there seems to be some other buffer in the slot after TX?

[vmaffione]

Yes, the buffer that you find after TX is the one that used to be in the same position in the RX ring of the pipe on the other end.

[larseggert]

OK, so that's a pretty distinct difference to how interface rings work. So I need to copy if I want to retain the data at the sender?

[vmaffione]

Yes, you need to copy somewhere else.

[larseggert]

OK, will do. You probably want to mention this somewhere in the documentation, since it's a pretty severe restriction compared to normal NIC rings IMO.

[larseggert]

Reopening this, since there seems to have been a change in netmap at some point that breaks my code. NIOCTXSYNC on a pipe seems to no longer advance the tail?

[giuseppelettieri]

Now the tail is advanced asynchronously, when the reader on the other end of the pipe has actually released the slot. This avoids the slot-swapping that the previous pipe implementation was doing and is needed for the correct implementation of a feature in lb. I hope that the semantics of your code can be adapted to this, since it should be the same as sending on an hardware port.

[larseggert]

Thanks, I will try that!

[larseggert]

It doesn't quite seem to work. The sending side places buffer 1 into a TX slot and calls NIOCTXSYNC. The receiving side swaps buffer 1 out of the RX ring, does some modifications, places it back into the TX ring, and calls NIOCTXSYNC. But on the original sender, the tail never moves.

I guess sending doesn't count as "releasing"? What does?

[larseggert]

OK, I have a workaround. I copy the data on RX, leaving the original buffer in the ring, and then call NIOCRXSYNC which (I guess) releases it. It's an extra memcpy and ioctl compared to operating on NICs, but that's OK for now