Add application-level ACKs for online messages

[JustinDrake]

When sending a message (e.g. a purchase order) over an open websockets connection, is there a way to know if it's been successfully received? As far as I can tell, websockets is used as a frame stream. So it seems possible for an application-level message to be partially sent and for openbazaar-go to not know whether or not it's been successfully received in full.

Should we add an application-level ACK for online messages sent over websockets?

[hoffmabc]

Did we have much of a problem with this in the Python websockets code @cpacia? It certainly reasonable to think this could happen, but I don't recall if we ever did anything about it or saw it happen that often in the previous version.

[JustinDrake]

Thinking slightly more abstractly, go-libp2p is transport agnostic and will support other transports in the future beyond TCP and WebSockets (e.g. WebRTC, UTP?). So we can't rely on transport-level ACKs. This then leads to two possible ways to do ACKs:

1. At the secio level. I don't thing secio has ACKs, but I'm not 100% sure.
2. At the application level

[cpacia]

I believe UTP has an ack. Not sure about webrtc.

[JustinDrake]

It turns out that secio does not have an ACK mechanism. Also, there are several new libp2p transports in the pipeline (including WebRTC, UTP, UDP, SHS, QUIC, I2P). Some of these transports are bound to not have an ACK mechanism (e.g. UDP).

It seems to me that therefore openbazaar-go needs an ACK mechanism at the application-level. What do you think @cpacia?

[cpacia]

I'm not terribly convinced on this one. Do we ever see enabling a transport without reliability built in? The only reason I would see using UDP is for hole punching reasons (though our current setup seems to be working fairly well). And even there UTP seems to be a better option than UDP. I've never worked with webRTC but I've read there's a reliable mode. Adding an additional ACK on top of the transport ACK not only adds protocol complexity but more network overhead.

[JustinDrake]

Do we ever see enabling a transport without reliability built in?

I was under the impression that openbazaar-go was embracing libp2p. (Is that the case?) For sure libp2p will support unreliable transports. So I think the answer is yes, we do see enabling a transport without reliability build in.

protocol complexity

Yeah, proper protocol design comes at a cost. Note that we need to notify the counterparty of errors. If we do not ACK, the counterparty would have to wait (e.g. with a timeout) for a possible error notification. If the success/failure is returned with the ACK the counterparty can confidently move on and update it's state machine.

more network overhead.

Negligible.

[JustinDrake]

Do we ever see enabling a transport without reliability built in?

On second thought openbazaar-go has already enabled a transport without reliability built in.

[cpacia]

Websockets are built on top of TCP. There are ACKs and re-transmissions. If the message isn't successfully delivered an error is raised.

[hoffmabc]

Can't IPFS do uTP?

[JustinDrake]

Websockets are built on top of TCP. There are ACKs and re-transmissions. If the message isn't successfully delivered an error is raised.

I don't think it's that easy, especially in a browser context. There are various Websockets modes, but IPFS uses websocket frame streams. That is, application messages are chunked into frames to form a continuous stream. The browser will give you some generic error eventually, but good luck knowing which frame errored and which message that corresponds to. If you know how to know whether or not an application message was fulled ACKed in JS I'm all ears!

[JustinDrake]

This was discussed today with @cpacia. Doing a simple ACK scheme (no sequence numbers, no retries) is both easy to implement and valuable in the context of browser implementations of OpenBazaar communicating with peers over unreliable transports (WebRTC, and WebSockets) where (despite the underlying reliable TCP) the browser does not communicate which fragments have been received.

[jjeffryes]

I'm re-prioritizing this, as we'll need it for the browser-based buying work.