Possibly broken broadcast channel's Recv interface

[pdyraga]

The broadcast channel Recv function expects the message handler as a callback:

keep-core/pkg/net/libp2p/channel.go

Line 126 in 3a446af

func (c *channel) Recv(ctx context.Context, handler func(m net.Message)) {

However, in possibly all cases all the handler is doing is writing to a buffered channel. Example:

keep-core/pkg/protocol/state/async_machine.go

Lines 74 to 78 in 3a446af

	recvChan := make(chan net.Message, asyncReceiveBuffer)
	handler := func(msg net.Message) {
	recvChan <- msg
	}
	am.channel.Recv(recvCtx, handler)

The broadcast channel has no idea what is happening with the receiver. If it takes too long to receive a message, the receiver's handler goroutine gets blocked:

keep-core/pkg/net/libp2p/channel.go

Lines 138 to 163 in 3a446af

	go func() {
	for {
	select {
	case <-ctx.Done():
	logger.Debug("context is done; removing message handler")
	c.removeHandler(messageHandler)
	return
	case msg := <-messageHandler.channel:
	// Go language specification says that if one or more of the
	// communications in the select statement can proceed, a single
	// one that will proceed is chosen via a uniform pseudo-random
	// selection.
	// Thus, it can happen this communication is called when ctx is
	// already done. Since we guarantee in the network channel API
	// that handler is not called after ctx is done (client code
	// could e.g. perform come cleanup), we need to double-check
	// the context state here.
	if messageHandler.ctx.Err() != nil {
	continue
	}
	handleWithRetransmissions(msg)
	}
	}
	}()

Instead of using a callback, we could explore using a channel. We would avoid unnecessary hop and the broadcast channel could be dropping messages from a full channel if needed.