Skip to content

Latest commit

 

History

History
422 lines (311 loc) · 17.4 KB

py-send.rst

File metadata and controls

422 lines (311 loc) · 17.4 KB

Sending

Unlike for receiving, each stream object can only use a single transport. There is currently no support for collective operations where multiple producers cooperate to construct a heap between them. It is still possible to do multi-producer, single-consumer operation if the heap IDs are kept separate.

Because each stream has only one transport, there is a separate class for each, rather than a generic Stream class. Because there is common configuration between the stream classes, configuration is encapsulated in a :pyspead2.send.StreamConfig.

Streams send pre-baked heaps, which can be constructed by hand, but are more normally created from an :py~spead2.ItemGroup by a :pyspead2.send.HeapGenerator. To simplify cases where one item group is paired with one heap generator, a convenience class :pyspead2.send.ItemGroup is provided that inherits from both.

spead2.send.HeapGenerator

spead2.send.HeapGenerator.add_to_heap

spead2.send.HeapGenerator.get_heap

spead2.send.HeapGenerator.get_start

spead2.send.HeapGenerator.get_end

Substreams

For some transport types it is possible to create a stream with multiple "substreams". Each substream typically has a separate destination address, but all the heaps within the stream are sent in order and the stream configuration (including the rate limits) apply to the stream as a whole. Using substreams rather than independent streams gives better control over the overall transmission rate, and uses fewer system resources.

When sending a heap, an optional parameter called substream_index selects the substream that will be used.

Blocking send

There are multiple stream classes, corresponding to different transports, and some of the classes have several variants of the constructor. They all implement the following interface, although this base class does not actually exist:

UDP

Note that since UDP is an unreliable protocol, there is no guarantee that packets arrive.

For each constructor overload, the endpoints parameter can also be replaced by two parameters that contain the hostname/IP address and port for a single substream (for backwards compatibility).

TCP

TCP/IP is a reliable protocol, so heap delivery is guaranteed. However, if multiple threads all call :py~spead2.send.AbstractStream.send_heap at the same time, they can exceed the configured max_heaps and heaps will be dropped.

Because spead2 was originally designed for UDP, the default packet size in :py~spead2.send.StreamConfig is quite small. Performance can be improved by increasing it (but be sure the receiver is configured to handle larger packets).

TCP/IP is also a connection-oriented protocol, and does not support substreams. The endpoints must therefore contain exactly one endpoint (it takes a list for consistency with ~spead2.send.UdpStream).

Raw bytes

In-process transport

Refer to the separate documentation <py-inproc>.

Asynchronous send

As for asynchronous receives, asynchronous sends are managed by asyncio. A stream can buffer up multiple heaps for asynchronous send, up to the limit specified by max_heaps in the :py~spead2.send.StreamConfig. If this limit is exceeded, heaps will be dropped, and the returned future has an :pyIOError exception set. An :pyIOError could also indicate a low-level error in sending the heap (for example, if the packet size exceeds the MTU).

The classes exist in the :pyspead2.send.asyncio modules, and mostly implement the same constructors as the synchronous classes. They implement the following abstract interface (the class does not actually exist):

TCP

For TCP, construction is slightly different: except when providing a custom socket, one uses a coroutine to connect:

spead2.send.asyncio.TcpStream.connect