A fast and light Python implementation of the Streaming Protocol for the Exchanged of Astronomical Data (SPEAD)
Fetching latest commit…
Cannot retrieve the latest commit at this time
PySPEAD This is a fast and light Python implementation of the Streaming Protocol for the Exchange of Astronomical Data New in 0.5.1: - Sanitised variable length string handling so as not to return array containing single string New in 0.5: - Added experimental string handling using mkfmt(('s',8)) ToDo: - Unpacking is now pretty quick, but for arrays, could we get one more speed-up by unpacking data directly into a numpy array, rather than to a tuple? - Right now, packets are pulled straight from the socket and put into a packet ringbuffer. The packet ringbuffer then feeds a callback that consumes the packets and creates a heap. While heap creation from packets is performed in C, a Python callback is currently used to hand the packets to SpeadHeap. (a) We could make heap assembly happen automatically, so that the Python callback is fed heaps directly. (b) While implementing this, we should also incorportate packet sorting by heap_cnt so that packets can arrive out of order, but heaps are assembled in order to the extent allowed by the available buffer. - Confusingly, there are currently two kinds of "heap" in spead.py: one is a SpeadHeap that is generated by a receiver, and one is a dictionary that is used by a transmitter. We might move the heap => packets pipeline into C as well, and make all heaps of the SpeadHeap type. - Packet sorting is done as each packet is inserted, but it could be done faster by buffering them and then quicksorting - _spead.pack() is currently the limiting factor on transmit. Can it be sped up? - tests/test_spead.py uses lots of explicit constants. We should migrate to using the named constants provided in the module. Known Issues: - Occasionally MacOSX hangs forever in unittest test_spead.py TestTransportUDPrx.test_get_packets_term. This seems to be b/c no data is being received. Hanging seems to be more frequent when buffer_socket.cpp has the DEBUG flag set. Linux doesn't seem to have the same problem. Cause unknown.