Stream2py design notes

Indexing infinite sequences

This has to do with how to implement a buffer reader's range (slicing) function that takes care of "interval queries" on the data.

buff[bt:tt] -> items_for_bt_tt_interval

Currently, the source_reader defines the key function that defines the index (order and reference) and this key is used by the buffer reader. We propose to move the concern of the key to the buffer reader instead.

The proposed architecture for this is

• Buffer writes are coupled with incrementing a number_of_items_inserted counter
• Buffer reads use, by default, absolute indices which are mapped to the buffer's index via number_of_items_inserted
• Any other indexing that is needed by (buffer) readers is achieved by converting the desired external index to the internal buffer's (absolute) index

This architecture was implemented in the IndexedBuffer class of creek.infinite_sequence.

A BufferReader object would have a key_trans attribute that specifies how to transform the readers key to the actual absolute base key, which is then translated to the physical buffer's key to actual get the data.

By default key_trans would be the identity, but any function could be specified (though it should be monotonous to get non-degenerate behavior).

This key_trans could be formulaic (such as a linear/affine transformation), but may need to be more complex in some other situations.

For example, consider the case where the data itself contains the index. No formula can solve our index mapping in this case. It needs to be explicit.

Since data in the buffer is ordered, one fall back method for explicit index mapping, would be to use sorted lists bisect search to find items matching a bt:tt query. This achieves O(log(n)) performance, which may be sufficient in many cases.

One can achieve O(1) query time performance by maintaining an explicit mapping. This mapping would be coupled with the key_trans function and used (possibly shared, through the sharing of the key_trans function) by any readers that need to work in that index.

To test

Test that reads are consistent when the buffer advances due to new writes

Same query key to a buffer that changed.

indexes transformed by monotonous function

ledger-based index mapping (maintained and applied)

Interface options in making a stream reader

A few proposals

straight code

source_reader = SourceReader(**s)
stream_buffer = StreamBuffer(source_reader, **b)
buffer_reader = BufferReader(stream_buffer, **r)

fluent interface

buffer_reader = SourceReader(**s).stream_buffer(**b).buffer_reader(**r)

nested aggregator

buffer_reader = Aggreg(
    source_reader=SourceReader(**s),
    stream_buffer=StreamBuffer(**b),
    buffer_reader=BufferReader(**r)
)

flat aggregator

buffer_reader = Aggreg(**s, **b, **r)

Discussion

As a general rule, I'm for breaking things into small objects like the three objects we're talking about here, then combining them to get interfaces that might be more appropriate for some contexts.

This means that the two aggregators above don't preclude the other two proposals. In fact, I would argue that they should be built using the fluent interface or straight code methods.

The nested aggregator nicely separates the three roles and their parameters. The same parameter name (with different values) can be used in more than one role. On the other hand, the flat aggregator allows one to share and align parameters, and only a few of the params are actually required, allows for a sparse simple interface for most cases.

More proposals

Lets take this as the point of departure:

stream_reader = SourceReader(**s).stream_buffer(**b).mk_reader(**r)

We want to make a StreamReader that gives us the chain automatically with default (smart or not) b and r params, but with a means to specify these if and when needed.

We could do the flat version:

stream_reader = StreamReader(**s, **b, **r)

Or more like this this:

dflt_stream_reader = StreamReader(**s)  # .stream_buffer(**b).mk_reader(**r) done automatically with defaults
stream_reader_with_custom_reader = StreamReader(**s).mk_reader(**r)  # .stream_buffer(**b) done automatically with defaults
stream_reader_with_custom_buffer = StreamReader(**s).stream_buffer(**b)  # .mk_reader(**r) done automatically with defaults
stream_reader_with_all_customs = StreamReader(**s).stream_buffer(**b).mk_reader(**r)

A problem to keep an eye on with the above is that we want multiple readers to share the same source and buffer.