Join GitHub today
GitHub is home to over 50 million developers working together to host and review code, manage projects, and build software together.Sign up
Streamtools as a Language
The following document intends to illustrate what Streamtools might look like if it was actually a language.
Lack of modularity
Every time we want to do some kind of operation that is not currently afforded by Streamtools we need to write Go, recompile, rerelease. This is especially annoying for situations where we are limited by the grammar of ST. The lack of modularity is also impacts people who may wish to curate their own work environment with blocks that are specific to their work.
Inconsistant Flow Control
There are several problems with how ST currently uses flow for control.
Inconsistant Block API
The inconsistent block API is product of inconsistent use of flow for control as well as inconsistent access to state. For instance, the
The above diagram represents a system where two messages enter the
This is undesirable, as it introduces all kinds of issues of synchronicity in the stream or unnecessary metadata embedded in each message. For instance, we could associate the wrong messages with one another or we could require a whole new system to deal with metadata in each message to resolve how the messages should be paired. Either way, this is added complexity that does not help in the development/understanding of a grammar consistent across blocks. This leads us to our first proposal:
1. Core blocks are responsible for ONE synchronous operation
This means that blocks like
What the above means is that all input routes block until all input routes have received a message. Once all routes have been satisfied the operation of the block is triggered.
The above suggestion is a bit at odds with our current instantiation of the idea of "rules" which are used to give parameters to a block. Many of our blocks have "rules" that persist throughout the duration the blocks life and never need to be changed. Which leads us to...
2. "Constant" routes and "Path" routes
Since our core blocks depend having all inputs satisfied once per block operation, this presents a problem when we have situations where we'd like a constant value to be applied as a parameter for a given block. We don't want to send one message per message in the above addition example if we want to add a constant to a stream.
Given the ability to configure how a route accepts and treats its value fixes this problem. In diagram 2, two modes of configuration are presented. The dialog on top is for a situation where a block is meant to accept a stream and retrieve a specific value from within an input object. In the case of the diagram, the route is configured accept the number for the
The bottom dialog in Diagram 2 presents an option to input JSON directly into the right stream input, so that it is always constant across operations. This is a "Constant" route. This provides us with 3 types of patterns:
The middle pattern in Diagram 3 presents a problem: how can we guarantee two streams appear at that block in a way that is useful to us?
3. blocking and flow control
Diagram 4 introduces some new blocks:
This pattern takes advantage of blocking in order to maintain consistency across events. When a message is broadcast from
NOTE: The right input for
4. Shared State
One of the deficiencies in our current implementation is that complex operations with state are rather difficult. This is because we can only access the state via one block -- a block that has one output. Determining what the output means in regards to the state can be rather tricky -- and has resulted in the inclusion metadata into our outputs.
One way to amend the current situation is to afford multiple outputs that signal various circumstances. Similar to the
In addition to multiple outputs, another way to fix the problem of having to worry about how many different state queries work together is to divorce the state from the block. This amounts to having a completely new element in ST -- a data store.
The data store does not participate in message flow. It is a node that is associated with various blocks that act as an API to its contents. Diagram 5 illustrates what some of this blocks may look like. The "association" is illustrated as a route that exists on the side -- this is for illustration purposes only. There is no directionality when associating a block with a store.
This affords us the ability to run multiple control flows with a single data store and affords the construction of complex patterns.
The above diagram illustrates a timeseries. values come in through
5. Building Blocks
Building blocks is still not something I've entirely figured out. The above diagram displays how a
Here are some scattered notes on how I think custom blocks should work:
to sum up, the goals of this proposal are:
Above example presents a new problem:
Another large bit to add to the "problems" list: