Skip to content
Streaming dataflow library for IoT applications. Program at a higher level, with reusable "things".
Branch: master
Clone or download
Latest commit c191a8f Aug 8, 2017

README.rst

ThingFlow

ThingFlow is a (Python3) framework for building IoT event processing dataflows. [1] The goal of this framework is to support the creation of robust IoT systems from reusable components. These systems must account for noisy/missing sensor data, distributed computation, and the need for local (near the data source) processing.

The source repository for ThingFlow-python is at https://github.com/mpi-sws-rse/thingflow-python.

Introduction

The fundamental abstractions in ThingFlow are:

  1. sensors, which provide a means to sample a changing value representing some quanity in the physical world,
  2. event streams, which are push-based sequences of sensor data readings, and
  3. things, which are reusable components to generate, transform, or consume the events on these streams.

Things can have simple, stateless logic (e.g. filter events based on a predicate) or implement more complex, stateful algorithms, such as Kalman filters or machine learning. Using ThingFlow, you describe the flow of data through these things rather than programming low-level behaviors.

Although ThingFlow presents a simple dataflow model to the user, internally it uses an event-driven programming model, building on Python's asyncio module. In addition to being a natural programming model for realtime sensor data, it reduces the potential resource consumption of Ant Events programs. The details of event scheduling are handled by the framework. Separate threads may be used on the "edges" of a dataflow, where elements frequently interact with external components that have blocking APIs.

ThingFlow integrates with standard Python data analytics frameworks, including NumPy, Pandas, and scikit-learn. This allows dataflows involving complex elements to be developed and refined offline and then deployed in an IoT environment using the same code base.

We call the implementation described here "ThingFlow-Python", as it should be possible to port the ideas of ThingFlow to other languages. Currently, one such port exists: "ThingFlow-MicroPython". This is a port ThingFlow to MicroPython, a limited version of Python 3 that runs "bare metal" on embadded devices. The ThingFlow-MicroPython port is included in the ThingFlow-Python repository under the subdirector micropython. It is documented in a chapter of the documentation.

Example

To give the flavor of ThingFlow, below is a short code snippet for the Raspberry Pi that reads a light sensor and then turns on an LED if the running average of the last five readings is greater than some threshold:

lux = SensorAsOutputThing(LuxSensor())
lux.map(lambda e: e.val).running_avg(5).map(lambda v: v > THRESHOLD)\
   .GpioPinOut()
scheduler.schedule_periodic(lux, 60.0)
scheduler.run_forever()

The first line instantiates a light sensor object and wraps it in an output thing to handle sampling and progagation of events.

The next two lines create a pipeline of things to process the data from the sensor. We call things which have a single input and output filters, as they can be composed to process a stream of events. The map filter extracts the data value from the sensor event, the running_avg filter averages the last five values, and the next map filter converts the value to a a boolean based on the threshold. The GpioPinOut thing is an adapter to the outside world. It turns on the LED based on the value of its input boolean value.

Finally, the last two lines of the example schedule the sensor to be sampled at a sixty second interval and then start the scheduler's main loop.

Dependencies

The ThingFlow proper is self-contained. You do not need any dependencies other than Python 3 (3.4 or later). Specific adapters and sensors may have additional dependencies (e.g. the MQTT adapters depend on MQTT client libraries).

Documentation

Documentation is hosted online at http://thingflow-python.readthedocs.io.

The source tree for the documentation is in the docs subdirectory - it is built using Sphinx. If you have Sphinx installed locally (along with the "Read the Docs" theme), you can also build it directly on your machine.

[1]ThingFlow was originally known as AntEvents.
You can’t perform that action at this time.