Python JavaScript Jupyter Notebook CSS HTML Makefile Shell
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Failed to load latest commit information.
docs added docs back Jul 20, 2017
docs_src updated index Jul 20, 2017
examples Fix to unit tests (including python 3) Jul 27, 2017
hyperstream Fix to unit tests (including python 3) Jul 27, 2017
plugins Debugging Tutorial 5. Jul 26, 2017
tests Fix to unit tests (including python 3) Jul 27, 2017
.codeclimate.yml Update .codeclimate.yml Jul 24, 2017
.csslintrc codeclimate configs Apr 4, 2017
.eslintignore codeclimate configs Apr 4, 2017
.eslintrc.yml codeclimate configs Apr 4, 2017
.gitignore Some downgrading of logging messages and other tidying Jun 27, 2017
.pep8 Create .pep8 Jul 24, 2017
.travis.yml reverting changes to travis Jul 24, 2017
CODE_OF_CONDUCT.md Create CODE_OF_CONDUCT.md Jun 22, 2017
LICENSE Create LICENSE Jun 22, 2017
MANIFEST.in Added manifest to upload requirements to pypi Jul 24, 2017
README Added manifest to upload requirements to pypi Jul 24, 2017
README.md Solved merge Jul 21, 2017
_config.yml Set theme jekyll-theme-dinky Jul 20, 2017
hyperstream.png added logo Nov 25, 2016
hyperstream.svg added logo Nov 25, 2016
hyperstream_config.json Added history of tool execution: Jun 23, 2017
main.py Added example main.py that runs the online engine Jul 24, 2017
pypi.sh Added manifest to upload requirements to pypi Jul 24, 2017
requirements.txt Updated requirements Jul 19, 2017
runtest.sh Added mongodb docker container to runtest Jun 16, 2017
runtests.sh Added script to run tests and start tutorial Jun 19, 2017
setup.cfg Update setup.cfg Jul 24, 2017
setup.py Added manifest to upload requirements to pypi Jul 24, 2017

README.md

HyperStream logo

HyperStream

DOI Join the chat at https://gitter.im/IRC-SPHERE-HyperStream/Lobby Build Status Dependency Status Test Coverage Issue Count Documentation Status

Hyperstream is a large-scale, flexible and robust software package for processing streaming data.

Hyperstream overcomes the limitations of other computational engines and provides high-level interfaces to execute complex nesting, fusion, and prediction both in online and offline forms in streaming environments. Although developed specifically for SPHERE, Hyperstream is a general purpose tool that is well-suited for the design, development, and deployment of algorithms and predictive models in a wide space of sequential predictive problems.

This software has been designed from the outset to be domain-independent, in order to provide maximum value to the wider community. Key aspects of the software include the capability to create complex interlinked workflows, and a computational engine that is designed to be "compute-on-request", meaning that no unnecessary resources are used.

Installation

Docker images

If you do not want to install all the packages separately you can use our Docker bundle available here.

Local machine

Install via pip

pip install hyperstream
python -c 'from hyperstream import HyperStream'

To get the latest version

pip install -U git+git://github.com/IRC-SPHERE/HyperStream.git#egg=hyperstream
pip install -r requirements.txt

Or clone the repository

git clone git@github.com:IRC-SPHERE/HyperStream.git
cd HyperStream
virtualenv venv
. venv/bin/activate
pip install -r requirements.txt
python -c 'from hyperstream import HyperStream'

Additionally, one of the requirements to run Hyperstream is a MongoDB server. By default, Hyperstream tries to connect to the port 27017 on the localhost.

To see the installation steps of MongoDB go to the official documentation. E.g. in a Debian OS it is possible to install with the following command

sudo apt-get install mongodb

Once the MongoDB server is installed, it can be started with the following command

service mongod start

Running tests

Run the following command

nosetests

Note that for the MQTT logging test to succeed, you will need to have an MQTT broker running (e.g. Mosquitto). For example:

docker run -ti -p 1883:1883 -p 9001:9001 toke/mosquitto

or on OSX you will need pidof and mosquitto:

brew install pidof
brew install mosquitto
brew services start mosquitto

Tutorials

The following tutorials show how to use HyperStream in a step-by-step guide.

    from hyperstream import HyperStream, StreamId, TimeInterval
    from hyperstream.utils import utcnow, UTC
    from datetime import timedelta

    hs = HyperStream(loglevel=20)
    M = hs.channel_manager.memory
    T = hs.channel_manager.tools
    clock = StreamId(name="clock")
    clock_tool = T[clock].window().last().value()
    ticker = M.get_or_create_stream(stream_id=StreamId(name="ticker"))
    now = utcnow()
    before = (now - timedelta(seconds=30)).replace(tzinfo=UTC)
    ti = TimeInterval(before, now)
    clock_tool.execute(sources=[], sink=ticker, interval=ti, alignment_stream=None)
    print(list(ticker.window().tail(5)))

    [StreamInstance(timestamp=datetime.datetime(2017, 6, 30, 16, 23, 39, tzinfo=<UTC>), value=datetime.datetime(2017, 6, 30, 16, 23, 39, tzinfo=<UTC>)), StreamInstance(timestamp=datetime.datetime(2017, 6, 30, 16, 23, 40, tzinfo=<UTC>), value=datetime.datetime(2017, 6, 30, 16, 23, 40, tzinfo=<UTC>)), StreamInstance(timestamp=datetime.datetime(2017, 6, 30, 16, 23, 41, tzinfo=<UTC>), value=datetime.datetime(2017, 6, 30, 16, 23, 41, tzinfo=<UTC>)), StreamInstance(timestamp=datetime.datetime(2017, 6, 30, 16, 23, 42, tzinfo=<UTC>), value=datetime.datetime(2017, 6, 30, 16, 23, 42, tzinfo=<UTC>)), StreamInstance(timestamp=datetime.datetime(2017, 6, 30, 16, 23, 43, tzinfo=<UTC>), value=datetime.datetime(2017, 6, 30, 16, 23, 43, tzinfo=<UTC>))]

HyperStream Viewer

The HyperStream Viewer is a python/Flask web-app for interacting with HyperStream. In order to keep HyperStream to a minimum, this web-app is released as a separate repository that takes the core as a dependency.

License

This code is released under the MIT license.

Acknowledgements

This work has been funded by the UK Engineering and Physical Sciences Research Council (EPSRC) under Grant EP/K031910/1 - "SPHERE Interdisciplinary Research Collaboration".