This is a main repository for netunicorn project.
netunicorn is a platform for building and running data pipelines on certain nodes for data collection experiments. It allows users to express arbitrary Python-based tasks and pipelines and assign them to particular nodes.
This platform is close to the Apache Airflow or CI/CD platforms, but instead optimizes the following:
- ability to use different infrastructures via specific connectors (e.g., SaltStack-based, Azure Container Instances, Mininet, etc.)
- short time between tasks
- work under the conditions of unstable network
- reproducibility and easy sharing of tasks and pipelines
Full documentation is available at docs and on the platform website.
Users are able to express arbitrary tasks and pipelines and run them on the already deployed platform.
To use the platform, a user should install the next packages:
pip install netunicorn # automatically installs client and library packages
To use the platform, administrators of the infrastructure should deploy it and provide you the next credentials:
- endpoint: API url of the platform
- username: your username
- password: your password
These credentials would be used to work with the platform via RemoteClient.
Please, refer to examples to see how to use the platform.
Administrators of the platform maintain the particular netUnicorn installation and underlying infrastructure.
Infrastructure is a set of nodes (like virtual machines, dynamically created containers, or physical nodes) that will be
used to run tasks and pipelines created by users.
See the full Administrator Documentation on our website.
The platform assumes that the infrastructure is already deployed and configured and centrally managed by one of the existing management tools (like Ansible, SaltStack, etc.). For this tool, a connector should be implemented.
See the list of the available connectors on the platform website
Also, the next services are required for the platform:
- PostgreSQL database for states and logs
- Docker registry
See the Simplified and Production Deployment documentation on our website