Skip to content
A connector in charge of persisting context data sources into other third-party databases and storage systems, creating a historical view of the context
Java Python Gherkin Shell Other
Branch: master
Clone or download
fgalan Merge pull request #1708 from telefonicaid/revert-1707-revert-1705-ta…
…sk/enable_JAVA_OPTS_for_flume

Revert "Revert "task/enable_JAVA_OPTS_for_flume""
Latest commit 707b201 Jul 23, 2019
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
cygnus-common Revert "Revert "task/enable_JAVA_OPTS_for_flume"" Jul 22, 2019
cygnus-ngsi Revert "Revert "task/enable_JAVA_OPTS_for_flume"" Jul 22, 2019
cygnus-twitter Step: 1.15.0 -> 1.15.0_SNAPSHOT Jun 28, 2019
doc add missed doc entries Jul 10, 2019
docker rollback use master branch Jul 23, 2019
rpm Update README.md Jan 18, 2017
.gitattributes FIX typo in .gitattributes Dec 15, 2014
.gitignore Undo Unwanted .gitignore commit Mar 19, 2019
.travis.yml REMOVE openjdk7 from travis conf Mar 21, 2019
CHANGES_NEXT_RELEASE resolve conflict Jul 23, 2019
ContributionPolicy.txt add symlink Oct 9, 2018
LICENSE FIX licence file (GPLv3 -> AGPLv3) Feb 28, 2014
README.md Update README.md Jul 11, 2019
compile.sh [1213] Add compile.sh scripts Oct 14, 2016
mkdocs.yml
populate_hdfs_new_encoding.sh [1271][cygnus-ngsi][populate_hdfs_new_encoding.sh] Update Nov 2, 2016
reporting_issues_and_contact.md [1421][cygnus][doc] Fix titles according to Github rendering Mar 30, 2017
telefonica_checkstyle.xml MOVE contents under flume/ intro root directory (removing empty flume… Mar 24, 2015
test.sh Update test.sh Jun 28, 2019

README.md

Cygnus

License Docker badge Support badge
Documentation badge Build badge Status

Cygnus is a connector in charge of persisting context data sources into other third-party databases and storage systems, creating a historical view of the context. Internally, Cygnus is based on Apache Flume, Flume is a data flow system based on the concepts of flow-based programming. It supports powerful and scalable directed graphs of data routing, transformation, and system mediation logic. It was built to automate the flow of data between systems. While the term 'dataflow' can be used in a variety of contexts, we use it here to mean the automated and managed flow of information between systems.

Each data persistence agent within Cygnus is composed of three parts - a listener or source in charge of receiving the data, a channel where the source puts the data once it has been transformed into a Flume event, and a sink, which takes Flume events from the channel in order to persist the data within its body into a third-party storage.

This project is part of FIWARE. For more information check the FIWARE Catalogue entry for the Core Context Management.

📚 Documentation 🎓 Academy 🐳 Docker Hub

Contents

Background

Internally, Cygnus is based on Apache Flume, a technology addressing the design and execution of data collection and persistence agents. An agent is basically composed of a listener or source in charge of receiving the data, a channel where the source puts the data once it has been transformed into a Flume event, and a sink, which takes Flume events from the channel in order to persist the data within its body into a third-party storage.

Cygnus is designed to run a specific Flume agent per source of data.

Current stable release is able to persist the following sources of data in the following third-party storages:

  • NGSI-like context data in:
    • HDFS, the Hadoop distributed file system.
    • MySQL, the well-known relational database manager.
    • CKAN, an Open Data platform.
    • MongoDB, the NoSQL document-oriented database.
    • STH Comet, a Short-Term Historic database built on top of MongoDB.
    • Kafka, the publish-subscribe messaging broker.
    • DynamoDB, a cloud-based NoSQL database by Amazon Web Services.
    • PostgreSQL, the well-known relational database manager.
    • Carto, the database specialized in geolocated data.
    • PostGIS, a spatial database extender for PostgreSQL object-relational database.
    • Orion, the FIWARE Context Broker.
    • Elasticsearch, the distributed full-text search engine with JSON documents.
    • Arcgis, the Arcgis is a geographic information system (GIS).
  • Twitter data in:

IMPORTANT NOTE: for the time being, cygnus-ngsi and cygnus-twitter agents cannot be installed in the same base path, because of an incompatibility with the required version of the httpclient library. Of course, if you are going to use just one of the agents, there is no problem at all.

cygnus-ngsi: Docker badge
cygus-twitter : Docker badge

Cygnus place in FIWARE architecture

Cygnus (more specifically, cygnus-ngsi agent) plays the role of a connector between Orion Context Broker (which is a NGSI source of data) and many FIWARE storages such as CKAN, Cosmos Big Data (Hadoop) and STH Comet. Of course, as previously said, you may add MySQL, Kafka, Carto, etc as other non FIWARE storages to the FIWARE architecture.

FIWARE architecture

Install

Fiware/Cygnus has three sub-modules cygnus-common, cygnus-ngsi and cygnus-twitter. Information about how to install these modules can be found at the corresponding section of Installation.

Following are the links of install section of Fiware/Cygnus sub-modules:

Roadmap

The roadmap of this FIWARE GE is described here.

Further documentation

The per agent Quick Start Guide found at readthedocs.org provides a good documentation summary (cygnus-ngsi, cygnus-twitter).

Nevertheless, both the Installation and Administration Guide and the User and Programmer Guide for each agent also found at readthedocs.org cover more advanced topics.

The per agent Flume Extensions Catalogue completes the available documentation for Cygnus (cygnus-ngsi, cygnus-twitter).

Other interesting links are:

Reporting issues and contact information

Any doubt you may have, please refer to the Cygnus Core Team.


License

Cygnus is licensed under Affero General Public License (GPL) version 3. You can find a copy of this license in the repository.

© 2019 Telefonica Investigación y Desarrollo, S.A.U

Are there any legal issues with AGPL 3.0? Is it safe for me to use?

There is absolutely no problem in using a product licensed under AGPL 3.0. Issues with GPL (or AGPL) licenses are mostly related with the fact that different people assign different interpretations on the meaning of the term “derivate work” used in these licenses. Due to this, some people believe that there is a risk in just using software under GPL or AGPL licenses (even without modifying it).

For the avoidance of doubt, the owners of this software licensed under an AGPL 3.0 license
wish to make a clarifying public statement as follows:

Please note that software derived as a result of modifying the source code of this software in order to fix a bug or incorporate enhancements is considered a derivative work of the product. Software that merely uses or aggregates (i.e. links to) an otherwise unmodified version of existing software is not considered a derivative work, and therefore it does not need to be released as under the same license, or even released as open source.

You can’t perform that action at this time.