ScorpioBroker

Scorpio NGSI-LD Broker

Hello there!

Welcome to the Scorpio NGSI-LD Broker. You are currently in the Scorpios main branch.

To get you quickly started you can just choose on of the provided docker-compose files and start it up.

We also provide all docker images on the Amazon Public ECR

Which option to choose?

java vs. native

We have a regular java version, which should run on any machine running java, and we have native compilations, currently primarily for linux amd64. There is a trade-off between the java version and the native one - the java one is faster, but the native one uses less memory - i.e. if your environment is memory constraint, you should go for a native version, otherwise, we would recommend the java version

all-in-one vs. micro-services

The implementation architecture of Scorpio is based on micro services, which allows individual scaling with micro services (e.g. assuming you need a lot of subscriptions, you create multiple instances of the subscription service) in the cloud. If there are multiple micro services, they need to communicate with each other and this is done using a message bus - the standard message bus is kafka, but also mqtt and rabbitmq are supported - a special version is sqs, which is the native message bus of AWS, which they use when Scorpio is used in their cloud as part of the Garnet framework - if you are not using AWS, do not use sqs. If you do not need to scale in the cloud, you can use the all-in-one-runner (aaio), which can also work with a message bus, but it does not have to, i.e. java-aaio uses direct internal communication (i guess a kind of "pseudo" message bus).

distroless docker image vs linux base image

When using a docker, a base version for the docker container image is needed. We use some kind of minimum linux based image - which changed occasionally (also due to security holes that are discovered). Since the last version, we also support a "distroless image" - which is a bare minimum docker image..

Local DB and Messagebus setup

If you have a local PostgreSQL (and Kafka) setup you can also download pre builds for Ubuntu, Windows and MacOS from the releases

Where to get examples

To get a good collection of NGSI-LD operations and examples we recommend to have a look at our test suite. It is Postman based and you can just import it together with the respective environment settings all-in-one or distributed

What is Scorpio and NGSI-LD

Scorpio is an NGSI-LD compliant context broker developed by NEC Laboratories Europe and NEC Technologies India. It implements the full NGSI-LD API as specified by the ETSI Industry Specification Group on cross cutting Context Information Management (ETSI ISG CIM).

The NGSI-LD API enables the management, access and discovery of context information. Context information consists of entities (e.g. a building) and their properties (e.g. address and geographic location) and relationships (e.g. owner). Thus Scorpio enables applications and services to request context information – what they need, when they need it and how they need it.

The functionalities of the NGSI-LD API include:

Create, update, append and delete context infomration.
Query context information, including filtering, geographic scoping and paging.
Subscribe to changes in context information and receive asynchronous notifications.
Register and discover sources of context information, which allows building distributed and federated deployments.

Scorpio is a FIWARE Generic Enabler. Therefore, it can be integrated as part of any platform “Powered by FIWARE”. FIWARE is a curated framework of open source platform components which can be assembled together with other third-party platform components to accelerate the development of Smart Solutions. The roadmap of this FIWARE GE is described here.

You can find more info at the FIWARE developers website and the FIWARE website. The complete list of FIWARE GEs and Incubated FIWARE GEs can be found in the FIWARE Catalogue.

📚 Documentation	🎓 Academy	quay.io	🐳 Docker Hub	🎯 Roadmap

Background

Scorpio is an NGSI-LD Broker that allows managing and requesting context information. It supports the following functionalities:

Context Producers can manage their context – creating, updating, appending and deleting context information.
Context Consumers can request the context information they require, either identifying the entity or discover relevant entities by providing the entity type, possibly filtering according to property value, existing relationship and/or geographic scope provided as a GeoJSON feature.
Two interaction styles are supported, a synchronous query-response, and an asynchronous subscribe / notify, where notifications can be based on a change in property or relationship, or on a fixed time interval.
Scorpio implements NGSI-LD’s optional temporal interface for requesting historic information, e.g. the property values measured within a specified time interval.
Scorpio supports multiple deployment configurations including centralized, distributed and federated ones. In addition to the Context Producers mentioned above, there can be Context Sources that themselves implement the NGSI-LD interface. These Context Sources can register themselves with the information they can provide on request (not the information (values) itself). A Scorpio Broker in a distributed setting can then discover the Context Sources that may have information for answering a request based on their registrations, request and aggregate the information from different Context Sources and provide it to the requesting Context Consumer.
In a federated setting the Context Source can itself be an NGSI-LD Broker. Federations can be used to combine information from multiple providers that want to (partially) share information. An important difference is then typically in the granularity of the registration, e.g. “I have information about entities of entity type building within a geographic area”, instead of “I have information about Building A”
Scorpio supports all the mentioned deployment configurations. Thus it provides scalability and the possibility of extending scenarios in an evolutionary way. For example two separate deployments can be combined or for scalability reasons different brokers can be used – completely transparent to Context Consumers that can still use a single point of access.

Installation and Building

Scorpio is developed in Java using Quarkus as microservice framework and Apache Maven as build tool. It requires Apache Kafka as a message bus and Postgres with PostGIS extensions as database.

The easiest way to get Scorpio running is using docker-compose. Please have a look here for examples.

Configuration

Scorpio can be configured via environment variables. Please have a look here for a full list of config options. To configure via environment variable provide the config option in all capital letters and replace the dot with an underscore. So e.g. quarkus.http.port becomes QUARKUS_HTTP_PORT.

The most common parameters are the following. #The port for the HTTP API quarkus.http.port=9090 #The PostgreSQL related settings dbhost=localhost dbport=5432 dbuser=ngb dbpass=ngb dbname=ngb (This is the name of the default database. Tenant databases will be prefixed with this setting.) #Kafka related settings bushost=localhost busport=9092 #Settings used by the auto registry and federation feature. This should be the public reachable name or IP of the Scorpio installation. gateway.host=localhost gateway.port=9090

Usage

By default the broker runs on port 9090 the base URL for interaction with the broker would be than http://localhost:9090/ngsi-ld/v1/

Simple Example

Generally speaking you can Create entities by sending an HTTP POST request to http://localhost:9090/ngsi-ld/v1/entities/ with a payload like this

{
    "id": "urn:ngsi-ld:testunit:123",
    "type": "AirQualityObserved",
    "dateObserved": {
        "type": "Property",
        "value": {
            "@type": "DateTime",
            "@value": "2018-08-07T12:00:00Z"
        }
    },
    "NO2": {
        "type": "Property",
        "value": 22,
        "unitCode": "GP",
        "accuracy": {
            "type": "Property",
            "value": 0.95
        }
    },
    "refPointOfInterest": {
        "type": "Relationship",
        "object": "urn:ngsi-ld:PointOfInterest:RZ:MainSquare"
    },
    "@context": [
        "https://schema.lab.fiware.org/ld/context",
        "https://uri.etsi.org/ngsi-ld/v1/ngsi-ld-core-context.jsonld"
    ]
}

In the given example the @context is in the payload therefore you have to set the ContentType header to application/ld+json

To receive entities you can send an HTTP GET to

http://localhost:9090/ngsi-ld/v1/entities/<entityId>

or run a query by sending a GET like this

http://localhost:9090/ngsi-ld/v1/entities/?type=Vehicle&limit=2
Accept: application/ld+json
Link: <http://<HOSTNAME_OF_WHERE_YOU_HAVE_AN_ATCONTEXT>/aggregatedContext.jsonld>; rel="http://www.w3.org/ns/json-ld#context";type="application/ld+json"

API Walkthrough

More detailed examples of what you can do with the NGSI-LD API provided by Scorpio can be found in the API Walkthrough.

Tests

Scorpio has two sets of tests. We use JUnit for unit tests and the FIWARE NGSI-LD Testsuite, which is npm test based, for system tests. Further details about testing can be found in the Testing Guide.

Further resources

For more detailed explaination on NGSI-LD or JSON-LD. Please look at the

You can find a set of example calls, as a Postman collection, in the Examples folder. These examples use 2 Variables

gatewayServer, which has to be <brokerIP>:<brokerPort>. When using default settings locally it would be localhost:9090
link, which is for the examples providing @context via the Link header. For the examples we host an example @context. Set link to https://raw.githubusercontent.com/ScorpioBroker/ScorpioBroker/master/Examples/index.json

Acknowledgements

EU Acknowledgetment

This activity has received funding from the European Union’s Horizon 2020 research and innovation programme under Grant Agreement No. 2020-EU-IA-0274 (SALTED), No. INEA/CEF/ICT/A2019/2063604 (ODALA), No. 731993 (Autopilot), No. 814918 (Fed4IoT) and No. 767498 (MIDIH, Open Call (MoLe).

ODALA project
SALTED project
AUTOPILOT project: Automated driving Progressed by Internet Of Things
Fed4IoT project
MIDIH Project, Open Call (MoLe)

Credit where credit is due

We like to thank everyone who has contributed to Scorpio. This goes for the entire Scorpio Devlopment Team as well as all external contributor. For a complete list have a look at the CREDITS file.

Code of conduct

As part of the FIWARE Community we try our best to adhere to the FIWARE Code of Conduct and expect the same from contributors.

This includes pull requests, issues, comments, code and in code comments.

As owner of this repo we limit communication here purely to Scorpio and NGSI-LD related topics.

We are all humans coming from different cultural backgrounds. We all have our different quirks, habits and mannerisms. Therefor misunderstandings can happen. We will give everyone the benefit of doubt that communication is done with good intentions in mind trying to advance Scorpio and NGSI-LD. We expect the same from contributors. However if someone is repeatedly trying to provoke, attack a person, shift discussions or ridicule someone we WILL make use of our house right and put an end to this.

If there is a dispute to be resolved we as owners of this repo have the final word.

License

Scorpio is licensed under BSD-3-Clause. For contributions this contribution license applies.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly