Skip to content
API collecting data from Cyface sensor hardware
Java HTML Shell Dockerfile JavaScript
Branch: master
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
config/checkstyle
gradle/wrapper
grafana
mongo-user-init-scripts
prometheus
src
.dockerignore
.gitignore
CHANGELOG.md
COPYING
Dockerfile
README.adoc
build.gradle
build.sh
docker-compose.yml
entrypoint.sh
gradlew
gradlew.bat
settings.gradle
shippable.yml

README.adoc

Collector

vert.x 3.8.0 purple mongo 4.0.11 purple

dev coverageBadge?branch=dev

This application represents the Cyface data collector software. It is used to collect traffic data from Cyface measurement devices, such as our sensor box or our smartphone application. Our smartphone SDK is available as GPL application for Android and iOS (or as Podspec) as well. If you require this software under a closed source license for you own projects, please contact us.

Building

Simply run build.sh inside the root of this project. This builds the jar file which is then packed into the Docker container which is build afterwards. This script also makes sure all required subdirectories are created.

Release a new Version

To release a new version:

  1. Create a new branch following the format release/x.y.z/PRJ-<Number>_some-optional-explanation. Where x.y.z is the number of the new version following semantic versioning, PRJ is the project this release has been created for, <Number> is the issue in the task tracker created for this release. You may also add an optional human readable explanation.

  2. Increase version numbers in build.gradle, docker-compose.yml, src/main/resources/webroot/openapi.yml and src/main/resources/management/openapi.yml and CHANGELOG.md.

  3. Commit and push everything to Github.

  4. Create Pull Requests to master and dev branches.

  5. If those Pull Requests are accepted merge them back, but make sure, you are still based on the most recent versions of master and dev.

  6. Create a tag with the version on the merged master branch and push that tag to the repository.

Execution

This section describes how to execute the Cyface Data Collector.

It starts with an explanation on how to set up all the required certificates. This is a necessary prerequisite for all the following steps. So DO NOT skip it.

Following the certificate setup is an explanation on how to run the Cyface Data Collector from a Docker environment. This is the recommended variant if you do not need to change the collector itself, but only need to develop against its API.

The section continues with an explanation on the supported configuration parameters. If you are not using the Docker environment, you will probably have to set a few of them to the correct values, before running the Cyface Data Collector.

The last two sections provide explanations on how to run the software directly from the terminal or from within an IDE such as Eclipse or IntelliJ. For these execution variants you need the parameters explained in the preceeding section.

Certificates

The Cyface Data Collector provides keys you may use for testing and during development. Those keys are located in src/test/resources. To use them outside of Unit tests you need to copy them to an appropriate location. ATTENTION: DO NOT USE THOSE KEYS IN A PRODUCTION ENVIRONMENT. Since they are in our repository, they are openly available to everyone on the internet, so everyone can compromise security on your server if you use the default keys.

The Cyface Data Collector requires two keys to issue and authenticate JWT tokens for users trying to communicate with the service. Just place the appropriate files as private_key.pem and public.pem in secrets/jwt, right next to the docker-compose.yml file .

To generate new keys follow the instructions in the Vert.x documentation for Using RSA keys.

Running from Docker

After you did build the system (see Building above) simply run docker-compose up inside the root of this project. This calls docker to bring up two Mongo database containers and a container running the Cyface data collector API.

The Collector API is by default available via port 8080. This means if you boot up everything using the default settings, the Collector API is accessible via http://localhost:8080/api/v2.

ATTENTION: The docker setup should only be used for development purposes. It exposes the Cyface data collector as well as the ports of both Mongo database instances freely on the local network. Use docker-compose ps to see which ports are mapped to which by Docker. For using such a setup in production, you may create your own Docker setup, based on our development one.

Running without Docker

Running the Cyface data collector without Docker, like for example from the terminal or from within your IDE is a little more complex. It requires a few set up steps and command knowledge as explained in the following paragraphs.

Running a Mongo Database for Data and User Storage

Before you can run the Cyface data collector you need to set up a Mongo database.

If you use the Docker environment as explained above, this is done for you. If you run the Cyface data collector on your own, you are responsible for providing a valid environment, including Mongo.

The database is used to store the collected data and information about valid user accounts. For information on how to install and run a Mongo database on your machine please follow the tutorial. If you take the default installation, the default settings of the Cyface data collector should be sufficient to connect to that instance. ATTENTION: However be aware that this is not recommended as a production environment.

Data Collector Arguments

The Cyface data collector supports a few parameters to fine tune the runtime. All of these parameters also provide reasonable defaults for quick setup. The parameters are provided using the typical Vertx -conf parameter with a value in JSON notation. The following parameters are supported:

  • jwt.private: The path of the file containing the private key used to sign JWT keys. This defaults to secrets/private_key.pem, which you should never use in production.

  • jwt.public: The path of the file containing the public key used to sign JWT keys. This defaults to secrets/public.pem, which you should never use in production.

  • http.port: The port the API is available at. This defaults to 8080.

  • http.host: The hostname under which the Cyface Data Collector is running. This can be something like localhost.

  • http.endpoint: The path to the endpoint the Cyface Data Collector is running. This can be something like /api/v2.

  • http.port.management: The port the management API is available at. This defaults to 13371.

  • mongo.userdb: Settings for a Mongo database storing credential information about all the users capable of logging into the system. This defaults to a Mongo database available at mongodb://127.0.0.1:27017. The value of this should be a JSON object configured as described here. In addition, if you use two different Mongo databases for credentials and data you should provide different values for the JSON key data_source_name.

  • admin.user: The username of a default administration account which is created if it does not exist upon start up. This defaults to admin. You must change this in a production environment.

  • admin.password: The password for the default administration account. This defaults to secret. You must change this in a production environment.

  • salt.path: The path to a salt file used to encrypt passwords stored in the user database even stronger. This defaults to secrets/salt. If the file does not exist a default salt is used. You should not do this in a production environment.

  • mongo.datadb: Settings for a Mongo database storing all data uploaded via the Cyface data collector. This defaults to a Mongo database available at mongodb://127.0.0.1:27017. The value of this should be a JSON object configured as described here. In addition, if you use two different Mongo databases for credentials and data you should provide different values for the JSON key data_source_name.

  • metrics.enabled: Set to either true or false. If true the collector API publishes metrics using micrometer. These metrics are accessible by a Prometheus server (Which you need to setup yourself) at port 8081.

Running from Command Line

To launch your tests:

./gradlew clean test

To package your application:

./gradlew clean assemble

To run your application:

./gradlew clean run

Running from IDE

To run directly from within your IDE you need to use the de.cyface.collector.Application class, which is a subclass of the Vert.x launcher. Just specify it as the main class in your launch configuration with the program argument run de.cyface.collector.verticle.MainVerticle.

Mongo Database

Setup

The following is not strictly necessary but advised if you run in production or if you encounter strange problems related to data persistence. Consider reading the Mongo Database Administration Guide and follow the advice mentioned there.

Administration

To load files from the Mongo GridFS file storage use the Mongofiles tool.

  • Showing files: mongofiles --port 27019 -d cyface-data list

  • Downloading files: mongofiles --port 27019 -d cyface-data get f5823cbc-b8f5-4c80-a4b1-7bf28a3c7944

  • Unzipping files: printf "\x78\x9c" | cat - f5823cbc-b8f5-4c80-a4b1-7bf28a3c7944 | zlib-flate -uncompress > test2

TODO

  • Setup Cluster

  • Vertx

  • MongoDb

Licensing

Copyright 2018,2019 Cyface GmbH

This file is part of the Cyface Data Collector.

The Cyface Data Collector is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any later version.

The Cyface Data Collector is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details.

You should have received a copy of the GNU General Public License along with the Cyface Data Collector. If not, see http://www.gnu.org/licenses/.

You can’t perform that action at this time.