Skip to content
A conglomeration of (data) transformations
Branch: master
Clone or download
bbrewer-papa-x Merge pull request #34 from smartcitiesdata/smrt-1287-static-version
SMRT-1287 - preparing for a release bump that doesn't touch this
Latest commit 97711e0 Jun 11, 2019
Type Name Latest commit message Commit time
Failed to load latest commit information.
.formatter.exs SMRT-1126 Refactor voltron to do field based transformations and adde… Apr 18, 2019
.gitignore Add observer_cli and add .elixir_ls to gitignore May 24, 2019
Dockerfile Adding bash dependency to dockerfile Jun 11, 2019
LICENSE Adding liscence Jun 11, 2019
mix.exs SMRT-1287 - preparing for a release bump that doesn't touch this Jun 11, 2019
mix.lock SMRT-1314 Refactoring tests to new public api Jun 6, 2019



Applies a standard set of transformations to all data messages received and delivers them to a configurable topic

External Dependencies

  • Kafka (input topic, output topic, dead letter queue (DLQ) topic)
  • Redis


The following environment variables should be set:

  • FROM_TOPIC - The Kafka topic that data messages will be pulled from
  • TO_TOPIC - The Kafka topic that transformed data messages will be delivered to
  • REDIS_HOST - The redis host (e.g. '') where dataset definitions are stored. Currently this data is used to retrieve the dataset schema. In the future it will be used to determine what transformations should be applied to a particular dataset's data message
  • KAFKA_BROKERS - A comma delimited list of one or more Kafka brokers in the form of host1:port,host2:port



Input messages must be in the format defined by the Smart Cities Data Library

  "dataset_id": "abfe-124134-dfhdfg-4486784",
  "operational": {},
  "payload": "{\"key1\":\"value1\", \"key2\":\"value2\"}"


  "dataset_id": "abfe-124134-dfhdfg-4486784",
  "operational": {
    "timing": [
        "startTime": "2019-05-06T19:51:51+00:00",
        "endTime": "2019-05-06T19:51:52+00:00",
        "app": "Voltron",
        "label": "Elixir.Voltron.Json.Trim"
  "payload": "{\"key1\":\"value1\", \"key2\":\"value2\"}"

Running Locally

You can use Divo to stand up the external dependencies locally using docker and docker-compose.

MIX_ENV=integration mix docker.start
MIX_ENV=integration iex -S mix


Running the Unit Tests

mix test

Running the Integration Tests

mix test.integration


Released under Apache 2 license.

You can’t perform that action at this time.