Skip to content
A conglomeration of (data) transformations
Branch: master
Clone or download
bbrewer-papa-x Merge pull request #34 from smartcitiesdata/smrt-1287-static-version
SMRT-1287 - preparing for a release bump that doesn't touch this
Latest commit 97711e0 Jun 11, 2019
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
config
lib/voltron
rel
scripts
test
.formatter.exs SMRT-1126 Refactor voltron to do field based transformations and adde… Apr 18, 2019
.gitignore Add observer_cli and add .elixir_ls to gitignore May 24, 2019
.travis.yml
Dockerfile Adding bash dependency to dockerfile Jun 11, 2019
LICENSE Adding liscence Jun 11, 2019
README.md
mix.exs SMRT-1287 - preparing for a release bump that doesn't touch this Jun 11, 2019
mix.lock SMRT-1314 Refactoring tests to new public api Jun 6, 2019

README.md

Master

Voltron

Applies a standard set of transformations to all data messages received and delivers them to a configurable topic

External Dependencies

  • Kafka (input topic, output topic, dead letter queue (DLQ) topic)
  • Redis

Configuration

The following environment variables should be set:

  • FROM_TOPIC - The Kafka topic that data messages will be pulled from
  • TO_TOPIC - The Kafka topic that transformed data messages will be delivered to
  • REDIS_HOST - The redis host (e.g. 'redis.my.domain.com') where dataset definitions are stored. Currently this data is used to retrieve the dataset schema. In the future it will be used to determine what transformations should be applied to a particular dataset's data message
  • KAFKA_BROKERS - A comma delimited list of one or more Kafka brokers in the form of host1:port,host2:port

Examples

Input

Input messages must be in the format defined by the Smart Cities Data Library

{
  "dataset_id": "abfe-124134-dfhdfg-4486784",
  "operational": {},
  "payload": "{\"key1\":\"value1\", \"key2\":\"value2\"}"
}

Output

{
  "dataset_id": "abfe-124134-dfhdfg-4486784",
  "operational": {
    "timing": [
      {
        "startTime": "2019-05-06T19:51:51+00:00",
        "endTime": "2019-05-06T19:51:52+00:00",
        "app": "Voltron",
        "label": "Elixir.Voltron.Json.Trim"
      }
    ]
  },
  "payload": "{\"key1\":\"value1\", \"key2\":\"value2\"}"
}

Running Locally

You can use Divo to stand up the external dependencies locally using docker and docker-compose.

MIX_ENV=integration mix docker.start
MIX_ENV=integration iex -S mix

Testing

Running the Unit Tests

mix test

Running the Integration Tests

mix test.integration

License

Released under Apache 2 license.

You can’t perform that action at this time.