Snowplow Micro

Snowplow Micro is built to enable companies running Snowplow to build automated CI/CD workflows ensuring upstream data quality.

It is a very small version of a full Snowplow data collection pipeline: small enough that it can be launched by an automated test suite. Snowplow Micro then exposes an API, that can be queried in order to validate your data collection setup as your digital products evolve.

Quick start

Using Docker

Snowplow Micro is hosted on Docker Hub (snowplow/snowplow-micro) with images available for both amd64 and arm64 architectures.

The following command can be used to run Micro with the default configuration:

docker run -p 9090:9090 snowplow/snowplow-micro:1.3.2

A distroless image is also provided, which benefits from being smaller and more secure, with the downside of basic utilities (such as a shell) not being present in the image.

The distroless image is tagged as snowplow/snowplow-micro:1.3.2-distroless on Docker Hub.

Configuring Micro

Stream Collector config

Stream Collector collects the events sent by Snowplow trackers. The options for configuring the collector are available here. A minimal example of this file exists at example/micro.conf

Iglu resolver config

The Iglu resolver is used to define where schemas are fetched from. The instructions for configuration are available here. An example of this file exists at example/iglu.json

Running Micro with configuration files

The configuration files must be placed in a folder that is mounted in the Docker container, and the port configured for Micro needs to be exposed. The following examples assume that the configuration files are in the directory ./example/ and that port 9090 is the one to bind on your host machine:

docker run \
  --mount type=bind,source=$(pwd)/example,destination=/config \
  -p 9090:9090 \
  snowplow/snowplow-micro:1.3.2 \
  --collector-config /config/micro.conf \
  --iglu /config/iglu.json

In order to use the embedded Iglu capabilities, the command is the same as above. You only need to add your local Iglu repository in the configurations directory named as iglu-client-embedded, as also shown in the example:

example
├── iglu-client-embedded
│   └── schemas
│       └── com.myvendor
│           └── myschema
│               └── jsonschema
│                   └── 1-0-0
├── iglu.json
└── micro.conf

Using Java

If you cannot use Docker, a Snowplow Micro jar file is hosted on the Github releases. As an example, run Micro as:

java -jar snowplow-micro-1.3.2.jar --collector-config example/micro.conf --iglu example/iglu.json

In case you also want an embedded Iglu, apply the same directory structure as shown above and run for example:

# Unix
java -cp snowplow-micro-1.3.2.jar:example com.snowplowanalytics.snowplow.micro.Main --collector-config example/micro.conf --iglu example/iglu.json
# Windows
java -cp snowplow-micro-1.3.2.jar;example com.snowplowanalytics.snowplow.micro.Main --collector-config example/micro.conf --iglu example/iglu.json

Send events and start testing

Once you have successfully started Snowplow Micro, its collector endpoint will be http://localhost:{PORT}, e.g. http://localhost:9090, to which you can point any of the Snowplow Trackers and start sending events.

Then, in your testing workflows, you can use Micro's REST API endpoints to assert on the collected events:

• /micro/all: summary
• /micro/good: good events
• /micro/bad: bad events
• /micro/reset: clears cache

Additionally, if you are using the embedded Iglu capabilities, you can also use the /micro/iglu endpoint to check whether your custom schemas can be resolved as:

• /micro/iglu/{vendorName}/{schemaName}/jsonschema/{schemaVersion} (for example /micro/iglu/com.myvendor/myschema/jsonschema/1-0-0)

Find out more

Technical Docs	Roadmap	Contributing
Technical Docs	Roadmap	Contributing

Maintainer quick start

Assuming Git and sbt:

git clone git@github.com:snowplow-incubator/snowplow-micro-examples.git
cd snowplow-micro

git clone --branch 2.7.1 --depth 1 git@github.com:snowplow/stream-collector.git
cd stream-collector
sbt publishLocal && cd ..

sbt test

Copyright and License

Snowplow Micro is copyright 2019-2022 Snowplow Analytics Ltd.

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this software except in compliance with the License.

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.