This repo contains the following:
- Docker files to create
- Source code for all
- Examples for how to use
atomin creating elements and apps.
- Source documentation for the Atom Docs Site
- A Wiki Page which contains links to github and docker repos for all known elements that have been released.
Please see The Atom Documentation for a comprehensive set of docs that cover the following:
- Design goals and SDK features
- The full Atom specification
- Tutorials on how to use Atom
- Tutorials on Docker and Docker-Compose
- Documentation for elements that have been added to the Atom ecosystem.
The documentation is rebuilt nightly on CircleCI and, when built, pulls the latest documentation from each element that it's linked to. To add an element to the documentation build, simply add the line below to the docs Dockerfile
COPY --from=$dockerhub_org/$dockerhub_repo /code/README.md /elements_docs/$unique_name.md
Please help us keep the documentation up-to-date and accurate! You'll find a
docker-compose file in the
doc folder that will build and launch a local copy of the docs server that can be used to develop and test documentation updates. Once launched, the server can be accessed at
It's quite easy to get up and running and using Atom! Atom ships pre-built Docker images with all of the language clients pre-installed.
Releases + Repos
Production builds are pushed to Dockerhub repositories to match the releases pushed on this repo, i.e. if there's release
vX.Y.Z there will be tags on all of the following repositories:
||Atom images to build elements atop|
||Nucleus images to use as the core of Atom|
||Documentation repo which spins up a webserver with the docs|
||Metrics image which contains Grafana and runs the Atom dashboards|
||General-purpose style formatter containing
Intel + ARM
Atom is built for both 64-bit intel and 64-bit ARM. All docker tags on the following repositories: are multi-arch manifests which are built for both
The following types/flavors of atom are built with each release and are pushed to atom and nucleus
||Base atom. Atom + all dependencies|
||Adds in openCV + other CV libraries atop base Atom.|
||Adds in CUDA support atop CV Atom|
||Adds in VNC support atop CV Atom|
Building your first element
Check out the Atom Walkthrough in the docs!
Redis Talks and Slides
Please see the Redis Talks and Slides page of the wiki for video and slides from the talks that have been given on Atom at various Redis conferences.
Developing for Atom
It's always a good idea to ensure your submodules are up-to-date. Run this command after cloning the repo and/or updating your branch
git submodule update --init --recursive
There are two ways to develop for Atom:
- Link your python application-level code into a prebuilt Atom docker image
- Re-build the Atom docker image
Most client-level changes should be able to use path #1. It's significantly easier, and the test/build process is simpler. The quickstart section here will cover path #1, see the Building Docker Images section for more detail on how to rebuild the container.
To launch the latest production Atom images with the code from your local repos linked in for quick development, run:
docker-compose pull docker-compose up -d
You can then run
docker exec -it atom bash
To pull up a shell in the atom docker image and go ahead developing from there for your language of choice. The code from this local repository will be linked into the container at
/atom, so you can do things (within the container) like:
cd /atom/languages/python && python setup.py install && pytest
cd /atom/languages/cpp && make install && make test
To stop the containers
docker-compose down -t0
Atom comes built-in with a very rich set of metrics reporting and dashboards. Out-of-the-box, there are dashboards for:
- A home/overview dashboard with system info and quick links
- Atom Nucleus Redis
- Atom Metrics Redis
- System Details (CPU, Memory, Disk, Network, etc.)
- Each element. Atom auto-discovers elements and makes a dashboard for each API call (counts, timing, etc.)
The dashboards are powered through Atom's tie-ins to Redis Time Series and Grafana.
Each Dashboard is explained in a bit of detail below.
To run these dashboards, simply run:
docker-compose pull docker-compose up -d
And then pull up Grafana in a local browser by going to http://localhost:3001.
Grafana will prompt you for a username and password. By default, it's
admin. You can then change it to whatever you'd like.
NOTE: The grafana database is reset on each docker-compose up/down, so any changes you make will be lost. See Updating Dashboards for information on how to make dashboard changes
This dashboard has some high-level system info (CPU, Memory, Disk, Network) and links to the other dashboards that Atom auto-generates.
There is a link to each of the Nucleus, Metrics and System dashboards, as well as an updating display with a link to each found Element's dashboard. There is a process running in the metrics container which scans for new elements every now and then and auto-creates dashboards when they are discovered. They'll then show up here.
Atom Nucleus Redis
Shows the state of the redis that all of the atom commands and streams run through. Check here for atom memory usage, any slow queries, and to get an idea of data transfer volume.
Atom Metrics Redis
Shows the state of the redis that all of the atom metrics run through. Check here for metrics redis memory usage, any slow queries, and to get an idea of data transfer volume.
Contains detailed dashboards for CPU, Memory, Disk, Network, Temperature and other system-level stats. Like
htop, but persisted with raw samples for 1hr and MIN/MAX/AVG aggregation over 1m intervals for 24hrs
Each Element has a dashboard which covers most of the Atom APIs (when implemented using Python) to show data volume, timing, and other useful things!
After working through the quickstart, it's recommended to try rebuilding the Atom docker images in order to test changes. Following that, it's recommended to check out the guide to making your first element
Contributions of issues and pull requests are welcome!
Building Docker Images
Atom depends on a few third-party dependencies which are all included as submodules in these repos. From the top-level in this repo, run:
git submodule update --init --recursive
This will pull all of the proper dependencies.
Set up buildx
We want to use the buildkit/buildx frontend for Docker, as this gives us some nifty features we rely on for the atom build.
We first want to enable using the buildx builder
The commands below will install the expected version (0.8.3) and set it as the default buildx version
docker buildx ls | grep "build-v0.8.3" || docker buildx create --name build-v0.8.3 --driver docker-container --driver-opt image=moby/buildkit:v0.8.3,network=host docker buildx use --default build-v0.8.3 docker buildx inspect --bootstrap
Building all docker images
There are four docker images to build for this repository:
If we look at
docker-compose-build.yml, it expects us to have a version of each image tagged
local-dev. The commands in this section will allow you to build all of these
We do not use
docker-compose build here, and instead build each of the tags directly. This is because
docker-compose build does not have good support for the
buildx frontend and all of its options as of yet.
The primary docker images built in this repo are
formatter are helper images that serve minor roles. As such, more info/documentation is needed for the
nucleus builds, so the following sections below apply only to the
- Docker Image Bases
- Build Argumens
Docker Image Bases
Atom has the following bases available. A base is a stage in the multi-stage Dockerfile which we then add the atom application-level code atom to create the output image. With the new
buildx frontend, we can selectively apply the final
atom stage of the multi-stage Dockerfile atop any other build stage in the Dockerfile which is pretty nifty. The table below describes the base stages which can be used
||Bare minimum requirements for Atom|
||Adds in openCV and other CV processing libraries atop
||Adds in NoVNC atop
||Adds in additional CUDA dependencies atop
You can pass the following build arguments to the build. You'll see them in each of the pre-compiled build commands below and you can override/edit as you'd like.
The build arguments primarily control which docker image we should begin our atom build with, and which base from the table above we should build before putting the application-level Atom code atop it.
||Which image to build from. Should work with most things debian-based|
||Which stage of the Dockerfile to use as the "base" for Atom. Should be an option from the Bases Table table. Docker will build this stage and than put the
You can choose to build multiple targets from the
Dockerfile. Each possible target is explained
We default to the "test" stage for atom in the below build command to add in test dependencies. If you want to change to a production version, you can change the target to "atom".
||Production build of atom. All dependencies, libraries, and atom utilities installed. Everything else stripped out|
||Production build of nucleus. All dependencies, libraries and atom utilities installed. Everything else stripped out|
||Pre-production build. Contains everything in production and all source used to build it|
||Production build of atom plus test dependencies/utilities|
Run the command below to build the nucleus. It will leverage/pull the most recent build cache which will likely save you some time
docker buildx build \ -f Dockerfile \ -t elementaryrobotics/nucleus:local-dev \ --pull \ --cache-from type=registry,ref=elementaryrobotics/atom:cache-x86_64 \ --build-arg STOCK_IMAGE=ubuntu:bionic-20210827 \ --build-arg ATOM_BASE=atom-base \ --load \ --progress=plain \ --target=nucleus \ .
At the end of this command you shold have a local image,
elementaryrobotics/nucleus:local-dev you can see from running
docker image ls
The below command will build the base atom and tag it as
elementaryrobotics/atom:local-dev. See the subsections below for the commands/modifications to build either the CV or CUDA atom variants.
docker buildx build \ -f Dockerfile \ -t elementaryrobotics/atom:local-dev \ --pull \ --cache-from type=registry,ref=elementaryrobotics/atom:cache-x86_64 \ --build-arg STOCK_IMAGE=ubuntu:bionic-20210827 \ --build-arg ATOM_BASE=atom-base \ --load \ --progress=plain \ --target=test \ .
docker buildx build \ -f Dockerfile \ -t elementaryrobotics/atom:local-dev \ --pull \ --cache-from type=registry,ref=elementaryrobotics/atom:cache-cv-x86_64 \ --build-arg STOCK_IMAGE=ubuntu:bionic-20210827 \ --build-arg ATOM_BASE=atom-base-cv \ --load \ --progress=plain \ --target=test \ .
docker buildx build \ -f Dockerfile \ -t elementaryrobotics/atom:local-dev \ --pull \ --cache-from type=registry,ref=elementaryrobotics/atom:cache-vnc-x86_64 \ --build-arg STOCK_IMAGE=ubuntu:bionic-20210827 \ --build-arg ATOM_BASE=atom-base-vnc \ --load \ --progress=plain \ --target=test \ .
docker buildx build \ -f Dockerfile \ -t elementaryrobotics/atom:local-dev \ --pull \ --cache-from type=registry,ref=elementaryrobotics/atom:cache-cuda-x86_64 \ --build-arg STOCK_IMAGE=nvcr.io/nvidia/cuda:10.2-cudnn8-devel-ubuntu18.04 \ --build-arg ATOM_BASE=atom-base-cuda \ --load \ --progress=plain \ --target=test \ .
docker buildx build \ -f metrics/Dockerfile \ -t elementaryrobotics/metrics:local-dev \ --pull \ --load \ --cache-from type=registry,ref=elementaryrobotics/metrics:cache-x86_64 \ --progress plain \ .
cd utilities/formatting && docker buildx build \ -f Dockerfile \ -t elementaryrobotics/formatter:local-dev \ --pull \ --load \ --cache-from type=registry,ref=elementaryrobotics/formatter:cache-x86_64 \ --progress plain \ .
Launching built containers
Now that we've built all of our images, we can run them with
docker-compose -f docker-compose.yml -f docker-compose-build.yml up -d
You should see everything launch/run normally. If you see any errors similar to:
ERROR: manifest for elementaryrobotics/metrics:local-dev not found: manifest unknown: manifest unknown
It means you don't have the locally built version to launch with. Please see the section for building that image and re-run the command. It may never have been run before or the image may have been deleted.
docker-compose -f docker-compose.yml -f docker-compose-build.yml down -t0
If you want to rebuild, your flow should be
- Stop containers
- Rebuild image of interest with command from above (atom/nucleus/metrics/formatter)
- Re-launch containers
In order to run tests, you'll want to develop with the "Rebuilding the Docker
Images" method as described above. Once you're in the
atom container, you
can run the following tests:
|Python Unit Tests||
|C Unit Tests||
|C++ Unit Tests||
|C++ Memory Check||
Formatting + Linting
Formatting + Linting checks are run in our CI/CD pipeline. Code will not be able
to be merged into
latest without passing.
To set up type checking and code completion in your text editor, see the wiki.
Formatting for Atom
Atom utilizes the following formatting guidelines:
blackfor formatting. All code must pass
- Atop black, follow the Elementary Robotics company-wide
.flake8. This is more strict than black and catches more things than black can/will catch since black only deals with stylistic errors and not the full set of errors that can be caught with good linting.
We have created a Docker container that comes
To check your formatting:
docker-compose run formatting
This, by default, will:
- Build the formatting container (if not already built)
- Launch the formatting container
- Return an exit code of 0 if all code passes, else an exit code of 1 if anything failed
If you'd like to configure the formatter to auto-format and then do the
check you can do so by adding
-e DO_FORMAT=y to the command:
docker-compose run -e DO_FORMAT=y formatting
This will add a step to call
black between steps (2) and (3) of the above list
but otherwise run the same process.
This repo contains a Dockerfile and builds a general-purpose formatter image that can be used to apply the style formatting/checks in this repo anywhere you write code.
To check your code:
docker run -e BLACK_EXCLUDE="" -e FLAKE8_EXCLUDE="" --mount src=$(pwd),target=/code,type=bind elementaryrobotics/formatter
After running this, check the return code with
echo $? to see if the checks passed. If the return code is 0, then the checks passed, if nonzero then there were formatting inconsistencies. The logs will also be indicative.
To reformat your code:
docker run -e DO_FORMAT=y -e BLACK_EXCLUDE="" -e FLAKE8_EXCLUDE="" --mount src=$(pwd),target=/code,type=bind elementaryrobotics/formatter
Build checks are set up s.t. the formatter must pass for code to be merged in.
The formatter exposes the following command-line options which are used in the run script
||""||If non-empty, run the auto-formatter for all C/C++ code mounted to the
||""||If non-empty, i.e. not "", run the auto-formatter before running the formatting check. Default is to not run the auto-formatter|
||"y"||Use Black as the auto-formatter of choice. This is default, but we may add other auto-formatters in the future. Set to empty to not use black.|
||"y"||If non-empty, i.e. not "", run the formatting/linting check automatically. This is the default.|
||""||If non-empty, instead of returning when finished, hang the container. This is nice if you want to then shell in to the container and play around with the formatter, but otherwise not that useful|
CLANG_FORMAT option to recursively auto-format C/C++ code mounted from
the current directory. Replace
src=$(pwd) to format a different directory.
docker run -e CLANG_FORMAT=y --mount src=$(pwd),target=/code/src,type=bind elementaryrobotics/formatter
Please see the READMEs in the individual language folders for more information about the language clients.