Overview
This repo contains the following:
- Docker files to create
atom
andnucleus
docker images - Source code for all
atom
language libraries - Examples for how to use
atom
in creating elements and apps. - Source documentation for the Atom Docs Site
- A Wiki Page which contains links to github and docker repos for all known elements that have been released.
Documentation
Please see The Atom Documentation for a comprehensive set of docs that cover the following:
- Design goals and SDK features
- The full Atom specification
- Tutorials on how to use Atom
- Tutorials on Docker and Docker-Compose
- Documentation for elements that have been added to the Atom ecosystem.
The documentation is rebuilt nightly on CircleCI and, when built, pulls the latest documentation from each element that it's linked to. To add an element to the documentation build, simply add the line below to the docs Dockerfile
COPY --from=$dockerhub_org/$dockerhub_repo /code/README.md /elements_docs/$unique_name.md
Please help us keep the documentation up-to-date and accurate! You'll find a docker-compose
file in the doc
folder that will build and launch a local copy of the docs server that can be used to develop and test documentation updates. Once launched, the server can be accessed at localhost:4567
.
Using Atom
It's quite easy to get up and running and using Atom! Atom ships pre-built Docker images with all of the language clients pre-installed.
Releases + Repos
Production builds are pushed to Dockerhub repositories to match the releases pushed on this repo, i.e. if there's release vX.Y.Z
there will be tags on all of the following repositories:
Repository | Description |
---|---|
elementaryrobotics/atom |
Atom images to build elements atop |
elementaryrobotics/nucleus |
Nucleus images to use as the core of Atom |
elementaryrobotics/atom-doc |
Documentation repo which spins up a webserver with the docs |
elementaryrobotics/metrics |
Metrics image which contains Grafana and runs the Atom dashboards |
elementaryrobotics/formatter |
General-purpose style formatter containing black , flake8 , isort , and others |
Intel + ARM
Atom is built for both 64-bit intel and 64-bit ARM. All docker tags on the following repositories: are multi-arch manifests which are built for both linux/amd64
and linux/arm64
:
Atom Types
The following types/flavors of atom are built with each release and are pushed to atom and nucleus
Tag | Description |
---|---|
vX.Y.Z |
Base atom. Atom + all dependencies |
vX.Y.Z-cv |
Adds in openCV + other CV libraries atop base Atom. |
vX.Y.Z-cuda |
Adds in CUDA support atop CV Atom |
vX.Y.Z-vnc |
Adds in VNC support atop CV Atom |
Building your first element
Check out the Atom Walkthrough in the docs!
Redis Talks and Slides
Please see the Redis Talks and Slides page of the wiki for video and slides from the talks that have been given on Atom at various Redis conferences.
Developing for Atom
Update Submodules
It's always a good idea to ensure your submodules are up-to-date. Run this command after cloning the repo and/or updating your branch
git submodule update --init --recursive
Quickstart
There are two ways to develop for Atom:
- Link your python application-level code into a prebuilt Atom docker image
- Re-build the Atom docker image
Most client-level changes should be able to use path #1. It's significantly easier, and the test/build process is simpler. The quickstart section here will cover path #1, see the Building Docker Images section for more detail on how to rebuild the container.
To launch the latest production Atom images with the code from your local repos linked in for quick development, run:
docker-compose pull
docker-compose up -d
You can then run
docker exec -it atom bash
To pull up a shell in the atom docker image and go ahead developing from there for your language of choice. The code from this local repository will be linked into the container at /atom
, so you can do things (within the container) like:
cd /atom/languages/python && python setup.py install && pytest
cd /atom/languages/cpp && make install && make test
To stop the containers
docker-compose down -t0
Dashboards
Atom comes built-in with a very rich set of metrics reporting and dashboards. Out-of-the-box, there are dashboards for:
- A home/overview dashboard with system info and quick links
- Atom Nucleus Redis
- Atom Metrics Redis
- System Details (CPU, Memory, Disk, Network, etc.)
- Each element. Atom auto-discovers elements and makes a dashboard for each API call (counts, timing, etc.)
The dashboards are powered through Atom's tie-ins to Redis Time Series and Grafana.
Each Dashboard is explained in a bit of detail below.
To run these dashboards, simply run:
docker-compose pull
docker-compose up -d
And then pull up Grafana in a local browser by going to http://localhost:3001.
Grafana will prompt you for a username and password. By default, it's admin
and admin
. You can then change it to whatever you'd like.
NOTE: The grafana database is reset on each docker-compose up/down, so any changes you make will be lost. See Updating Dashboards for information on how to make dashboard changes
Home Dashboard
This dashboard has some high-level system info (CPU, Memory, Disk, Network) and links to the other dashboards that Atom auto-generates.
There is a link to each of the Nucleus, Metrics and System dashboards, as well as an updating display with a link to each found Element's dashboard. There is a process running in the metrics container which scans for new elements every now and then and auto-creates dashboards when they are discovered. They'll then show up here.
Atom Nucleus Redis
Shows the state of the redis that all of the atom commands and streams run through. Check here for atom memory usage, any slow queries, and to get an idea of data transfer volume.
Atom Metrics Redis
Shows the state of the redis that all of the atom metrics run through. Check here for metrics redis memory usage, any slow queries, and to get an idea of data transfer volume.
System Details
Contains detailed dashboards for CPU, Memory, Disk, Network, Temperature and other system-level stats. Like htop
, but persisted with raw samples for 1hr and MIN/MAX/AVG aggregation over 1m intervals for 24hrs
Elements
Each Element has a dashboard which covers most of the Atom APIs (when implemented using Python) to show data volume, timing, and other useful things!
Next Steps
After working through the quickstart, it's recommended to try rebuilding the Atom docker images in order to test changes. Following that, it's recommended to check out the guide to making your first element
Contributing
Contributions of issues and pull requests are welcome!
Building Docker Images
Update Submodules
Atom depends on a few third-party dependencies which are all included as submodules in these repos. From the top-level in this repo, run:
git submodule update --init --recursive
This will pull all of the proper dependencies.
Set up buildx
We want to use the buildkit/buildx frontend for Docker, as this gives us some nifty features we rely on for the atom build.
We first want to enable using the buildx builder
export DOCKER_CLI_EXPERIMENTAL=enabled
The commands below will install the expected version (0.8.3) and set it as the default buildx version
docker buildx ls | grep "build-v0.8.3" || docker buildx create --name build-v0.8.3 --driver docker-container --driver-opt image=moby/buildkit:v0.8.3,network=host
docker buildx use --default build-v0.8.3
docker buildx inspect --bootstrap
Building all docker images
There are four docker images to build for this repository:
atom
nucleus
metrics
formatter
If we look at docker-compose-build.yml
, it expects us to have a version of each image tagged local-dev
. The commands in this section will allow you to build all of these local-dev
images.
We do not use docker-compose build
here, and instead build each of the tags directly. This is because docker-compose build
does not have good support for the buildx
frontend and all of its options as of yet.
The primary docker images built in this repo are atom
and nucleus
. metrics
and formatter
are helper images that serve minor roles. As such, more info/documentation is needed for the atom
and nucleus
builds, so the following sections below apply only to the atom
and nucleus
builds:
- Docker Image Bases
- Build Argumens
Docker Image Bases
Atom has the following bases available. A base is a stage in the multi-stage Dockerfile which we then add the atom application-level code atom to create the output image. With the new buildx
frontend, we can selectively apply the final atom
stage of the multi-stage Dockerfile atop any other build stage in the Dockerfile which is pretty nifty. The table below describes the base stages which can be used
Name | Description |
---|---|
atom-base |
Bare minimum requirements for Atom |
atom-base-cv |
Adds in openCV and other CV processing libraries atop atom-base |
atom-base-vnc |
Adds in NoVNC atop atom-base-cv |
atom-base-cuda |
Adds in additional CUDA dependencies atop atom-base-cv |
Build Arguments
You can pass the following build arguments to the build. You'll see them in each of the pre-compiled build commands below and you can override/edit as you'd like.
The build arguments primarily control which docker image we should begin our atom build with, and which base from the table above we should build before putting the application-level Atom code atop it.
Arg | Default | Description |
---|---|---|
STOCK_IMAGE |
ubuntu:bionic-20210827 |
Which image to build from. Should work with most things debian-based |
ATOM_BASE |
atom-base |
Which stage of the Dockerfile to use as the "base" for Atom. Should be an option from the Bases Table table. Docker will build this stage and than put the atom stage atop it for the final image |
Build Targets
You can choose to build multiple targets from the Dockerfile
. Each possible target is explained
below.
We default to the "test" stage for atom in the below build command to add in test dependencies. If you want to change to a production version, you can change the target to "atom".
Target | Description |
---|---|
atom |
Production build of atom. All dependencies, libraries, and atom utilities installed. Everything else stripped out |
nucleus |
Production build of nucleus. All dependencies, libraries and atom utilities installed. Everything else stripped out |
atom-source |
Pre-production build. Contains everything in production and all source used to build it |
test |
Production build of atom plus test dependencies/utilities |
nucleus
Building Run the command below to build the nucleus. It will leverage/pull the most recent build cache which will likely save you some time
docker buildx build \
-f Dockerfile \
-t elementaryrobotics/nucleus:local-dev \
--pull \
--cache-from type=registry,ref=elementaryrobotics/atom:cache-x86_64 \
--build-arg STOCK_IMAGE=ubuntu:bionic-20210827 \
--build-arg ATOM_BASE=atom-base \
--load \
--progress=plain \
--target=nucleus \
.
At the end of this command you shold have a local image, elementaryrobotics/nucleus:local-dev
you can see from running docker image ls
atom
Building The below command will build the base atom and tag it as elementaryrobotics/atom:local-dev
. See the subsections below for the commands/modifications to build either the CV or CUDA atom variants.
docker buildx build \
-f Dockerfile \
-t elementaryrobotics/atom:local-dev \
--pull \
--cache-from type=registry,ref=elementaryrobotics/atom:cache-x86_64 \
--build-arg STOCK_IMAGE=ubuntu:bionic-20210827 \
--build-arg ATOM_BASE=atom-base \
--load \
--progress=plain \
--target=test \
.
atom
Building CV docker buildx build \
-f Dockerfile \
-t elementaryrobotics/atom:local-dev \
--pull \
--cache-from type=registry,ref=elementaryrobotics/atom:cache-cv-x86_64 \
--build-arg STOCK_IMAGE=ubuntu:bionic-20210827 \
--build-arg ATOM_BASE=atom-base-cv \
--load \
--progress=plain \
--target=test \
.
atom
Building VNC docker buildx build \
-f Dockerfile \
-t elementaryrobotics/atom:local-dev \
--pull \
--cache-from type=registry,ref=elementaryrobotics/atom:cache-vnc-x86_64 \
--build-arg STOCK_IMAGE=ubuntu:bionic-20210827 \
--build-arg ATOM_BASE=atom-base-vnc \
--load \
--progress=plain \
--target=test \
.
atom
Building CUDA docker buildx build \
-f Dockerfile \
-t elementaryrobotics/atom:local-dev \
--pull \
--cache-from type=registry,ref=elementaryrobotics/atom:cache-cuda-x86_64 \
--build-arg STOCK_IMAGE=nvcr.io/nvidia/cuda:10.2-cudnn8-devel-ubuntu18.04 \
--build-arg ATOM_BASE=atom-base-cuda \
--load \
--progress=plain \
--target=test \
.
metrics
Building docker buildx build \
-f metrics/Dockerfile \
-t elementaryrobotics/metrics:local-dev \
--pull \
--load \
--cache-from type=registry,ref=elementaryrobotics/metrics:cache-x86_64 \
--progress plain \
.
formatter
Building cd utilities/formatting && docker buildx build \
-f Dockerfile \
-t elementaryrobotics/formatter:local-dev \
--pull \
--load \
--cache-from type=registry,ref=elementaryrobotics/formatter:cache-x86_64 \
--progress plain \
.
Launching built containers
Now that we've built all of our images, we can run them with
docker-compose -f docker-compose.yml -f docker-compose-build.yml up -d
You should see everything launch/run normally. If you see any errors similar to:
ERROR: manifest for elementaryrobotics/metrics:local-dev not found: manifest unknown: manifest unknown
It means you don't have the locally built version to launch with. Please see the section for building that image and re-run the command. It may never have been run before or the image may have been deleted.
Stopping containers
docker-compose -f docker-compose.yml -f docker-compose-build.yml down -t0
Rebuilds
If you want to rebuild, your flow should be
- Stop containers
- Rebuild image of interest with command from above (atom/nucleus/metrics/formatter)
- Re-launch containers
Testing
In order to run tests, you'll want to develop with the "Rebuilding the Docker
Images" method as described above. Once you're in the atom
container, you
can run the following tests:
Test | Working Directory | Command |
---|---|---|
Python Unit Tests | /atom/languages/python/tests |
pytest |
C Unit Tests | /atom/languages/c |
make -j8 test |
C++ Unit Tests | /atom/languages/cpp |
make -j8 test |
C++ Memory Check | /atom/languages/cpp |
valgrind -v --tool=memcheck --leak-check=full --num-callers=40 --log-file=valgrind.log --error-exitcode=1 test/build/test_atom_cpp |
Formatting + Linting
Formatting + Linting checks are run in our CI/CD pipeline. Code will not be able
to be merged into latest
without passing.
Python
To set up type checking and code completion in your text editor, see the wiki.
Formatting for Atom
Atom utilizes the following formatting guidelines:
- Use
black
for formatting. All code must passblack --check
. - Atop black, follow the Elementary Robotics company-wide
.flake8
. This is more strict than black and catches more things than black can/will catch since black only deals with stylistic errors and not the full set of errors that can be caught with good linting.
We have created a Docker container that comes
preinstalled with .flake8
and black
.
To check your formatting:
docker-compose run formatting
This, by default, will:
- Build the formatting container (if not already built)
- Launch the formatting container
- Run
black --check
andflake8
- Return an exit code of 0 if all code passes, else an exit code of 1 if anything failed
If you'd like to configure the formatter to auto-format and then do the .flake8
check you can do so by adding -e DO_FORMAT=y
to the command:
docker-compose run -e DO_FORMAT=y formatting
This will add a step to call black
between steps (2) and (3) of the above list
but otherwise run the same process.
General-purpose Formatting
This repo contains a Dockerfile and builds a general-purpose formatter image that can be used to apply the style formatting/checks in this repo anywhere you write code.
To check your code:
docker run -e BLACK_EXCLUDE="" -e FLAKE8_EXCLUDE="" --mount src=$(pwd),target=/code,type=bind elementaryrobotics/formatter
After running this, check the return code with echo $?
to see if the checks passed. If the return code is 0, then the checks passed, if nonzero then there were formatting inconsistencies. The logs will also be indicative.
To reformat your code:
docker run -e DO_FORMAT=y -e BLACK_EXCLUDE="" -e FLAKE8_EXCLUDE="" --mount src=$(pwd),target=/code,type=bind elementaryrobotics/formatter
Build checks are set up s.t. the formatter must pass for code to be merged in.
The formatter exposes the following command-line options which are used in the run script
Option | Default | Description |
---|---|---|
CLANG_FORMAT |
"" | If non-empty, run the auto-formatter for all C/C++ code mounted to the /code directory. Recursively searches for code for format. Will ignore all other options and exit without formatting python code. |
DO_FORMAT |
"" | If non-empty, i.e. not "", run the auto-formatter before running the formatting check. Default is to not run the auto-formatter |
FORMAT_BLACK |
"y" | Use Black as the auto-formatter of choice. This is default, but we may add other auto-formatters in the future. Set to empty to not use black. |
DO_CHECK |
"y" | If non-empty, i.e. not "", run the formatting/linting check automatically. This is the default. |
DO_HANG |
"" | If non-empty, instead of returning when finished, hang the container. This is nice if you want to then shell in to the container and play around with the formatter, but otherwise not that useful |
C/C++ Formatting
Use the CLANG_FORMAT
option to recursively auto-format C/C++ code mounted from
the current directory. Replace src=$(pwd)
to format a different directory.
docker run -e CLANG_FORMAT=y --mount src=$(pwd),target=/code/src,type=bind elementaryrobotics/formatter
Language-Specific Docs
Please see the READMEs in the individual language folders for more information about the language clients.