The Things Network Stack for LoRaWAN Development
The Things Network Stack components are primarily built in Go, while we use Node for web front-ends. It is assumed that you have decent knowledge and experience with these technologies. If you want to get more familiar with Go, we strongly recommend to take A Tour of Go.
The Things Network's development environment heavily relies on
make. Under the hood,
make calls other tools such as
yarn etc. Recent versions are supported; Node v10.x and Go v1.12.x. Let's first make sure you have
On macOS using Homebrew:
brew install go node yarn
On Ubuntu (or on Windows using the Windows Subsystem for Linux):
curl -sL https://deb.nodesource.com/setup_10.x | sudo -E bash - sudo apt-get install -y build-essential nodejs curl -sSL https://dl.google.com/go/go1.12.3.linux-amd64.tar.gz | sudo tar -xz -C /usr/local sudo ln -s /usr/local/go/bin/* /usr/local/bin
Cloning the repository
If you are unfamiliar with forking projects on GitHub or cloning them locally, please see the GitHub documentation.
As most of the tasks will be managed by
mage we will first initialize the tooling. You may want to run this commands from time to time:
For convenience, you can initialize the development databases with some defaults.
Note: this requires Docker Desktop.
This starts a CockroachDB and Redis database in Docker containers, creates a database, migrates tables and creates a user
admin with password
Managing the development databases
You can also use the following commands to start, stop and erase databases.
make dev.databases.start # Starts all databases in a Docker container make dev.databases.stop # Stops all databases # The contents of the databases will be saved in .dev/data. make dev.databases.erase # Stop all databases and erase storage.
CockroachDB is a distributed SQL database that we use in the Identity Server.
You can use
make dev.databases.sql to enter an SQL shell.
Redis is an in-memory data store that we use as a database for "hot" data.
You can use
make dev.databases.redis-cli to enter a Redis-CLI shell.
./mage go:test js:test jsSDK:test
There is a single binary for the server,
ttn-lw-stack, as well as a binary for the command-line interface
ttn-lw-cli. The single binary contains all components start one or multiple components. This allows you to run the stack with one command in simple deployment scenarios, as well as distributing micro-services for more advanced scenarios.
We provide binary releases for all supported platforms, including packages for various package managers at https://github.com/TheThingsNetwork/lorawan-stack/releases. We suggest you use the compiled packages we provide in production scenarios.
For development/testing purposes we suggest either running required binaries via
go run (e.g.
go run ./cmd/ttn-lw-cli from repository root for CLI), or using
go build directly. Note, that the frontend (if used) needs to be built via
./mage js:build before
go build or
go run commands are run.
If you want, you can build a release snapshot with
This will compile binaries for all supported platforms,
rpm and Snapcraft packages, release archives in
dist, as well as Docker images.
Note: The operating system and architecture represent the name of the directory in
distin which the binaries are placed. For example, the binaries for Darwin x64 (macOS) will be located at
Releasing a new version consists of the following steps:
- Bumping the version
- Writing the version files
- Creating the version bump commit
- Creating the version tag
- Building the release and pushing to package managers (this is done by CI)
- Edit the release notes
Our development tooling helps with this process. The
mage command has the following commands for version bumps:
version:bumpMajor bumps a major version (from 3.4.5 -> 4.0.0). version:bumpMinor bumps a minor version (from 3.4.5 -> 3.5.0). version:bumpPatch bumps a patch version (from 3.4.5 -> 3.4.6). version:bumpRC bumps a release candidate version (from 3.4.5-rc1 -> 3.4.5-rc2). version:bumpRelease bumps a pre-release to a release version (from 3.4.5-rc1 -> 3.4.5).
These bumps can be combined (i.e.
version:bumpMinor version:bumpRC bumps 3.4.5 -> 3.5.0-rc1). Apart from these bump commands, we have commands for writing version files (
version:files), creating the bump commit (
version:commitBump) and the version tag (
A typical release process is executed directly on the
master branch and looks like this:
mage version:bumpPatch version:files version:commitBump version:tag // bump, write files, commit and tag. git push origin $(mage version:current) // push the tag git push origin master // push the master branch
Note that you must have sufficient repository rights to push to
After pushing the tag, our CI system will start building the release. When this is done, you'll find a new release on the releases page. After this is done, you'll need to edit the release notes. The release process will do its best to generate release notes for us, but they typically require a bit of editing.
Note: If you don't work on changes in the API you can skip this section.
Our APIs are defined in
.proto files in the
api folder. These files describe the messages and interfaces of the different components of the Stack. If this is the first time you hear the term "protocol buffers" you should probably read the protocol buffers documentation before you continue.
.proto files, we generate code using the
protoc compiler. As we plan to compile to a number of different languages, we decided to put the compiler and its dependencies in a Docker image, so make sure you have Docker installed before you try to compile them.
The actual commands for compilation are handled by our Makefile, so the only thing you have to execute, is:
./mage proto:clean proto:all
The Things Network Stack for LoRaWAN includes two frontend applications: the Console and OAuth Provider. Both applications use React as frontend framework. The
oauth packages of the backend expose their respective web servers and handle all logic that cannot be done in the browser. Otherwise both applications are single page applications (SPA) that run entirely in the browser.
The Console is the official management application of the stack. It can be used to register applications, end devices or gateways, monitor network traffic, or configure network related options, among other things. The console uses an OAuth access token to communicate with the stack.
The OAuth app provides the necessary frontend for the OAuth provider of the stack. It is used e.g. to display the authorization screen that users get prompted with when they want to authorize a third-party app to access the stack.
Building the frontend
You can control whether to build the frontend for production or development by setting the
$NODE_ENV environment variable to either
production. The frontend can then be built using:
This will initiate the following actions:
- Install a version-fixed binary of
- Install JS SDK node dependencies via
- Build the JS SDK
- Extract backend locale messages
- Install frontend node dependencies via
- Build the frontend (using
webpack) and output into
The difference of a development build includes:
- Including source maps
- Using DLL bundle for modules to reduce build time
- A couple of special build options to improve usage with
After successfully running the build command, the stack has all necessary files to run the Console and OAuth provider applications.
Serving the frontend for development
For development purposes, the frontend can be run using
webpack-dev-server. After following the Getting Started section to initialize the stack and doing an initial build of the frontend via
mage js:build, it can be served using:
export NODE_ENV=development mage js:serve
The development server runs on
http://localhost:8080 and will proxy all api calls to port
1885. The serve command watches any changes inside
pkg/webui and refreshes automatically.
In order to set up the stack to support running the frontend via
webpack-dev-server, the following environment setup is needed:
NODE_ENV=development TTN_LW_LOG_LEVEL=debug TTN_LW_IS_OAUTH_UI_JS_FILE=libs.bundle.js,oauth.js TTN_LW_CONSOLE_UI_JS_FILE=libs.bundle.js,console.js TTN_LW_CONSOLE_UI_CANONICAL_URL=http://localhost:8080/console TTN_LW_CONSOLE_OAUTH_AUTHORIZE_URL=http://localhost:8080/oauth/authorize TTN_LW_CONSOLE_OAUTH_TOKEN_URL=http://localhost:8080/oauth/token TTN_LW_IS_OAUTH_UI_CANONICAL_URL=http://localhost:8080/oauth TTN_LW_IS_EMAIL_NETWORK_IDENTITY_SERVER_URL=http://localhost:8080/oauth.js TTN_LW_CONSOLE_UI_ASSETS_BASE_URL=http://localhost:8080/assets
Note: We recommend using an environment switcher like
direnv to help you setting up environments for different tasks easily.
All of the configuration options above can also be set using configuration files or runtime flags. For more info in this regard, see this guide.
For frontend testing, we use
jest. We currently don't enforce any coverage minimum, but consider testing for complex logic. We use both snapshot testing for React Components with
enzyme and plain
jest testing for arbitrary logic.
To run the frontend tests, use:
The Things Network Stack for LoRaWAN employs i18n to provide usage experience in different languages. As such, also the frontend uses translatable messages. For this purpose we use
react-intl, which helps us greatly to define text messages used in the frontend.
The workflow for defining messages is as follows:
- Add a
- This can be done either inline, or by adding it to
- Use the message in components (e.g.
After adding messages this way, it needs to be added the locales file
pkg/webui/locales/*.js by using:
Note: When using
mage js:serve, this command will be run automatically after any change
The message definitions in
pkg/webui/locales can be used to provide translations in other languages (e.g.
fr.js). Keep in mind that locale files are checked in and commited, any discrepancy in the locales file with the defined messages will lead to a CI failure.
Frontend Folder Structure
. ├── assets assets (eg. vectors, images) used by the frontend ├── components react components shared throughout the frontend ├── console root of the console application │ ├── api api definitions to communicate with the stack │ ├── containers container components │ ├── lib utility classes and functions │ ├── store redux actions, reducers and logic middlewares │ ├── views whole view components of the console (~pages) ├── containers global react container components ├── lib global utility classes and functions ├── locales frontend and backend locale jsons used for i18n ├── oauth root of the oauth application │ ├── api api definitions to communicate with the stack │ ├── store redux actions, reducers and logic middlewares │ ├── views whole view components of the oauth provider (~pages) ├── styles global stylus (~css) styles and mixins ├── console.js entry point of the console app ├── oauth.js entry point of the oauth app ├── manifest.go generated manifest of the frontend, containing file hashes ├── template.go go template module used to render the frontend HTML