- Developer Requirements (Local)
- Developer Requirements (Docker)
- Project Layout
- Running VinylDNS Locally
- Testing
- Java (version: >= 8, <= 11)
- Scala 2.12
- sbt 1.4+
- curl
- docker
- docker-compose
- GNU Make 3.82+
- grunt
- Node.js/npm v12+
- Python 3.5+
- coreutils
- Linux:
apt install coreutils
oryum install coreutils
- macOS:
brew install coreutils
- Linux:
Since almost everything can be run with Docker and GNU Make, if you don't want to setup a local development environment, then you simply need:
Docker
v19.03+ (earlier versions may work fine)Docker Compose
v2.0+ (earlier versions may work fine)GNU Make
v3.82+Bash
3.2+- Basic utilities:
awk
,sed
,curl
,grep
, etc may be needed for scripts
- Basic utilities:
SYSTEM_DESIGN.md provides a high-level architectural overview of VinylDNS and interoperability of its components.
The main codebase is a multi-module Scala project with multiple sub-modules. To start working with the project, from the
root directory run sbt
. Most of the code can be found in the modules
directory. The following modules are present:
root
- this is the parent project, if you run tasks here, it will run against all sub-modulescore
: core modules that are used by both the API and portal, such as cryptography implementations.api
: the API is the main engine for all of VinylDNS. This is the most active area of the codebase, as everything else typically just funnels through the API.portal
: The portal is a user interface wrapper around the API. Most of the business rules, logic, and processing can be found in the API. The only features in the portal not found in the API are creation of users and user authentication.docs
: documentation for VinylDNS.
Code that is used across multiple modules in the VinylDNS ecosystem live in core
.
src/main
- the main source codesrc/test
- unit tests
The API is the RESTful API for interacting with VinylDNS. The following technologies are used:
- Akka HTTP - Used primarily for REST and HTTP calls.
- FS2 - Used for backend change processing off of message queues. FS2 has back-pressure built in, and gives us tools like throttling and concurrency.
- Cats Effect - A replacement of
Future
with theIO
monad - Cats - Used for functional programming.
- PureConfig - For loading configuration values.
The API has the following dependencies:
- MySQL - the SQL database that houses the data
- SQS - for managing concurrent updates and enabling high-availability
- Bind9 - for testing integration with a real DNS system
The API code can be found in modules/api
.
src/it
- integration testssrc/main
- the main source codesrc/test
- unit testssrc/universal
- items that are packaged in the Docker image for the VinylDNS API
The package structure for the source code follows:
vinyldns.api.domain
- contains the core front-end logic. This includes things like the application services, repository interfaces, domain model, validations, and business rules.vinyldns.api.engine
- the back-end processing engine. This is where we process commands including record changes, zone changes, and zone syncs.vinyldns.api.protobuf
- marshalling and unmarshalling to and from protobuf to types in our systemvinyldns.api.repository
- repository implementations live herevinyldns.api.route
- HTTP endpoints
The project is built using:
The portal is mostly a shim around the API. Most actions in the user interface are translated into API calls.
The features that the Portal provides that are not in the API include:
- Authentication against LDAP
- Creation of users - when a user logs in for the first time, VinylDNS automatically creates a user and new credentials for them in the database with their LDAP information.
The portal code can be found in modules/portal
.
app
- source code for portal back-endmodels
- data structures that are used by the portalviews
- HTML templates for each web pagecontrollers
- logic for updating data
conf
- configurations and endpoint routespublic
- source code for portal front-endcss
- stylesheetsimages
- images, including icons, used in the portaljs
- scriptsmocks
- mock JSON used in Grunt teststemplates
- modal templates
test
- unit tests for portal back-end
Code used to build the microsite content for the API, operator and portal guides at https://www.vinyldns.io/. Some
settings for the microsite are also configured in build.sbt
of the project root.
src/main/resources
- Microsite resources and configurationssrc/main/mdoc
- Content for microsite web pages
VinylDNS can be started in the background by running the quickstart instructions located in the README. However, VinylDNS can also be run in the foreground.
If you are using a Mac running macOS with one of the new Apple M1 chips, you will need to update some dependencies to newer versions before attempting to run VinylDNS locally. To verify whether your computer has one of these chips, go to About This Mac in the Apple menu in the top-left corner of your screen. If next to Chip you see Apple M1, or any later chip such as the Apple M1 Pro or Apple M1 Max, then you will need to apply these changes to the code.
Update protoc from version 2.6.1:
PB.targets in Compile := Seq(PB.gens.java("2.6.1") -> (sourceManaged in Compile).value),
PB.protocVersion := "-v261"
to version 3.21.7:
PB.targets in Compile := Seq(PB.gens.java("3.21.7") -> (sourceManaged in Compile).value),
PB.protocVersion := "-v3.21.7"
Update sbt.version=1.4.0
to sbt.version=1.7.2
Update protobuf from version 2.6.1:
"com.google.protobuf" % "protobuf-java" % "2.6.1",
to version 3.21.7:
"com.google.protobuf" % "protobuf-java" % "3.21.7",
Update the sbt-protoc plugin from version 0.99.18:
addSbtPlugin("com.thesamet" % "sbt-protoc" % "0.99.18")
to version 1.0.6:
addSbtPlugin("com.thesamet" % "sbt-protoc" % "1.0.6")
Before starting the API service, you can start the dependencies for local development:
quickstart/quickstart-vinyldns.sh --deps-only
This will start a container running in the background with necessary prerequisites.
Once the prerequisites are running, you can start up sbt by running sbt
from the root directory.
project api
to change the sbt project to the APIreStart
to start up the API server- To enable interactive debugging, you can run
set Revolver.enableDebugging(port = 5020, suspend = true)
before runningreStart
- To enable interactive debugging, you can run
- Wait until you see the message
VINYLDNS SERVER STARTED SUCCESSFULLY
before working with the server - To stop the VinylDNS server, run
reStop
from the api project - To stop the dependent Docker containers:
utils/clean-vinyldns-containers.sh
See the API Configuration Guide for information regarding API configuration.
To run the portal locally, you first have to start up the VinylDNS API Server. This can be done by following the instructions for Staring the API Server or by using the QuickStart:
quickstart/quickstart-vinyldns.sh --api
Once that is done, in the same sbt
session or a different one, go to project portal
and then
execute ;preparePortal; run
.
See the Portal Configuration Guide for information regarding portal configuration.
- First, start up your Scala build tool:
build/sbt.sh
(orsbt
if running outside of Docker). - (Optionally) Go to the project you want to work on, for example
project api
for the API;project portal
for the portal. - Run all unit tests by just running
test
. - Run a single unit test suite by running
testOnly *MySpec
. - Run a single unit by filtering the test name using the
-z
argumenttestOnly *MySpec -- -z "some text from test"
. - If you are working on a unit test and production code at the same time, use
~
(e.g.,~testOnly *MySpec
) to automatically background compile for you!
Integration tests are used to test integration with dependent services. We use Docker to spin up those backend services for integration test development.
- Type
quickstart/quickstart-vinyldns.sh --reset --deps-only
to start up dependent background services - Run sbt (
build/sbt.sh
orsbt
locally) - Go to the target module in sbt, example:
project api
- Run all integration tests by typing
it:test
. - Run an individual integration test by typing
it:testOnly *MyIntegrationSpec
- You can background compile as well if working on a single spec by using
~it:testOnly *MyIntegrationSpec
- You must restart the dependent services (
quickstart/quickstart-vinyldns.sh --reset --deps-only
) before you rerun the tests. - For the mysql module, you may need to wait up to 30 seconds after starting the services before running the tests for setup to complete.
You can run all unit and integration tests for the api and portal by running build/verify.sh
When adding new features, you will often need to write new functional tests that black box / regression test the API.
- The API functional tests are written in Python and live under
modules/api/src/test/functional
. - The Portal functional tests are written in JavaScript and live under
modules/portal/test
.
To run functional tests you can simply execute the following commands:
build/func-test-api.sh
build/func-test-portal.sh
These command will run the API functional tests and portal functional tests respectively.
To run functional tests you can simply execute build/func-test-api.sh
, but if you'd like finer-grained control, you
can work with the Makefile
in test/api/functional
:
# Build and then run the function test container
make -C test/api/functional build run
During iterative test development, you can use make run-local
which will bind-mount the current functional tests in
the container, allowing for easier test development.
Additionally, you can pass --interactive
to make run
or make run-local
to drop to a shell inside the container.
From there you can run tests with the /functional_test/run.sh
command. This allows for finer-grained control over the
test execution process as well as easier inspection of logs.
You can run a specific test by name by running make build
and make run -- -k <name of test function>
. Any arguments after
make run --
will be passed to the test runner test/api/functional/run.sh
.
Finally, you can execute make run-deps-bg
to all of the dependencies for the functional test, but not run the tests.
This is useful if, for example, you want to use an interactive debugger on your local machine, but host all of the
VinylDNS API dependencies in Docker.
We use pytest for python tests. It is helpful that you browse the documentation so that you are familiar with pytest and how our functional tests operate.
We also use PyHamcrest for matchers in order to write easy to read tests. Please browse that documentation as well so that you are familiar with the different matchers for PyHamcrest. There aren't a lot, so it should be quick.
In the modules/api/src/test/functional
directory are a few important files for you to be familiar with:
vinyl_python.py
- this provides the interface to the VinylDNS API. It handles signing the request for you, as well as building and executing the requests, and giving you back valid responses. For all new API endpoints, there should be a corresponding function in the vinyl_clientutils.py
- provides general use functions that can be used anywhere in your tests. Feel free to contribute new functions here when you see repetition in the code
In the modules/api/src/test/functional/tests
directory, we have directories / modules for different areas of the
application.
batch
- for managing batch updatesinternal
- for internal endpoints (not intended for public consumption)membership
- for managing groups and usersrecordsets
- for managing record setszones
- for managing zones
Our functional tests use pytest
contexts. There is a main test context that lives in shared_zone_test_context.py
that creates and tears down a shared test context used by many functional tests. The beauty of pytest is that it will
ensure that the test context is stood up exactly once, then all individual tests that use the context are called using
that same context.
The shared test context sets up several things that can be reused:
- An
ok
user and group - A
dummy
user and group - a separate user and group helpful for testing access controls and authorization - An
ok.
zone accessible only by theok
user andok
group - A
dummy.
zone accessible only by thedummy
user anddummy
group - An IPv6 reverse zone
- A normal IPv4 reverse zone
- A classless IPv4 reverse zone
- A parent zone that has child zones - used for testing NS record management and zone delegations
Each of the test zones are configured in a partition
. By default, there are four partitions. These partitions are
effectively copies of the zones so that parallel tests can run without interfering with one another.
For instance, there are four zones for the ok
zone: ok1
, ok2
, ok3
, and ok4
. The functional tests will handle
distributing which zone is being used by which of the parallel test runners.
As such, you should never hardcode the name of the zone. Always get the zone from the shared_zone_test_context
.
For instance, to get the ok
zone, you would write:
zone = shared_zone_test_context.ok_zone
zone_name = shared_zone_test_context.ok_zone["name"]
zone_id = shared_zone_test_context.ok_zone["id"]
- Try to use the
shared_zone_test_context
whenever possible! This reduces the time it takes to run functional tests (which is in minutes). - Be mindful of changes to users, groups, and zones in the shared test context, as doing so could impact downstream tests
- If you do modify any entities in the shared zone context, roll those back when your function completes!
When functional tests are run, we spin up several Docker containers. One of the Docker containers is a Bind9 DNS server.
If you need to add or modify the test DNS zone files, you can find them in
quickstart/bind9/zones