Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

lint: Check all markdown files in the repo #2836

Merged
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
62 changes: 34 additions & 28 deletions README.md
@@ -1,5 +1,5 @@
SCION
=====
# SCION

[![Documentation](https://godoc.org/github.com/scionproto/scion?status.svg)](http://godoc.org/github.com/scionproto/scion)
[![Build Status](https://badge.buildkite.com/cbb8e648c58d567991b445317d68955db5235b627dcfc97ab9.svg?branch=master)](https://buildkite.com/scionproto/scionproto)
[![Go Report Card](https://goreportcard.com/badge/github.com/scionproto/scion)](https://goreportcard.com/report/github.com/scionproto/scion)
Expand Down Expand Up @@ -38,7 +38,8 @@ Necessary steps in order to run SCION:
1. Make sure that you are using a clean and recently updated **Ubuntu 16.04**.

1. Install [Bazel](https://bazel.build) version 0.26.1:
```

```bash
wget https://github.com/bazelbuild/bazel/releases/download/0.26.1/bazel-0.26.1-installer-linux-x86_64.sh
bash ./bazel-0.26.1-installer-linux-x86_64.sh --user
rm ./bazel-0.26.1-installer-linux-x86_64.sh
Expand All @@ -48,30 +49,33 @@ Necessary steps in order to run SCION:
[Go workspace](https://golang.org/doc/code.html#GOPATH) setup, and that
`~/.local/bin`, and `$GOPATH/bin` can be found in your `$PATH` variable. For example:

```
echo 'export GOPATH="$HOME/go"' >> ~/.profile
echo 'export PATH="$HOME/.local/bin:$GOPATH/bin:$PATH"' >> ~/.profile
source ~/.profile
mkdir -p "$GOPATH"
```
```bash
echo 'export GOPATH="$HOME/go"' >> ~/.profile
echo 'export PATH="$HOME/.local/bin:$GOPATH/bin:$PATH"' >> ~/.profile
source ~/.profile
mkdir -p "$GOPATH"
```

1. Check out scion into the appropriate directory inside your go workspace (or
put a symlink into the go workspace to point to your existing scion
checkout):
```

```bash
mkdir -p "$GOPATH/src/github.com/scionproto"
cd "$GOPATH/src/github.com/scionproto"
git clone --recursive git@github.com:scionproto/scion
cd scion
```

If you don't have a github account, or haven't setup ssh access to it, this
command will make git use https instead:
`git config --global url.https://github.com/.insteadOf git@github.com:`

1. Install required packages with dependencies:
```
./env/deps
```

```bash
./env/deps
```

1. Install `docker` and `docker-compose`. Please follow the instructions for
[docker-ce](https://docs.docker.com/install/linux/docker-ce/ubuntu/) and
Expand All @@ -82,21 +86,23 @@ Necessary steps in order to run SCION:
1. Create the topology and configuration files (according to
`topology/Default.topo`):

`./scion.sh topology`

The resulting directory structure will be created:

./gen/ISD{X}/AS{Y}/
{elem}{X}-{Y}-{Z}/
as.yml
path_policy.yml
supervisord.conf
topology.yml
certs/
ISD{X}-AS{Y}-V0.crt
ISD{X}-V0.trc
keys/
as-sig.key
`./scion.sh topology`

The resulting directory structure will be created:

```bash
./gen/ISD{X}/AS{Y}/
{elem}{X}-{Y}-{Z}/
as.yml
path_policy.yml
supervisord.conf
topology.yml
certs/
ISD{X}-AS{Y}-V0.crt
ISD{X}-V0.trc
keys/
as-sig.key
```

The default topology looks like [this](doc/fig/default_topo.png).

Expand Down
6 changes: 6 additions & 0 deletions TESTING.md
Expand Up @@ -10,15 +10,18 @@ All commands are relative to the top-level directory
1. Run all language specific tests: `./scion.sh test py`, `./scion.sh test go`

### Python specific usage

1. Run all tests in a specific file: `./scion.sh test py test/lib/packet/opaque_field_test.py`
1. Run a specific test: `./scion.sh test py test.lib.packet.opaque_field_test:TestHopOpaqueFieldCalcMac`
1. Run a specific test function: `./scion.sh test py test/lib/packet/opaque_field_test.py:TestHopOpaqueFieldCalcMac.test_prev`
1. See which tests are being run: `./scion.sh test py -v`

## Code coverage

To get a html report of the testing coverage, run `./scion.sh coverage`

## Integration Tests

Several integration tests can be found under `python/integration`. Before running any of
those tests, you have to start the infrastructure (`./scion.sh start`).

Expand All @@ -27,16 +30,19 @@ the infrastructure, however, it doesn't compile it. If you haven't compiled the
yet or made local changes then first run `make`.

### Docker

It's possible to run the entire infrastructure in a docker container. For more information
see `docker/README.md`.

### Per-app Docker

It's possible to run every service instance in its own docker container. When creating the
topology, specify the `-d` flag.

If you want to use `docker-compose` commands, `./tools/dc` might be helpful.

### CI

The SCION project uses Buildkite as its continuous integration platform. To run an approximated
local version of the CI pipeline (does not include acceptance tests), first build a docker image
as described in `docker/README.md` and then run `tools/ci/local`.
31 changes: 18 additions & 13 deletions acceptance/README.md
@@ -1,11 +1,10 @@

Acceptance testing framework
============================
# Acceptance testing framework

To add an acceptance test, create a new `xxx_acceptance` folder in
`/acceptance`, with `xxx` replaced by the name of your test.

The folder must contain a `test` executable, which must support the following arguments:

* `name`, which returns the name of the acceptance test.
* `setup`, which runs the setup portion of the acceptance test. If the return
value of the application is non-zero, the test is aborted.
Expand All @@ -16,41 +15,47 @@ The folder must contain a `test` executable, which must support the following ar

For an example, see `acceptance/reconnecting_acceptance`.

Basic Commands
==============
## Basic Commands

To run all defined tests, use:
```

```bash
acceptance/run
```

To run only the tests matching a certain regular expression, use:
```

```bash
acceptance/run REGEX
```
where REGEX is replaced with a regular expression of your choice.

Manual Testing
==============
where `REGEX` is replaced with a regular expression of your choice.

## Manual Testing

To run fine-grained operations for a single test, use one of the following:
```

```bash
acceptance/ctl setup TESTNAME
acceptance/ctl run TESTNAME
acceptance/ctl teardown TESTNAME
```

This calls the functions in `acceptance/xxx_acceptance/test.sh` directly,
without any prior setup. This also means docker images are **not** rebuilt,
even if application code has changed.

To run the `ctl` commands above, the environment needs to be built first. To do that, run:
```

```bash
acceptance/ctl gsetup
```

This will also rebuild the docker images, taking new code into account.

To run the `setup`, `run` and `teardown` phases of a single test (without gsetup):
```

```bash
acceptance/ctl grun TESTNAME
```

Expand Down
13 changes: 7 additions & 6 deletions acceptance/common/README.md
@@ -1,8 +1,8 @@
## Acceptance Testing
# Acceptance Testing

This module provides a simple acceptance testing library.

### Structure
## Structure

An acceptance test is structured fairly simple. It exposes a way
to implement the necessary sub-commands for `acceptance/run`.
Expand All @@ -13,6 +13,7 @@ command with `--help` flag. Furthermore, a test should have one class per
sub-command that sub-classes `common.CmdBase` (see below).

`TestBase` registers two flags that also can be set using environment variables:

- `--artifacts/ACCEPTANCE_ARTIFACTS` defines the directory for artifacts
(required)
- `--disable-docker/DISABLE_DOCKER` disables the dockerized topology.
Expand All @@ -21,6 +22,7 @@ sub-command that sub-classes `common.CmdBase` (see below).
The sub-commands should sub-class `common.CmdBase`. `CmdBase` has defined some
common properties and methods that are useful for test writing and interacting
with the infrastructure, such as:

- `scion` can be used to start and stop the scion infrastructure,
or interact with individual service processes.
- `dc` can be used to interact with docker compose.
Expand All @@ -29,8 +31,7 @@ Sub-commands are registered with the `@Test.subcommand` decorator.
By default, the `name` and `teardown` sub-command are already implemented.
The `setup` and `run` command must be implemented by each test individually.


### Writing Your Own Test
## Writing Your Own Test

Write your own test by adding a directory to `acceptance` with the suffix
`_acceptance`. This suffix is required by the acceptance framework.
Expand All @@ -40,7 +41,7 @@ make it executable with `chmod +x`.

A minimal working test can be written as follows:

````python
```python
#!/usr/bin/env python3

import logging
Expand Down Expand Up @@ -101,4 +102,4 @@ class TestRun(CmdBase):
if __name__ == '__main__':
init_log()
Test.run()
````
```
48 changes: 30 additions & 18 deletions docker/README.md
Expand Up @@ -11,49 +11,61 @@ rebuild takes <= 15s.
The `scion_base` docker image contains all the dependencies of scion, and so it needs to be
regenerated any time the dependencies change. It is built via:

./docker.sh base
```bash
./docker.sh base
```

The `scion` docker image contains a snapshot of your working tree, and is layered on top of the
`scion_base` image (and hence should be rebuilt if `scion_base` changes). It is built via:

./docker.sh build
```bash
./docker.sh build
```

To run the `scion` docker image:

./docker.sh run
```bash
./docker.sh run
```

This will drop you into a bash shell, in a stripped down ubuntu environment. Your current working
tree has been copied into the image **at build time**. First build and create a topology
`make -s; ./scion.sh topology`, then you can use `./scion.sh run` to start the SCION processes.

If you would like to execute commands from the outside of the container, use

./docker.sh start
./docker.sh exec CMD
./docker.sh stop
```bash
./docker.sh start
./docker.sh exec CMD
./docker.sh stop
```

`./docker.sh` `run` and `start` will mount the `gen`, `logs` and `gen-certs` directories from a
temp directory. You can pass your own directory with `SCION_MOUNT`.

SCION_MOUNT=/tmp/scion_out ./docker.sh run "./scion.sh"
```bash
SCION_MOUNT=/tmp/scion_out ./docker.sh run "./scion.sh"
```

or

SCION_MOUNT=/tmp/scion_out ./docker.sh start
./docker.sh exec ./scion.sh topology -d
./docker.sh exec ./integration/integration_test.sh
./docker.sh stop`
```bash
SCION_MOUNT=/tmp/scion_out ./docker.sh start
./docker.sh exec ./scion.sh topology -d
./docker.sh exec ./integration/integration_test.sh
./docker.sh stop
```

Make sure you collect any relevant data from the container before running `./docker.sh stop` as
this stops and removes the container. The temp directory will still be available.

See `./docker.sh help` for further commands/usage.

## Notes:
## Notes

- As `docker.sh` copies your _working tree_, if you're trying to test a commit before sending for
review/etc, make sure your working directory is clean before building the image. Any new files
must be at least added to git, even if you haven't committed them, otherwise docker.sh will
skip them.
- When running `./scion.sh topology -d` in a container, make sure the zookeeper instance on your
host is stopped.
- As `docker.sh` copies your _working tree_, if you're trying to test a commit before sending for
review/etc, make sure your working directory is clean before building the image. Any new files
must be at least added to git, even if you haven't committed them, otherwise docker.sh will skip
them.
- When running `./scion.sh topology -d` in a container, make sure the zookeeper instance on your
host is stopped.
28 changes: 14 additions & 14 deletions docker/perapp/README.md
Expand Up @@ -4,29 +4,29 @@ Per-application Docker images for SCION

## Images

- Build base images {app, base, python, debug}: `make base`
- Build app images: `make apps`
- Build debug images: `make debug`
- Build base images {app, base, python, debug}: `make base`
- Build app images: `make apps`
- Build debug images: `make debug`

Build all images: `make all`, specific image: `make border` or specific debug image:
`make border_debug`

### Base Images

- app: Contains compiled app binaries
- base: Contains `libc`, `libcap2` and `su-exec`
- debug: `strace` and toybox
- python: base image plus python and pip packages and python code
- app: Contains compiled app binaries
- base: Contains `libc`, `libcap2` and `su-exec`
- debug: `strace` and toybox
- python: base image plus python and pip packages and python code

### App Images

- dispatcher: Runs `/app/dispatcher`
- border: Runs `/app/border`
- sig: Runs `/app/sig`
- beacon: Runs `/app/bin/beacon_server`
- path: Runs `/app/bin/path_server`
- certificate: Runs `/app/bin/cert_server`
- sciond: Runs `/app/bin/sciond`
- dispatcher: Runs `/app/dispatcher`
- border: Runs `/app/border`
- sig: Runs `/app/sig`
- beacon: Runs `/app/bin/beacon_server`
- path: Runs `/app/bin/path_server`
- certificate: Runs `/app/bin/cert_server`
- sciond: Runs `/app/bin/sciond`

Dispatcher, sig and border are based on `base`, the other images on `python`.

Expand Down