Skip to content
This repository has been archived by the owner on May 7, 2021. It is now read-only.

Commit

Permalink
Add kola runner & platforms READMEs
Browse files Browse the repository at this point in the history 
Adds some markdown files which detail how the kola runner works
internally as well as what constitutes a platform in mantle (and some
common things that need to be implemented).
  • Loading branch information
@arithx
arithx committed Jun 26, 2018
1 parent 222e5f9 commit ff26618
Show file tree
Hide file tree
Showing 2 changed files with 275 additions and 0 deletions.
53 changes: 53 additions & 0 deletions platforms.md
View file
@@ -0,0 +1,53 @@
# Mantle platforms

Platforms are an API interface to different environments to run clusters,
create images, collect logging information, etc.

## Authentication

Authentication differs based on the platform. Some platforms like `aws` utilize the
configuration files from their command-line tooling while others define their
own custom configuration format and default locations (like [DigitalOcean](https://github.com/coreos/mantle/tree/master/auth/do.go)).
Generally if any extensions / custom configurations are needed a new file is
created inside of the `auth` package which will define the default location,
the structure of the configuration, and a function to parse the configuration
file (usually named Read<platform>Config and emits a
`map[string]<platform>Config` object).

## API

Platform APIs wrap each cloud provider's golang SDK and live inside of
`platform/api/<platform>/`. There is no direct requirement for what
functionality is present in the API.

## Cluster & Machine

Clusters must implement the `Cluster` [interface](https://github.com/coreos/mantle/tree/master/platform/platform.go#L75-L97).
Machines must implement the `Machine` [interface](https://github.com/coreos/mantle/tree/master/platform/platform.go#L40-L73).

## Adding a new platform to the kola runner

To add a new platform to the `kola` runner the following things must be added:
1. A platform specific struct inside of `cmd/kola/kola.go` which contains the
fields that should be logged to give more information about a test run.
Generally this will contain things like `version` or `ami` and `region`. [This](https://github.com/coreos/mantle/tree/master/cmd/kola/kola.go#L138-L142)
is an example of the struct and [this](https://github.com/coreos/mantle/tree/master/cmd/kola/kola.go#L179-L183) shows the data
being added to the output (which can be found in
`_kola_temp/<platform>-latest/properties.json`).
2. The platform specific options inside of `cmd/kola/options.go`
([for example DigitalOcean](https://github.com/coreos/mantle/tree/master/cmd/kola/kola.go#L179-L183)). The flags will
generally contain an override for the configuration file location, region,
type/size, and the image.
3. The platform needs to be added to the `kolaPlatforms` list inside of
`cmd/kola/options.go` [here](https://github.com/coreos/mantle/tree/master/cmd/kola/options.go#L32)
4. The platform options & new cluster inside of `kola/harness.go`. The platform
options variables are defined [here](https://github.com/coreos/mantle/tree/master/kola/harness.go#L54-L60) and the
`NewCluster` method is defined [here](https://github.com/coreos/mantle/tree/master/kola/harness.go#L143-L161).

## Other things to consider adding

It is generally preferred that new platforms add garbage collection methods via
the `ore <platform> gc` command.

For platforms which support adding custom images `ore` commands to upload &
create images from the image file.
222 changes: 222 additions & 0 deletions runner-readme.md
View file
@@ -0,0 +1,222 @@
# Kola Runner

## Table of Contents

* [General Function Overview](#general-function-overview)
* [Visual Function Flow](#visual-function-flow)
* [Detailed Workflow](#detailed-workflow)
* [Common Closures](#common-closures)
* [Logging](#logging)

The `kola` runner is very similar to the standard `go test` runner with some
modifications to allow tighter control over the run loop and extensibility
for logging (without parsing) & output control.

## General Function Overview

### kola/harness

#### kola/harness: RunTests

Responsible for filtering down the test list based on the given criteria,
creating the `harness/suite: Suite` object, and outputting the final result.

#### kola/harness: runTest

Creates the test cluster, runs the individual test function, and cleans up the
test cluster.

### harness/suite

#### harness/suite: Run

Creates the output directory, the `test.tap` file and any profile related
files then calls `harness/suite: runTests`.

#### harness/suite: runTests

Sets up the `harness/harness: H` object and calls `harness/harness: tRunner`
with a closure to call `harness/harness: Run` for each test.

### harness/harness

#### harness/harness: tRunner

Responsible for the timing, reporting, and execution of a closure.

#### harness/harness: Run

Handles the setup of child `harness/harness: H` objects, loggers, and the
running of closures as subtests.

## Visual Function Flow

The first 4 steps handle filtering down the test list, creating the
clusters, and building the suite. The next 4 set up the reporting
structure of the test group and run the child tests. The following 2 get
ready to run each individual test. And the final step runs the actual
test function registered in the test.

```
cmd/kola/kola
|
v
kola/harness: RunTests
|
v
harness/suite: Run
|
v
harness/suite: runTests
|
v
harness/harness: tRunner
|
v
harness/harness: Run
|
v
harness/harness: tRunner
|
v
kola/harness: runTest
|
v
harness/harness: Run
|
v
harness/harness: tRunner
|
v
kola/register/register: Run
```

## Detailed Workflow

1. The `kola` cmd calls into `kola/harness: RunTests`
2. `kola/harness: RunTests` calls `kola/harness: filterTests` to build a test
list filtered down by the given pattern & platform from all tests in
`kola/register/register: Tests` object.
3. `kola/harness: RunTests` checks if any of the tests do not exactly match
the given pattern and have either a `MinVersion` or `MaxVersion` tag, if so
then it will call `kola/harness: getClusterSemver` to spin up a machine on
the given platform to extract the OS semver. The tests will then be filtered
down again with the semver. Note that if a pattern matches an individual test
the `kola/harness: getClusterSemver` check will be skipped and the test will
be run without regard to the `MinVersion` or `MaxVersion` tags.
4. `kola/harness: RunTests` will then construct a `harness/suite: Options`
object and construct a `harness/test: Test` object containing the name of each
test and a closure (#1) calling `kola/harness: runTest`.
5. `kola/harness: RunTests` constructs a `harness/suite: Suite` object via
`harness/suite: NewSuite` using the `harness/suite: Options` and
`harness/test: Test` objects and proceeds to call the `harness/suite: Run`
function on the `harness/suite: Suite` object.
6. `harness/suite: Run` starts by creating or cleaning up the output directory
by calling the `harness/harness: CleanOutputDir` function. It then creates the
`test.tap` file inside of the output directory and prints a string to the file
containing `1..%d` where %d is the amount of tests being run.
7. `harness/suite: Run` then checks if the following options were selected and
if so creates the corresponding files in the output path:

| Option | Filename |
| -------------- | ---------- |
| MemProfile | mem.prof |
| BlockProfile | block.prof |
| CpuProfile | cpu.prof |
| ExecutionTrace | exec.trace |

8. `harness/suite: Run` then calls `harness/suite: runTests` passing
`os.Stdout` and the `tap io.Writer` object.
9. `harness/suite: runTests` starts by setting the `running` variable on the
`harness/suite: Suite` object, which is the count of running tests, to 1 and
creating the `harness/harness: H` object.
10. `harness/suite: runTests` then calls `harness/harness: tRunner` passing
the `harness/harness: H` object and a closure (#2) which loops each test in the
`harness/suite: Suite` object calling `harness/harness: Run` on each, passing
the name of the test, the `harness/test: Test` object, and a boolean pointer
set to false, followed by a goroutine call to receive from the signal channel
on the `harness/harness: H` object.
11. `harness/harness: tRunner` starts by creating a `context.WithCancel`
object, the result of `harness/harness: parentContext` is passed in which will
either be the context object of the `harness/harness: H` objects parent or
`context.Background()` if the object doesn't have a parent.
12. `harness/harness: tRunner` then defers a closure which will detect the
status of the test run, calculate the ending time, run any subtests, call
`harness/harness: report` (which will flush the test result to the parent
via the `harness/harness: flushToParent` function), and send `true` on the
`harness/harness: H` `signal` channel.
13. `harness/harness: tRunner` will then calculate the start time and call the
closure it received as an argument with the `harness/harness: H` variable as a
parameter, this will be the closure that was created in
`harness/suite: runTests` which will call `harness/harness: Run` for each test.
14. `harness/harness: Run` runs each function as a subtest of the
`harness/harness: H` object it is passed with the name passed. It starts by
marking the `hasSub` variable on the `harness/harness: H` object to true and
checking that the test name it received is a valid test via the
`harness/match: fullName` function.
15. `harness/harness: Run` will then create a new `harness/harness: H` object
which has the object it received as the parent and a `log` object.
16. `harness/harness: Run` then does a goroutine call on
`harness/harness: tRunner` passing in the new `harness/harness: H` object,
the closure function it was passed, which is the call to
`kola/harness: runTest`, and the boolean pointer it was passed.
17. `harness/harness: tRunner` will then run through and call
`kola/harness: runTest`.
18. `kola/harness: runTest` is the harness responsible for running a single
test grouping (test groupings tests. It will create the cluster that
will be used by the tests, validate that the machines spun up properly,
and then call `kola/register/register: Run` on the
`kola/register/register: Test` object, which is a function pointer which
accepts a `kola/cluster/cluster: TestCluster` object and is defined
inside of the individual test files.

## Common Closures

1. `kola/harness: RunTests`

Accepts a `harness/harness: H` object and calls `kola/harness: runTest`

```
func(h *harness.H) {
runTest(h, []*register.Test{test}, pltfrm, false)
}
```

2. `harness/suite: runTests`

Accepts a `harness/harness: H` object. Loops each test in the
`harness/suite: Suite` object calling `harness/harness: Run`. This is being
pass as an argument to `harness/harness: tRunner`. `harness/harness:tRunner`
will time the the outer block and call `harness/harness: Run` which will run
the `test` function as a subtest.

For instance, `harness/harness: tRunner` will be called with the
`harness/harness: H` object representing the entire test run. It will then
execute this closure which will loop through every test and call
`harness/harness: Run` which will run each as a subtest for reporting purposes
inside of goroutines.

```
func(t *H) {
for name, test := range s.tests {
t.Run(name, test, util.BoolToPtr(false))
}
// Run catching the signal rather than the tRunner as a separate
// goroutine to avoid adding a goroutine during the sequential
// phase as this pollutes the stacktrace output when aborting.
go func() { <-t.signal }()
}
```

## Logging

The `kola` runner supports custom reporting via the
`harness/reporters: Reporter` interface. By default plain text will be output
into `stdout` and a JSON file will be produced inside of the `_kola_temp` run
log (e.x.: `_kola_temp/<platform>-latest/reports/report.json`). New output
formats can be added by creating a new struct which implements the
`harness/reporters: Reporter` interface and instantiating an object of said
reporter inside of the `harness: Options` object created in
`kola/harness: RunTests`.

[For example](https://github.com/coreos/mantle/blob/52407c3ae8cd0837511c665af2c7870393e024bb/kola/harness.go#L295-L297) this is how the JSON reporter is added.

0 comments on commit ff26618

Please sign in to comment.