Skip to content

Commit

Permalink
compliance: #27 - method and reproduction notes.
Browse files Browse the repository at this point in the history
 On branch dev/next-release
 Your branch is ahead of 'origin/dev/next-release' by 2 commits.
   (use "git push" to publish your local commits)

 Changes to be committed:
	modified:   tests/compliance/README.md

 Changes not staged for commit:
	modified:   extern/riscv-compliance (modified content)
	modified:   extern/riscv-gnu-toolchain (modified content, untracked content)
	modified:   extern/riscv-isa-sim (modified content)
	modified:   extern/sail-riscv (modified content, untracked content)
  • Loading branch information
ben-marshall committed Oct 15, 2020
1 parent 0ce1194 commit 3359186
Showing 1 changed file with 56 additions and 0 deletions.
56 changes: 56 additions & 0 deletions tests/compliance/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,31 @@
- [Documentation](https://github.com/riscv/riscv-compliance/tree/master/doc)


## Methodology:

This is the general methodology for generating the compliance tests for
the Scalar Crypto ISE.

1. Implement the ISE in Spike
2. Implement the ISE in SAIL
3. Create a simple host-agonistic test harness to generate input/output test
vectors. This is found in `tests/kat-gen`.
4. Run the harness on Spike and SAIL, checking for diffs and fixing as
appropriate. Again, see `tests/kat-gen/README.md` for an explanation
of this.
5. Other TG members run the same harness on their implementations, and diff
against the same output from SAIL/Spike. If everything matches, we can be
confident that we have all implemented the same thing.
6. If we are all confident our implementations are correct, the compliance
tests are generated from the test harness.
7. The generated compliance tests can then be re-run on all of our various
implementations as a final check.

Once the first set of compliance test vectors are generated, we can all
use the existing riscv-compliance framework rather than the hacky
kat-generator tool. The kat-generator is just there to bootstrap the
process.

## Getting Started

- Make sure that you are in the root of the `riscv-crypto` repository, and
Expand Down Expand Up @@ -95,3 +120,34 @@
Which just wrap up the above long-form commands and run all of the
available compliance tests.


## Re-generating the reference outputs.

- Generate example outputs from your *trusted simulator*. We will use
SAIL in this example.

```make
make -C tests/compliance compliance-generate-rv32ik
make -C tests/compliance compliance-run-sail-csim-rv32
```

- Ignore any reported test failures, and copy the generated test
signatures into the *reference* signatures directory:

```
make -C tests/compliance compliance-update-signatures-rv32ik
```

- Now, run a *different* simulator or target against the newly generated
reference signatures to check they agree.
Here, we run the SAIL OCaml simulator and Spike.

```make
make -C tests/compliance compliance-run-sail-ocaml-rv32
make -C tests/compliance compliance-run-spike-rv32
```

If they don't agree, one (or both) of the simualtors is defintley wrong.

- The same process can be run, substituting `rv32` for `rv64` in all cases.

0 comments on commit 3359186

Please sign in to comment.