Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Compliance Tests: Scalar Instructions #27

Closed
ben-marshall opened this issue Aug 13, 2020 · 1 comment
Closed

Compliance Tests: Scalar Instructions #27

ben-marshall opened this issue Aug 13, 2020 · 1 comment
Labels
architectural-tests Anything relating to the riscv-arch-test suite for the crypto extension. help wanted Extra attention is needed
Projects
RISC-V Scalar Cryptography Extensions

Comments

@ben-marshall
Copy link
Member

ben-marshall commented Aug 13, 2020 •
edited
Loading

Create riscv-compliance tests for the cryptography extension or explain why we can't in some cases.

  • A useful document on what the architectural compliance tests are and what they try to achieve.
  • Compliance Test Suite Documentation
  • Instructions borrowed from Bitmanip will have their tests written by the Bitmanip TG. We will need to tell the compliance group which instructions are included in the crypto test suites.
@ben-marshall ben-marshall created this issue from a note in RISC-V Scalar Cryptography Extensions (Engineering Deliverables) Aug 13, 2020
@ben-marshall ben-marshall added architectural-tests Anything relating to the riscv-arch-test suite for the crypto extension. help wanted Extra attention is needed labels Aug 13, 2020
@ben-marshall ben-marshall changed the title Compliance Tests Compliance Tests: Scalar Instructions Aug 13, 2020
@ben-marshall ben-marshall mentioned this issue Aug 19, 2020
Closed
21 tasks
ben-marshall added a commit that referenced this issue Oct 8, 2020
@ben-marshall
Working on per-instruction KAT generation. #27
cd482d6 
- Simple C program which feeds random inputs into each instruction and
  records the result.

- Can be used later to check different simulators work correctly, and
  to generate compliance tests.

 On branch dev/next-release
 Your branch is ahead of 'origin/dev/next-release' by 1 commit.
   (use "git push" to publish your local commits)

 Changes to be committed:
	modified:   Makefile
	modified:   benchmarks/Makefile
	modified:   benchmarks/common.mk
	new file:   benchmarks/kat-gen/Makefile.in
	new file:   benchmarks/kat-gen/README.md
	new file:   benchmarks/kat-gen/kat-gen.c
	modified:   benchmarks/share/riscv-crypto-intrinsics.h

 Changes not staged for commit:
	modified:   extern/riscv-gnu-toolchain (modified content)
	modified:   extern/riscv-isa-sim (modified content)
ben-marshall added a commit that referenced this issue Oct 15, 2020
@ben-marshall
compliance: Start work on #27
5cb6110 
- Documented a work flow for the compliance tests in tests/compliance/README.md

- A single (not very good) test for sha2sum0 is there as an example.

- Passes on Spike and SAIL (for what it's worth right now).

- Next steps are to generate the tests from the kat generator.

 On branch dev/next-release
 Your branch is up-to-date with 'origin/dev/next-release'.

 Changes to be committed:
	modified:   .gitmodules
	modified:   bin/conf.sh
	new file:   extern/riscv-compliance
	new file:   tests/compliance/Makefile
	new file:   tests/compliance/README.md
	new file:   tests/compliance/riscv-compliance.patch

 Changes not staged for commit:
	modified:   extern/riscv-compliance (modified content)
	modified:   extern/riscv-gnu-toolchain (modified content)
	modified:   extern/riscv-isa-sim (modified content)
	modified:   extern/sail-riscv (modified content, untracked content)
ben-marshall added a commit that referenced this issue Oct 15, 2020
@ben-marshall
compliance: #27 tentative tests for scalar instructions.
0ce1194 
- Includes fix to RV32 SAIL model for sha512 instructions.

- Shortcomings:

  - No coverage numbers (yet). Need help from Imperas?

  - Same source/dest registers used every time. This can be fixed
    easily though.

- Generated tests for all RV32 and RV64 scalar ISA instructions.

- KAT values use Spike as the golden reference.

- SAIL CSIM and OCAML simulators agree with spike.

- Update the riscv-compliance patch with reference output values.

 On branch dev/next-release
 Your branch is ahead of 'origin/dev/next-release' by 1 commit.
   (use "git push" to publish your local commits)

 Changes to be committed:
	modified:   sail/riscv_insts_crypto_rv32.sail
	modified:   tests/compliance/Makefile
	modified:   tests/compliance/README.md
	new file:   tests/compliance/generate_tests.py
	modified:   tests/compliance/riscv-compliance.patch
	new file:   tests/compliance/test_template.S

 Changes not staged for commit:
	modified:   extern/riscv-compliance (modified content)
	modified:   extern/riscv-gnu-toolchain (modified content, untracked content)
	modified:   extern/riscv-isa-sim (modified content)
	modified:   extern/sail-riscv (modified content, untracked content)
ben-marshall added a commit that referenced this issue Oct 15, 2020
@ben-marshall
compliance: #27 - method and reproduction notes.
3359186 
 On branch dev/next-release
 Your branch is ahead of 'origin/dev/next-release' by 2 commits.
   (use "git push" to publish your local commits)

 Changes to be committed:
	modified:   tests/compliance/README.md

 Changes not staged for commit:
	modified:   extern/riscv-compliance (modified content)
	modified:   extern/riscv-gnu-toolchain (modified content, untracked content)
	modified:   extern/riscv-isa-sim (modified content)
	modified:   extern/sail-riscv (modified content, untracked content)
ben-marshall added a commit that referenced this issue Oct 15, 2020
@ben-marshall
compliance: #27 - more readme notes.
a4555d2 
 On branch dev/next-release
 Your branch is ahead of 'origin/dev/next-release' by 3 commits.
   (use "git push" to publish your local commits)

 Changes to be committed:
	modified:   tests/compliance/README.md

 Changes not staged for commit:
	modified:   extern/riscv-compliance (modified content)
	modified:   extern/riscv-gnu-toolchain (modified content, untracked content)
	modified:   extern/riscv-isa-sim (modified content)
	modified:   extern/sail-riscv (modified content, untracked content)
@ben-marshall
Copy link
Member Author

There are now very basic compliance tests in the patch for riscv-compliance on the riscv-crypto master branch. They suffer from the following shortcomings:

  • Always using the same src/dest register addresses. This can be easily randomised.
  • No notion of coverage. It's not clear how this is structured or recorded.
    • All SBox values for AES/SM4 instructions must be tested at minimum.
  • Because the tests are all automatically generated from random values, there are no interesting cases.
  • There are no instruction-sequence tests for thinks like forwarding of values etc.

ben-marshall added a commit that referenced this issue Oct 16, 2020
@ben-marshall
compliance: #27 - WiP compliance test plan.
8df82fc 
 On branch dev/next-release
 Your branch is up-to-date with 'origin/dev/next-release'.

 Changes to be committed:
	modified:   README.md
	modified:   tests/compliance/README.md
	new file:   tests/compliance/test-plan-scalar.adoc

 Changes not staged for commit:
	modified:   extern/riscv-compliance (modified content)
	modified:   extern/riscv-gnu-toolchain (modified content)
	modified:   extern/riscv-isa-sim (modified content)
	modified:   extern/sail-riscv (modified content, untracked content)
ben-marshall added a commit that referenced this issue Oct 16, 2020
@ben-marshall
compliance: #27 constant time requirements
7bcb0eb 
 On branch dev/next-release
 Your branch is up-to-date with 'origin/dev/next-release'.

 Changes to be committed:
	modified:   tests/compliance/test-plan-scalar.adoc

 Changes not staged for commit:
	modified:   extern/riscv-compliance (modified content)
	modified:   extern/riscv-gnu-toolchain (modified content)
	modified:   extern/riscv-isa-sim (modified content)
	modified:   extern/sail-riscv (modified content, untracked content)
ben-marshall added a commit that referenced this issue Oct 21, 2020
@ben-marshall
compliance: scalar test plan - more info on mul/clmul
658012d 
- See #27

 On branch dev/next-release
 Your branch is ahead of 'origin/dev/next-release' by 1 commit.
   (use "git push" to publish your local commits)

 Changes to be committed:
	modified:   test-plan-scalar.adoc

 Changes not staged for commit:
	modified:   ../../extern/riscv-compliance (modified content)
	modified:   ../../extern/riscv-gnu-toolchain (modified content)
	modified:   ../../extern/riscv-isa-sim (modified content)
	modified:   ../../extern/sail-riscv (modified content, untracked content)
RISC-V Scalar Cryptography Extensions automation moved this from Engineering Deliverables to Finished Tasks Aug 6, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
architectural-tests Anything relating to the riscv-arch-test suite for the crypto extension. help wanted Extra attention is needed
Projects
RISC-V Scalar Cryptography Extensions
  
Finished Tasks
Milestone
No milestone
Development

No branches or pull requests
1 participant
@ben-marshall