Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update Rust APIs #292

Merged
merged 78 commits into from
Dec 12, 2023
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
78 commits
Select commit Hold shift + click to select a range
9114ecb
fix memory error in single_stage_multi_reduction_kernel (#235)
ImmanuelSegol Oct 3, 2023
33d1583
Added separate device context struct, returned lde
DmytroTym Oct 3, 2023
1c14466
Merge branch 'main' into cuda_refactoring
DmytroTym Oct 3, 2023
3de0e79
wip - msm and eq
Oct 3, 2023
df1fc7e
added lde to cmake
DmytroTym Oct 4, 2023
3e9c0e7
wip draft with msm correctness test for bls12_381 + Merge commit 'df1…
Oct 4, 2023
d8e26e8
Montgomery param added in lde.cu mul function
DmytroTym Oct 4, 2023
f1819d9
Merge remote-tracking branch 'upstream/feat/233-update-api' into deve…
Oct 4, 2023
7d4e266
fixed on_device for ntt and lde
DmytroTym Oct 4, 2023
ec67196
CamelCase
DmytroTym Oct 5, 2023
d6dfd59
fixed msm_test, int unification, google guilde
DmytroTym Oct 5, 2023
09ef4fd
Merge branch 'feat/233-update-api' into cuda_refactoring
DmytroTym Oct 5, 2023
6f44433
Merge commit '09ef4fdba19e6c4e4745c80062bb02a9bcc70d87' into develop/…
Oct 6, 2023
fda01fa
wip - ntt crash debugging
Oct 10, 2023
028bed1
Hotfix to go regression when 2 curves are imported (#245)
DmytroTym Oct 12, 2023
1d892b2
Merge branch 'feat/233-update-api' into develop/dima/feat-233-update-…
DmytroTym Oct 13, 2023
1cf7b2e
Exclude target directory from format checks (#247)
jeremyfelder Oct 16, 2023
88c9c85
[CI]: Add concurrency groups at workflow level (#238)
jeremyfelder Oct 17, 2023
09d8c5d
Fix readme badge links, fix CI cpp formatter (#249)
jeremyfelder Oct 17, 2023
fd62fe5
Support bw6-761 (#188)
liuxiaobleach Oct 21, 2023
3e7d7b9
async MSM with a rust wrapper
DmytroTym Oct 21, 2023
141b1c8
wip ntt tests with corretness
Oct 21, 2023
a199bc8
hotfix for correctness > 2^9
Oct 22, 2023
ea00037
wip on device inout mixing with correctness
Oct 24, 2023
b8a06d5
cleanup
Oct 24, 2023
13f75a2
preserving twiddles after first call
Oct 24, 2023
6bcc56b
fixed twiddles preserving
Oct 24, 2023
bb36b1b
formatting
Oct 24, 2023
466b669
removed some printing
Oct 24, 2023
ac5b850
disable ecntt temporarily
Oct 24, 2023
bb98347
format
Oct 24, 2023
434a010
rust fmt
Oct 24, 2023
70eeada
exclude target from format
Oct 24, 2023
286811f
passing ntt after merge
Oct 24, 2023
1b2c03b
hotfix for linking issue
Oct 24, 2023
20ca1e7
format
Oct 24, 2023
0b2bd78
Merge commit '133a1b28bc04900272e7121692c2e5bc2d8dcbcf' into develop/…
Oct 25, 2023
27d2c8f
format
Oct 25, 2023
ec86b44
draft of pr comments + correctness restored
Oct 28, 2023
9556ecf
wip refactor + format
Oct 28, 2023
b0d0215
domain wip
Oct 29, 2023
d0569b4
rust format
Nov 1, 2023
dd628ed
Merged feature branch in and Rust MSM correctness
DmytroTym Nov 1, 2023
36442ae
rust build for correct curve
DmytroTym Nov 1, 2023
ac6593d
Slowdown fixed by passing release flag to cmake
DmytroTym Nov 2, 2023
32eaef1
Merge remote-tracking branch 'vitaliy/develop/vhnat/feat-233-update-a…
DmytroTym Nov 2, 2023
a469fb5
fix typos in logs (#255)
vuittont60 Nov 2, 2023
5c868ab
Fix typos (#257)
omahs Nov 2, 2023
eb543f0
WIP field and curve
DmytroTym Nov 4, 2023
318c6f4
still wip field and curve
DmytroTym Nov 4, 2023
e4e9130
Two curve NTT correctness issue hotfix (#254)
DmytroTym Nov 5, 2023
4ad37ae
field and curve in rust 1.0
DmytroTym Nov 5, 2023
7cc105d
Refactored rust into several crates
DmytroTym Nov 7, 2023
e13bf8b
Arkworks is now an option, bn254 crate created
DmytroTym Nov 9, 2023
45c9b00
Rust msm and ntt wip
DmytroTym Nov 11, 2023
29cad66
include colab (#261)
ImmanuelSegol Nov 13, 2023
7baea7c
separable compilation for Rust (#244)
vhnatyk Nov 16, 2023
1623f01
A version of rust msm done, cuda runtime wrapped
DmytroTym Nov 17, 2023
69dc6c1
refactor rust by creating a curve folder
DmytroTym Nov 20, 2023
e34e936
vec_ops instead of lde for now
DmytroTym Nov 20, 2023
5a96065
format
DmytroTym Nov 20, 2023
5813f29
Update bug_issue.md
jeremyfelder Nov 26, 2023
2790f18
Update feature_request.md (#278)
jeremyfelder Nov 26, 2023
856629d
Rust/large bucket factor msm (#271)
jeremyfelder Nov 26, 2023
fad317a
Docs: Clarify enabling tests in cmake build process (#288)
bigsky77 Nov 28, 2023
c42c26f
fixed cmake
DmytroTym Nov 29, 2023
f8610dd
Improved modular multiplier (#289)
DmytroTym Dec 5, 2023
af7de7d
Very rough CUDA-side NTT API blueprint
DmytroTym Dec 5, 2023
bd9ccc6
Another iteration of ntt.cuh
DmytroTym Dec 5, 2023
0c87502
fix
DmytroTym Dec 5, 2023
8a824d7
CUDA-side NTT that compiles
DmytroTym Dec 7, 2023
00f965e
Rust NTT prototype v0.1
DmytroTym Dec 8, 2023
14103a7
forgotten files
DmytroTym Dec 8, 2023
d764b0d
Merge commit 'dfa5b10adbf7f63f3a3d0e0adb5cfc7b10508b19' into develop/…
DmytroTym Dec 9, 2023
0a55625
Fixed NTT correctness for all sizes, cmake patched
DmytroTym Dec 10, 2023
03c2592
NTT: separated butterfly from inverse/direct NTT
DmytroTym Dec 11, 2023
98a3030
NTT: in coset only gen is exposed; inv twiddles removed
DmytroTym Dec 12, 2023
8887761
Merge branch 'main' into develop/dima/feat-233-update-rust-apis
DmytroTym Dec 12, 2023
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .github/ISSUE_TEMPLATE/bug_issue.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@
name: ":bug: Bug Report"
about: Create a bug report to help us improve the repo
title: "[BUG]: "
labels: bug
labels: type:bug
---

## Description
Expand Down
2 changes: 1 addition & 1 deletion .github/ISSUE_TEMPLATE/feature_request.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@
name: ":sparkles: Feature Request"
about: Request the inclusion of a new feature or functionality
title: "[FEAT]: "
labels: enhancement
labels: type:feature
---

## Description
Expand Down
101 changes: 0 additions & 101 deletions .github/workflows/dev-pr.yml

This file was deleted.

8 changes: 7 additions & 1 deletion .github/workflows/main-build.yml
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,10 @@ on:
branches:
- main

concurrency:
group: ${{ github.workflow }}-${{ github.ref }}
cancel-in-progress: true

env:
CARGO_TERM_COLOR: always
ARCH_TYPE: native
Expand Down Expand Up @@ -59,9 +63,11 @@ jobs:
cuda: '12.0.0'
method: 'network'
# https://docs.nvidia.com/cuda/archive/12.0.0/cuda-installation-guide-microsoft-windows/index.html
sub-packages: '["cudart", "nvcc", "thrust"]'
sub-packages: '["cudart", "nvcc", "thrust", "visual_studio_integration"]'
- name: Build Rust Targets
if: needs.check-changed-files.outputs.rust == 'true' || needs.check-changed-files.outputs.cpp_cuda == 'true'
env:
CUDA_PATH: ${{steps.cuda-toolkit.outputs.CUDA_PATH}}
run: cargo build --release --verbose

build-golang-linux:
Expand Down
9 changes: 6 additions & 3 deletions .github/workflows/main-format.yml
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,10 @@ on:
branches:
- main

concurrency:
group: ${{ github.workflow }}-${{ github.ref }}
cancel-in-progress: true

jobs:
formatting-rust:
name: Check Rust Code Formatting
Expand Down Expand Up @@ -33,6 +37,5 @@ jobs:
- name: Checkout
uses: actions/checkout@v3
- name: Check clang-format
run: |
if [[ $(find ./ -path ./icicle/build -prune -iname *.h -or -iname *.cuh -or -iname *.cu -or -iname *.c -or -iname *.cpp | xargs clang-format --dry-run -ferror-limit=1 -style=file 2>&1) ]]; then echo "Please run clang-format"; exit 1; fi

run: unformatted_files=$(find ./ -path ./icicle/build -prune -o -path ./target -prune -iname *.h -or -iname *.cuh -or -iname *.cu -or -iname *.c -or -iname *.cpp | xargs clang-format --dry-run -ferror-limit=1 -style=file >&2); if [[ $unformatted_files ]]; then echo $unformatted_files; echo "Please run clang-format"; exit 1; fi

6 changes: 5 additions & 1 deletion .github/workflows/main-test.yml
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,10 @@ on:
branches:
- main

concurrency:
group: ${{ github.workflow }}-${{ github.ref }}
cancel-in-progress: true

env:
CARGO_TERM_COLOR: always
ARCH_TYPE: native
Expand Down Expand Up @@ -57,7 +61,7 @@ jobs:
if: needs.check-changed-files.outputs.cpp_cuda == 'true'
run: |
mkdir -p build
cmake -S . -B build
cmake -DBUILD_TESTS=ON -S . -B build
cmake --build build
- name: Run C++ Tests
working-directory: ./icicle/build
Expand Down
89 changes: 64 additions & 25 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,13 +5,15 @@
![image (4)](https://user-images.githubusercontent.com/2446179/223707486-ed8eb5ab-0616-4601-8557-12050df8ccf7.png)


<div align="center">

![Build status](https://github.com/ingonyama-zk/icicle/actions/workflows/main-build.yml/badge.svg)
![Discord server](https://img.shields.io/discord/1063033227788423299?label=Discord&logo=Discord&logoColor=%23&style=plastic)
![Follow us on twitter](https://img.shields.io/twitter/follow/Ingo_zk?style=social)

</div>
<p align="center">
<img src="https://github.com/ingonyama-zk/icicle/actions/workflows/main-build.yml/badge.svg" alt="Build status">
<a href="https://discord.gg/EVVXTdt6DF">
<img src="https://img.shields.io/discord/1063033227788423299?logo=discord" alt="Chat with us on Discord">
</a>
<a href="https://twitter.com/intent/follow?screen_name=Ingo_zk">
<img src="https://img.shields.io/twitter/follow/Ingo_zk?style=social&logo=twitter" alt="Follow us on Twitter">
</a>
</p>

## Background

Expand All @@ -34,6 +36,7 @@ ICICLE is a CUDA implementation of general functions widely used in ZKP. ICICLE
- [BLS12-381]
- [BLS12-377]
- [BN254]
- [BW6-671]

## Build and usage

Expand All @@ -43,6 +46,10 @@ ICICLE is a CUDA implementation of general functions widely used in ZKP. ICICLE
- [NVCC] (version 12.0 or newer)
- cmake 3.18 and above
- follow [these instructions](https://github.com/ingonyama-zk/icicle/tree/main/icicle#prerequisites-on-ubuntu)
- Any Nvidia GPU

If you don't have access to a Nvidia GPU check out [google-colab](#google-colab). If you require more compute power and are looking to build or do research with ICICLE refer to our [grant program][GRANT_PROGRAM].


### Steps

Expand All @@ -59,6 +66,14 @@ nvcc -o build/<binary_name> ./icicle/curves/index.cu -lib -arch=native

We are using [googletest] library for testing. To build and run [the test suite](./icicle/README.md) for finite field and elliptic curve arithmetic, run from the `icicle` folder:

For testing, ensure the `BUILD_TESTS` option is enabled in cmake. If not, toggle it on by adding `-DBUILD_TESTS=ON` in the cmake configuration command:

```sh
cmake -S . -B build -DBUILD_TESTS=ON
```

Proceed with the following commands:

```sh
mkdir -p build
cmake -S . -B build
Expand All @@ -68,6 +83,7 @@ cd build && ctest

NOTE: If you are using cmake versions < 3.24 add `-DCUDA_ARCH=<target_cumpute_arch>` to the command `cmake -S . -B build`


### Rust Bindings

For convenience, we also provide rust bindings to the ICICLE library for the following primitives:
Expand All @@ -82,7 +98,7 @@ For convenience, we also provide rust bindings to the ICICLE library for the fol
- Scalar Vector Multiplication
- Point Vector Multiplication

A custom [build script][B_SCRIPT] is used to compile and link the ICICLE library. The environement variable `ARCH_TYPE` is used to determine which GPU type the library should be compiled for and it defaults to `native` when it is not set allowing the compiler to detect the installed GPU type.
A custom [build script][B_SCRIPT] is used to compile and link the ICICLE library. The environment variable `ARCH_TYPE` is used to determine which GPU type the library should be compiled for and it defaults to `native` when it is not set allowing the compiler to detect the installed GPU type.

> NOTE: A GPU must be detectable and therefore installed if the `ARCH_TYPE` is not set.

Expand Down Expand Up @@ -115,20 +131,21 @@ Create a JSON file with the curve parameters. The curve is defined by the follow
- ``curve_name`` - e.g. ``bls12_381``.
- ``modulus_p`` - scalar field modulus (in decimal).
- ``bit_count_p`` - number of bits needed to represent `` modulus_p`` .
- ``limb_p`` - number of bytes needed to represent `` modulus_p`` (rounded).
- ``ntt_size`` - log of the maximal size subgroup of the scalar field.
- ``limb_p`` - number of (32-bit) limbs needed to represent `` modulus_p`` (rounded up).
- ``ntt_size`` - log of the maximal size subgroup of the scalar field.
- ``modulus_q`` - base field modulus (in decimal).
- ``bit_count_q`` - number of bits needed to represent `` modulus_q`` .
- ``limb_q`` number of bytes needed to represent `` modulus_p`` (rounded).
- ``weierstrass_b`` - Weierstrauss constant of the curve.
- ``weierstrass_b_g2_re`` - Weierstrauss real constant of the g2 curve.
- ``weierstrass_b_g2_im`` - Weierstrauss imaginary constant of the g2 curve.
- ``gen_x`` - x-value of a generator element for the curve.
- ``gen_y`` - y-value of a generator element for the curve.
- ``gen_x_re`` - real x-value of a generator element for the g2 curve.
- ``gen_x_im`` - imaginary x-value of a generator element for the g2 curve.
- ``gen_y_re`` - real y-value of a generator element for the g2 curve.
- ``gen_y_im`` - imaginary y-value of a generator element for the g2 curve.
- ``limb_q`` - number of (32-bit) limbs needed to represent `` modulus_q`` (rounded up).
- ``weierstrass_b`` - `b` of the curve in Weierstrauss form.
- ``weierstrass_b_g2_re`` - real part of the `b` value in of the g2 curve in Weierstrass form.
- ``weierstrass_b_g2_im`` - imaginary part of the `b` value in of the g2 curve in Weierstrass form.
- ``gen_x`` - `x` coordinate of a generator element for the curve.
- ``gen_y`` - `y` coordinate of a generator element for the curve.
- ``gen_x_re`` - real part of the `x` coordinate of generator element for the g2 curve.
- ``gen_x_im`` - imaginary part of the `x` coordinate of generator element for the g2 curve.
- ``gen_y_re`` - real part of the `y` coordinate of generator element for the g2 curve.
- ``gen_y_im`` - imaginary part of the `y` coordinate of generator element for the g2 curve.
- ``nonresidue`` - nonresidue, or `i^2`, or `u^2` - square of the element that generates quadratic extension field of the base field.

Here's an example for BLS12-381.
```
Expand All @@ -142,14 +159,15 @@ Here's an example for BLS12-381.
"bit_count_q" : 381,
"limb_q" : 12,
"weierstrass_b" : 4,
"weierstrass_b_g2_re":4,
"weierstrass_b_g2_im":4,
"weierstrass_b_g2_re" : 4,
"weierstrass_b_g2_im" : 4,
"gen_x" : 3685416753713387016781088315183077757961620795782546409894578378688607592378376318836054947676345821548104185464507,
"gen_y" : 1339506544944476473020471379941921221584933875938349620426543736416511423956333506472724655353366534992391756441569,
"gen_x_re" : 352701069587466618187139116011060144890029952792775240219908644239793785735715026873347600343865175952761926303160,
"gen_x_im" : 3059144344244213709971259814753781636986470325476647558659373206291635324768958432433509563104347017837885763365758,
"gen_y_re" : 1985150602287291935568054521177171638300868978215655730859378665066344726373823718423869104263333984641494340347905,
"gen_y_im" : 927553665492332455747201965776037880757740193453592970025027978793976877002675564980949289727957565575433344219582
"gen_y_im" : 927553665492332455747201965776037880757740193453592970025027978793976877002675564980949289727957565575433344219582,
"nonresidue" : -1
}
```

Expand All @@ -167,17 +185,27 @@ The script does the following:
- Creates a file with the curve name in ``src/curves`` with the relevant objects for the curve.
- Creates a test file with the curve name in ``src``.

Also files from ``./icicle/curves/<curve_name>/supported_operations.cu`` should be added individually to ``add_library`` section of [``./icicle/CMakeLists.txt``][CMAKELISTS]

Testing the new curve could be done by running the tests in ``tests_curve_name`` (e.g. ``tests_bls12_381``).

## Docker

We offer a simple Docker container so you can simply run ICICLE without settig everything up locally.
We offer a simple Docker container so you can simply run ICICLE without setting everything up locally.

```
docker build -t <name_of_your_choice> .
docker run --gpus all -it <name_of_your_choice> /bin/bash
```

## Google Colab

[Colab](https://colab.google/) is a hosted Jupyter Notebook service that requires no setup to use and provides free access to computing resources including GPUS!

You can easily run ICICLE in Google Colab on a free GPU instance, this is a great option for those who want to get started with ICICLE instantly without any local setup or GPU.

Follow this [guide][GOOGLE_COLAB_ICICLE] for more details.

## Contributions

Join our [Discord Server][DISCORD] and find us on the icicle channel. We will be happy to work together to support your use case and talk features, bugs and design.
Expand All @@ -190,12 +218,19 @@ If you are changing code, please make sure to change your [git hooks path][HOOKS
git config core.hooksPath ./scripts/hooks
```

In case `clang-format` is missing on your system, you can install it using the following command:

```sh
sudo apt install clang-format
```

This will ensure our custom hooks are run and will make it easier to follow our coding guidelines.

### Hall of Fame

- [Robik](https://github.com/robik75), for his on-going support and mentorship
- [Robik](https://github.com/robik75), for his ongoing support and mentorship
- [liuxiao](https://github.com/liuxiaobleach), for being a top notch bug smasher
- [gkigiermo](https://github.com/gkigiermo), for making it intuitive to use ICICLE in Google Colab.

## Help & Support

Expand All @@ -212,6 +247,7 @@ See [LICENSE-MIT][LMIT] for details.
[BLS12-381]: ./icicle/curves/bls12_381/supported_operations.cu
[BLS12-377]: ./icicle/curves/bls12_377/supported_operations.cu
[BN254]: ./icicle/curves/bn254/supported_operations.cu
[BW6-671]: ./icicle/curves/bw6_671/supported_operations.cu
[NVCC]: https://docs.nvidia.com/cuda/#installation-guides
[CRV_TEMPLATE]: ./icicle/curves/curve_template/
[CRV_CONFIG]: ./icicle/curves/index.cu
Expand All @@ -222,5 +258,8 @@ See [LICENSE-MIT][LMIT] for details.
[googletest]: https://github.com/google/googletest/
[HOOKS_DOCS]: https://git-scm.com/docs/githooks
[HOOKS_PATH]: ./scripts/hooks/
[CMAKELISTS]: https://github.com/ingonyama-zk/icicle/blob/f0e6b465611227b858ec4590f4de5432e892748d/icicle/CMakeLists.txt#L28
[GOOGLE_COLAB_ICICLE]: https://github.com/gkigiermo/rust-cuda-colab
[GRANT_PROGRAM]: https://docs.google.com/forms/d/e/1FAIpQLSc967TnNwxZZ4akejcSi4KOUmGrEc68ZZV-FHLfo8KnP1wbpg/viewform

<!-- End Links -->
5 changes: 3 additions & 2 deletions curve_parameters/bls12_377.json
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@
"modulus_p" : 8444461749428370424248824938781546531375899335154063827935233455917409239041,
"bit_count_p" : 253,
"limb_p" : 8,
"ntt_size" : 32,
"ntt_size" : 47,
"modulus_q" : 258664426012969094010652733694893533536393512754914660539884262666720468348340822774968888139573360124440321458177,
"bit_count_q" : 377,
"limb_q" : 12,
Expand All @@ -16,5 +16,6 @@
"g2_gen_x_re" : 233578398248691099356572568220835526895379068987715365179118596935057653620464273615301663571204657964920925606294,
"g2_gen_x_im" : 140913150380207355837477652521042157274541796891053068589147167627541651775299824604154852141315666357241556069118,
"g2_gen_y_re" : 63160294768292073209381361943935198908131692476676907196754037919244929611450776219210369229519898517858833747423,
"g2_gen_y_im" : 149157405641012693445398062341192467754805999074082136895788947234480009303640899064710353187729182149407503257491
"g2_gen_y_im" : 149157405641012693445398062341192467754805999074082136895788947234480009303640899064710353187729182149407503257491,
"nonresidue" : -5
}
Loading