Efficient scalar mul and other Scalar improvements #1530

mitschabaude · 2024-04-01T08:36:39Z

This revises the scalar multiplication gadgets and internal representation of Scalar.

BEFORE:

Scalar used to be represented as 255 bits, which imposed an unavoidable cost of 255 rows on any gadget that wanted to scale a group element by a Field. This is a very typical use case for scaling, see Signature.verify() and Nullifier.verify()
On top of that, the 255-bit representation was shifted and we lacked a good way of handling that. As a hack, we used a scaleShifted() helper which unshifts "in the exponent" at the cost of doing 3 full scalar muls instead of 1.
Group.scale() didn't support -1, 0 and 1 as scalars

AFTER:

Scalar uses the representation that Pickles uses as well and which our scalar mul gadgets were designed for: 1 low bit and 254 high bits. The high bits are a single field element. The representation is still shifted, but in a slightly different way than before.
We add efficient gadgets for scaling by a Field, and for converting a Field to a Scalar
We no longer expose shifting in any API: Scalar.fromBits() performs the shift behind the scenes in provable code now, as do Group.scale() and Scalar.fromField()
Group.scale() supports all scalars, and there's a new test to check that
Some results:
- Signature.verify() now uses 350 instead of 800 rows
- The nullifier example zkapp uses 9790 instead of 10774 rows

Closes #1440
Closes #984
Closes #980

This reverts commit 04f8648.

mitschabaude · 2024-04-08T19:20:52Z

src/lib/provable/field.ts

@@ -319,24 +320,7 @@ class Field {
   * See {@link Field.isEven} for examples.
   */
  isOdd() {
-    if (this.isConstant()) return new Bool((this.toBigInt() & 1n) === 1n);


I moved this gadget into gadgets/comparison.ts to reuse it for scaleField()

mitschabaude · 2024-04-08T19:22:51Z

src/lib/provable/gadgets/native-curve.ts

this is where they new scaling gadgets live! needs careful crypto review

mitschabaude · 2024-04-08T19:23:52Z

src/lib/provable/group.ts

@@ -101,67 +106,49 @@ class Group {
        return fromProjective(g_proj);
      }
    } else {
-      const { x: x1, y: y1 } = this;


this code was moved to native-group.ts to reuse there. it was also rewritten a bit and a bug fixed

mitschabaude · 2024-04-08T19:24:55Z

src/lib/provable/group.ts

+   * **Warning**: If one of the inputs is zero, the result will be garbage and the proof useless.
+   * This case has to be prevented or handled separately by the caller of this method.
+   */
+  addNonZero(g2: Group, allowZeroOutput = false): Group {


I added this, I think it's actually a bit more common to use this than fully complete addition

mitschabaude · 2024-04-08T19:25:55Z

src/lib/provable/test/foreign-field.unit-test.ts

@@ -108,83 +102,3 @@ equivalent({ from: [f], to: f })(
    return ForeignScalar.fromBits(bits);
  }
 );
-
-// scalar shift in foreign field arithmetic vs in the exponent


this test was no longer valid, so I removed it

mitschabaude · 2024-04-08T19:26:49Z

src/lib/provable/test/test-utils.ts

@@ -62,6 +66,26 @@ function bytes(length: number) {
  });
 }

+function pointSpec<T>(field: ProvableSpec<bigint, T>, Curve: CurveAffine) {


just a testing utility, ends up unused (I used it during development to monitor the size of some circuits)

mitschabaude · 2024-04-08T19:27:15Z

src/snarky.d.ts

+     * `numBits <= n - 2` where `n` is the bit length of the scalar field.
+     * In our case, n=255 so numBits <= 253.
+     */
+    scaleFastUnpack(


this is the scaling gadget that Pickles uses as well

mitschabaude · 2024-04-08T19:27:46Z

tests/vk-regression/plain-constraint-system.ts

@@ -173,6 +174,16 @@ const BasicCS = constraintSystem('Basic', {
  },
 });

+const CryptoCS = constraintSystem('Crypto', {


new vk regression test

querolita · 2024-04-09T17:14:32Z

src/lib/provable/gadgets/scalar.ts

+  let tHi0 = existsOne(() => t0.toBigInt() >> 5n);
+
+  // prove split
+  // since we know that t0 < 2^88, this proves that t0High < 2^83


tHi0 you mean?

I'm afraid you're reviewing outdated code :O
yeah I meant tHi0

yeah, yesterday I was commit-reviewing to gain more context. Today I will focus on the updated native-curve.ts file

src/lib/provable/gadgets/scalar.ts

querolita · 2024-04-09T17:27:01Z

src/lib/provable/gadgets/scalar.ts

@@ -61,7 +61,7 @@ function scale(P: { x: Field; y: Field }, s: Field): Group {

 /**
 * Internal helper to compute `(t + 2^254)*P`.
- * `t` is expected to be split into 250 high bits (t >> 5) and 5 low bits (t & 0xf1).
+ * `t` is expected to be split into 250 high bits (t >> 5) and 5 low bits (t & 0x1f).


ah okay, that's now 5 bits

querolita · 2024-04-10T12:38:05Z

src/lib/provable/gadgets/scalar.ts

+  let tHi0 = existsOne(() => t0.toBigInt() >> 5n);
+
+  // prove split
+  // since we know that t0 < 2^88, this proves that t0High < 2^83


yeah, yesterday I was commit-reviewing to gain more context. Today I will focus on the updated native-curve.ts file

querolita

Very clear comments to understand the technique for an outsider. Couldn't find anything fishy after review.

src/lib/provable/gadgets/native-curve.ts

mitschabaude added 30 commits April 1, 2024 10:36

simplify internal repr of scalar

5008583

expose scale_fast_unpack

5420cb2

first attempt at efficient scale gadget

b8c19b9

wip debugging

a00a15a

fix accessing ml curve point

57de1f4

improve efficiency by using incomplete additions

2eb6354

rename gadget file

d659493

refactor 3 statements to loop

43a0637

move addition gadget to gadget file

dcc58ee

remove cyclic dependency on group

2192f29

simplify witness generation code

bb0716c

change scalar type to shifted 5 / 250 representation

ddbec87

bring back fromBits, remove unused shifting methods

011d6a8

add scale from field element to test

bc9b4ae

bindings

b508437

dump vks

0f93cc2

switch to repr with 1 low bit

04f8648

but scaling with 0 or 1 doesn't work now

9f11f0c

0 doesn't work anyway

32fd2b3

vk regression

7ce21a7

fix

5bc9543

Revert "switch to repr with 1 low bit"

4c06ce1

This reverts commit 04f8648.

add missing range checks, document more assumptions, add comments

6886184

remove scalar limitations from group unit test

8fbfe9b

dump vks

5107a95

move isOdd gadget to reuse it

d8fb076

dedicated gadget of 130 rows for scaling by Field

51ee9f5

use 1, 254 split and handle edge cases

f88222d

reduce constraints

00680ed

tighten fromBits gadget

0aad84e

mitschabaude added 11 commits April 8, 2024 16:15

remove unused split5 gadgets

37534df

dump vks

ad7e7a3

renaming and test utils

8d5d338

test utils

494e627

minor

a8fea23

Merge branch 'main' into feature/no-shifted-scale

fddead2

changelog

35eb475

remove shifting stuff from foreign field unit test

7bcc84b

comment

e21fa33

add nullifier to vk-regression

9c799ed

mina

9e9ca06

mitschabaude marked this pull request as ready for review April 8, 2024 19:16

mitschabaude commented Apr 8, 2024

View reviewed changes

mitschabaude mentioned this pull request Apr 8, 2024

Efficient scalar mul and other Scalar improvements o1-labs/o1js-bindings#260

Merged

mitschabaude added the v1 Prerequisite for o1js v1.0 label Apr 9, 2024

querolita reviewed Apr 9, 2024

View reviewed changes

Trivo25 approved these changes Apr 9, 2024

View reviewed changes

querolita reviewed Apr 10, 2024

View reviewed changes

mitschabaude added 3 commits April 10, 2024 16:06

Merge branch 'main' into feature/no-shifted-scale

4b4787e

fix changelog

224b95c

dump vks

dab6eb3

querolita approved these changes Apr 15, 2024

View reviewed changes

src/lib/provable/gadgets/native-curve.ts Outdated Show resolved Hide resolved

mitschabaude added 4 commits April 15, 2024 21:09

Merge branch 'main' into feature/no-shifted-scale

0877e26

fixes and tweaks to scaling gadgets

523330a

remove redundant constraint

45837fc

dump vks

9822654

mitschabaude merged commit cfdb2a5 into main Apr 16, 2024
12 checks passed

mitschabaude deleted the feature/no-shifted-scale branch April 16, 2024 08:49

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Efficient scalar mul and other Scalar improvements #1530

Efficient scalar mul and other Scalar improvements #1530

mitschabaude commented Apr 1, 2024 •

edited

mitschabaude Apr 8, 2024

mitschabaude Apr 8, 2024

mitschabaude Apr 8, 2024

mitschabaude Apr 8, 2024

mitschabaude Apr 8, 2024

mitschabaude Apr 8, 2024

mitschabaude Apr 8, 2024

mitschabaude Apr 8, 2024

querolita Apr 9, 2024

mitschabaude Apr 9, 2024

querolita Apr 10, 2024

querolita Apr 9, 2024

querolita Apr 10, 2024

querolita left a comment

Efficient scalar mul and other Scalar improvements #1530

Efficient scalar mul and other Scalar improvements #1530

Conversation

mitschabaude commented Apr 1, 2024 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

querolita left a comment

Choose a reason for hiding this comment

mitschabaude commented Apr 1, 2024 •

edited