BLS extensions

[arvidn]

This is a patch set, please review one commit at a time.

The BLS extension feature is not quite complete yet, it's missing:

• updated cost measurements
• a unit test of validating a zk-rollup (I might punt on this to a separate PR)

BLS softfork extension

These new operators are added under the BLS softfork extension operator set. This set currently only contains the new coinid operator. They will be available under the invocation: (softfork <cost> 0 <program> <env>). This operator set is also enabled outside the softfork guard when run with the hard-fork flag set. The intention is to make all of these operators available by default after the 2.0 hard fork activates.

unit tests

With this feature, I'm adding quite a lot of new test cases for operators, so the files test-core-ops.txt, test-more-ops.txt etc. were moved into a sub directory, op-tests.

There's a new python script. tools/generate-bls-tests.py, which generates the op-tests/test-blspy-*.txt test cases. This primarily ensures that the BLS operations are equivalent to the corresponding operations in blspy (which is what we use in the full node)

Some of these tests can be quite slow, especially in debug builds. In order to speed up cargo test, I made test_ops parameterized on the different test files to run them in parallel.

new operators

name	arguments	opcode	description
g1_add	(G1 G1 ...)	29	This is an alias for the existing point_add operator.
g1_subtract	(G1 G1 ... )	49	Returns the first G1 point argument minus all following G1 points. no arguments yields the G1 Identity.
g1_multiply	(G1 scalar)	50	Returns the specified G1 point multiplied by the scalar.
g1_negate	(G1)	51	Returns the negated G1 point.
g2_add	(G2 G2 ...)	52	Returns the sum of all specified G2 points. Zero arguments yields the G2 identity.
g2_subtract	(G2 G2 ... )	53	Returns the first G2 point argument minus all following G2 points. Zero arguments yields the G2 Identity.
g2_multiply	(G2 scalar)	54	Returns the specified G2 point multiplied by the scalar.
g2_negate	(G2)	55	Returns the negated G2 point.
g1_map	(msg [dst])	56	Returns the G1 point msg hashes to.
g2_map	(msg [dst])	57	Returns the G2 point msg hashes to.
bls_pairing_identity	(G1 . G2) (G1 . G2) ...	58	Returns 1 if the pairing of all G1 and G2 pairs is the (Gt) identity and null otherwise.
bls_verify	G2 (G1 . msg) (G1 . msg) ...	59	Validates the messages (msg) given their public key (G1) against the signature (G2).

g1_map

Hashes the specified message to G1 using SHA-256 and ExpandMsgXmd, using the specified dst (domain separation tag). dst is optional, defaulting to BLS_SIG_BLS12381G1_XMD:SHA-256_SSWU_RO_AUG_.

g2_map

Hashes the specified message to G2 using SHA-256 and ExpandMsgXmd, using the specified dst (domain separation tag). dst is optional, defaulting to BLS_SIG_BLS12381G2_XMD:SHA-256_SSWU_RO_AUG_.

bls_verify

Returns 1 if the signature is valid, null otherwise. The validation uses the Augmented scheme, which means the public keys (G1) are prepended to the messages before being hashed to G2 points. The domain separation tag is BLS_SIG_BLS12381G2_XMD:SHA-256_SSWU_RO_AUG_. The validation includes the pair of the negated G1 generator and the signature in the underlying pairing operation.

Changes from Cameron's patch

This is based on Cameron's original BLS operator patch with the following main changes:

• all Gt types and operations were removed. This means we no longer need to patch bls12_381 to support serializing Gt points.
• the bls pairing operation does not return the Gt point, but just a bool indicating whether it's the identity or not. This is the check we care about for validating signatures.
• there's a new, higher level, paring operation that performs a full (aggregated) signature validation in the Augmented scheme. This is the scheme we use in AGG_SIG_ME and AGG_SIG_UNSAFE
• The pairing operations take a list of pairs. (G1, G2)-pairs for bls_pairing_identity and (G1, message)-pairs for bls_verify. The pairs are passed as single cons boxes with the two elements as its children. They used to be passed as lists of 2 items.
• the pairing operator used to take either one list of pairs or two parameters, interpreted as a single pair. The special case of a single pair was dropped.
• the opcodes for the operators were changed

[richardkiss]

One comment I want to make is in a long sequence of operations, it's a lot cheaper to keep numbers in projective form and defer normalizing to affine form until the end. Projective form is analogous to a fraction A/B; your (X, Y) becomes (X, Y, Z) and the "real" X and Y values are X/Z and Y/Z. For finite fields, the denominator B is not unique, because you can multiple the top and bottom by N/N for any N. To normalize, you multiply through by inv(Z), so the Z coordinate is 1 (so affine (X, Y) is just (X, Y, 1) in projective coordinate).

This inversion is expensive, and deferring it to the end to provide a canonical affine representation can greatly reduce the number of inversions needed over a series of operations. So it might be useful to perform all operations in the projective space, and automatically turn affine coordinates into projective (by setting Z = 1), and require an affine normalization at the end before any comparison or whatever is done. It might even be possible to stay in projective coordinates the whole way through.

[arvidn]

@richardkiss I think the way we should solve the issue you raise is by making the Allocator interface support more kinds of atoms natively (not just bytes). That way we can at least convert BigNum, G1 and G2 points lazily into bytes (and hopefully not at all).

This strategy will likely make it a lot cheaper, but it also means the CLVM cost won't accurately represent the true compute cost anymore.

[richardkiss]

As for naming, I wonder if the bls_ prefix can be dropped for operators that start with bls_g*. I understand the desire to scope, but maybe we just let g1_ and g2_ be the scope. We might even call bls_map_to_g1 g1_map, etc.

[richardkiss]

And maybe we could call bls_pairing_identity gt_pairing_identity

[richardkiss]

We need bls_g1_add with no arguments (to check the identity).

[richardkiss]

I guess we also want some tests with one argument.

[arvidn]

we have this test case (for point_add) here: https://github.com/Chia-Network/clvm_rs/blob/main/test-more-ops.txt#L711

Generally I've split the test cases between test-bls-ops.txt for manually added test cases and the test-blspy-*.txt for the generated ones. I suppose most of the G1 and G2 points can't be validated by manual inspection, and should be generated.

[richardkiss]

Again, we want a test 0 arguments and tests with 1 argument.

[arvidn]

yes, I'll add this to test-bls-ops.txt

[richardkiss]

We want to test multiplying by 0, and multiplying by large scalars (at least 50% of the size of the order of the group, so bit 253 set). I believe multiplying by large scalars could use significantly more CPU. Well, actually probably not since I expect relic is trying to protect against side channel leak.

[arvidn]

I'll add the zero case to test-bls-ops.txt and I'll add multiplying with large scalars to the generated test cases

[richardkiss]

This seems like it's way too expensive when you consider it's just copying the value and changing a bit. There's not much point in making it cost more than logxor with a constant.

[arvidn]

This operator also validates that the point is on the curve, which isn't trivial. I could implement this as just a bit operation, but it would make it accept invalid points, I believe.

[richardkiss]

If we're concerned about invalid (ie. untrusted) points, we could have a separate operator to validate them. Or just do (g1_add untrusted_point), which should also do an implicit validation. (Just spitballing here.)

[richardkiss]

I think it's not likely that we're going to take a point and negate it and not perform any other operations on it that would validate it, so not validating here may not be a big concern.

[richardkiss]

Test with 0 and 1 parameter.

[arvidn]

the case of no arguments is covered here: 83ff075#diff-8e926fa1adc34a468edbbfcb76a3996682c288dc76919bad5a6876cbe68fde84R66

I'll add the single argument case next to it

[richardkiss]

0 & 1 parameter

[richardkiss]

And more than 2 parameters.

[arvidn]

the no argument case is covered here: 83ff075#diff-8e926fa1adc34a468edbbfcb76a3996682c288dc76919bad5a6876cbe68fde84R75
I'll add the single argument case next to it

[richardkiss]

Add tests for 0 and large scalars.

[arvidn]

I'll add the case of multiplying with 0 to test-bls-ops.txt and large scalars to these generated test cases

[richardkiss]

Same comment about costs. Literally, a single bit changes.

[richardkiss]

I'm not convinced that this shape for the arguments makes the most sense. I think need to actually implement a proof to see. If our loops produce N values from g1 and N values from g2, to call this, we'd need to do a zip operation in clvm which might better be done in the vm implementation. Compare to (bls_pairing_identity g1_list g2_list). I don't know it's better, which is why I think we need to implement the toy to see.

[arvidn]

good point. my impression so far is that this list is unlikely to be dynamic. in the snarkjs case, it's hard-coded to be 4 pairs

[arvidn]

here's the equivalent call in snarkjs' proof verification: https://github.com/iden3/snarkjs/blob/master/src/groth16_verify.js#L62-L68

It has a fixed number of arguments.

    const res = await curve.pairingEq(
        curve.G1.neg(pi_a) , pi_b,
        cpub , vk_gamma_2,
        pi_c , vk_delta_2,
        vk_alpha_1, vk_beta_2
    );

[richardkiss]

If this is typical, then yes, varargs makes the most sense.

[richardkiss]

Here, I think the signature should come first so the varargs can come last. It would Probably simplify the rust implementation too.

Oh wait, I guess it's not actually varargs... it looks like it's a single argument that's a list.

Here again I wonder if we want the zipping of g1 and g2 to be done in the VM rather than clvm. We need to come up with a use case so we can check.

[arvidn]

yeah, I think I'm leaning towards varargs rather than a list of pairs. both for bls_pairing_identity and bls_verify

[richardkiss]

We definitely want to test 0 and we want to test large scalars.

[richardkiss]

Mostly requesting more test cases, and we need to consider more whether pairings should be zipped or not.

[richardkiss]

I also find it curious that there is no access to the G1 or G2 generators. We can generate the G1 generator with (pubkey_from_exp 1) but there's no corresponding way to do it with G2. Seems weird for users to have to hard-code it. Maybe you don't need it for zk proofs (I'm not sure) but I think you do need it to prep for bls_verify since there's an implicit usage of the generators there.

[arvidn]

I suggested an operator that would just return the generator as a constant, but Bram did not like that idea. He preferred relying on compression and referencing previous blocks to de-duplicate constants like that.

[richardkiss]

I suggested an operator that would just return the generator as a constant, but Bram did not like that idea. He preferred relying on compression and referencing previous blocks to de-duplicate constants like that.

As I noted above, we already have one that implicitly uses and can create the g1 generator (well, sort of), so it's a bit strange that we don't have one for g2. Also, I'm still extremely wary of cross-block compression.

[arvidn]

I like the shorter names g1_* and g2_*. Does anyone object to this change?

[arvidn]

I have addressed all review comments (afaict). I appended individual commits for each issue, to make it easier to review.

Please review the 7 most recent commits.

I also pulled out the one commit to move the test-core-ops.txt, test-more-ops.txt etc. into its own directory. That patch has landed in main now (see #284 ). The reason for this is that it doesn't have to be part of the BLS patch and pulling it out lets me address the issue with coinid in parallel. I rebased this PR on top of main one that landed.

[arvidn]

@richardkiss is the compressed form guaranteed to just flip the most significant bit when negating? If so, perhaps it would make sense to implement this as just a validation check + the bit-flip. It's probably faster than to_compressed().

[richardkiss]

I believe so, yes. My understanding is that the encoding has a sign bit, so just figure out what that is and flip it. The only time this doesn't work is the zero (aka "infinity" just to be confusing) value, which I think has its own bit indicating 0, and (unsurprisingly) should pass through unchanged.

[arvidn]

I'm inclined to submit my patch to update the costs (along with the benchmarking tool) to a separate PR. I think this one is big enough as it is. Unless someone would prefer I amend this one.

[Quexington]

My comments have been addressed, but I don't know enough about the rust implementation here to really judge it, so my approval should be taken as an approval of the syntax and test cases only.

[richardkiss]

Curiously, this file makes test-blspy-verify.txt and test-blspy-pairing.txt a bit redundant since it can generate them. That means technically we could get away with not checking in the .txt files since they could just be generated as-needed by the CI or tests. I don't think we need to make this change at this time, but it's an interesting thing to think about.

[arvidn]

yeah. I'm tempted to make the CI ensure the generated test cases are up-to-date. It's convenient to be able to run cargo test locally without having to generate these files too.

[richardkiss]

Looks good.

[arvidn]

I'm squashing the fixup commits into their respective original commits they change:

git diff main >temp1.diff
git rebase main -i
git diff main >temp2.diff
diff temp1.diff temp2.diff

That's how I ensured re-juggling the commits didn't alter the overall change.

[coveralls-official]

Pull Request Test Coverage Report for Build 4988488162

• 203 of 231 (87.88%) changed or added relevant lines in 5 files are covered.
• No unchanged relevant lines lost coverage.
• Overall coverage increased (+0.8%) to 82.081%

---

Changes Missing Coverage	Covered Lines	Changed/Added Lines	%
src/op_utils.rs	18	20	90.0%
tools/src/bin/generate-fuzz-corpus.rs	0	3	0.0%
src/chia_dialect.rs	0	11	0.0%
src/f_table.rs	0	12	0.0%

Totals
Change from base Build 4961436774:	0.8%
Covered Lines:	1672
Relevant Lines:	2037

---

💛 - Coveralls