SIMD-0095: Extendable Output (XOF) Hashing Support #95

ankeleralph · 2023-12-14T09:57:24Z

This proposal introduces three new concepts to the Solana runtime:

Support extendable Output Functions (XOF) hashing, based on cSHAKE
Support cSHAKE as a customable
version of SHAKE
Support the STROBE protocol based on cSHAKE

Using the above new concepts would enable regular Solana programs to:

Use merlin transcripts, automating the Fiat-Shamir transform for
zero-knowledge proofs, which turns interactive proofs into non-interactive proofs
Use the widely used BulletProofs zero-knowledge library

ripatel-fd · 2023-12-29T21:51:27Z

@ankeleralph Thanks for submitting this SIMD. Somewhat related: The Firedancer team has also been working on a native new cryptographic set accumulator to replace the epoch account hash, but we haven't gotten to publishing a SIMD yet. This accumulator needs a 2048-bit hash function. Conveniently, accounts are already hashed with BLAKE3, which can be trivially expanded to an XOF while calculating the final block of the hash function.

Solana already suffers from a proliferation of too many hash functions, and it costs a lot of time to write optimized implementations for validator clients. We should therefore avoid standardizing two different XOF hash functions.

Without considering the cryptographic quality of SHAKE vs BLAKE3, I would prefer BLAKE3: It is already widely used and native to the Solana runtime. Exposing it to smart contracts is therefore also a much smaller implementation change than introducing a new hash function.

Could you amend this proposal to evaluate BLAKE3 XOF as an alternative? I'm sure there are some arguments against BLAKE3 for your proposed use cases.

ripatel-fd · 2023-12-29T21:57:30Z

proposals/0095-xof-hashing.md

+Another alternative is to implement the BulletProof zero-knowledge library as a 
+native program entirely, however, this would limit the use cases that can 
+additionally be enabled by supporting the customable extendable output functin 
+cSHAKE, and the merlin transcripts. Though, supporting a native zero-knowledge 
+proof library would likely be more efficient. 


It seems like introducing syscalls for cSHAKE would only serve to accelerate computation.

When these syscalls are introduced, usually before/after benchmarks are provided.
How much faster would your proposed syscalls be compared to an eBPF implementation?

I will re-double this message. Adding new syscalls can hurt overall performance for all smart contracts. It would be useful to see an SBPF implementation as that may be sufficient, especially if only one or a handful of apps need this right now.

I think there are much more benefits that other developers could have from existing cSHAKE syscalls. A cSHAKE implementation would enable to build customisable output length hash functions, domain separation for different protocol components and potential higher security due to customisation features. Overall it would make it a more versatile and potentially robust choice for some cryptographic operations within the Solana VM.

@ankeleralph I'm comparing against a pure eBPF implementation here, not to other syscalls. A pure eBPF implementation is even more versatile than the proposed syscall because you can optimize and extend the logic (e.g. use different parameters for the sponge function) without a hard fork.

ripatel-fd · 2023-12-29T22:03:06Z

proposals/0095-xof-hashing.md

+define_syscall!(fn sol_cshake128(vals: *const u8, val_len: u64, func_name: 
+*const u8, cust_string: *const u8, hash_result: *mut u8) -> u64);
+define_syscall!(fn sol_cshake256(vals: *const u8, val_len: u64, func_name: 
+*const u8, cust_string: *const u8, hash_result: *mut u8) -> u64);


As far as I can tell, cshake is only a thin wrapper over SHAKE and Keccak. keccak256 is already exposed via a syscall. Why not separately expose a SHAKE syscall and implement the wrapper in eBPF? This would be more flexible and have small overhead in eBPF.

Although other syscalls don't do this yet, I'd strongly recommend a batch-style API that takes multiple inputs.
On modern x86, the fastest hashing technique for the SHA2/3 and BLAKE2/3 families of hash functions is typically a SIMD approach where 8 or 16 hash states are calculated at once. This is typically 2-4x faster over hashing one message at a time.

ripatel-fd · 2023-12-29T22:07:43Z

proposals/0095-xof-hashing.md

+define_syscall!(fn sol_strobe128_ad(...) -> u64);
+define_syscall!(fn sol_strobe128_meta_ad(...) -> u64);
+define_syscall!(fn sol_strobe128_key(...) -> u64);
+define_syscall!(fn sol_strobe128_prf(...) -> u64);
+
+define_syscall!(fn sol_strobe256_ad(...) -> u64);
+define_syscall!(fn sol_strobe256_meta_ad(...) -> u64);
+define_syscall!(fn sol_strobe256_key(...) -> u64);
+define_syscall!(fn sol_strobe256_prf(...) -> u64);


I'm not convinced that separate syscalls would provide a significant speedup over an eBPF implementation of strobe that uses a batch SHAKE or Keccak syscall as described here: https://github.com/solana-foundation/solana-improvement-documents/pull/95/files#r1438427328

AFAICT, the most expensive operation in the Strobe framework is Keccak/SHAKE hashing. The rest seems to be just byte array concatenation, which is decently fast in eBPF.

Agreed. We need benchmarks to see how long the keccak portion takes vs the other work.

I think its obvious that most performance will be required by the underlying call to the sponge function (in this case cSHAKE or Keccak), as the metadata operations will not require a lot of overhead. Let me know if you want to see some more concrete numbers for benchmarks, happy to quickly do a few to support the adoption of this proposal.

Can we remove these metadata operations then? My overall concern as a validator maintainer is the maintenance burden.

It would take me a week to build cross-client testing infrastructure for all of these syscalls. This would also include formal verification (https://saw.galois.com/) of the Rust (Labs/Agave) and C (Firedancer) implementations for equivalence. All this overhead would be unnecessary if implemented in eBPF.

lheeger-jump

Overall the proposal is a bit confusing. It would be useful to know who would actually be the concrete app(s)/user(s) of all of these new syscalls and hashing formulations. Its seems as though the STROBE bits can be implemented in the VM and do not require their own syscalls. Benchmarks will also be necessary. I would recommend also addressing @ripatel-fd's comments.

lheeger-jump · 2024-01-02T21:12:05Z

proposals/0095-xof-hashing.md

+
+This proposal introduces three new concepts to the Solana runtime:
+
+- Support extendable Output Functions (XOF) hashing, based on cSHAKE 


BLAKE3 has XOF, could we not make the existing syscall for BLAKE3 just support XOF?

Regarding, cSHAKE vs BLAKE3, as @samkim-crypto already mentioned this would require developers to change the proof generation as well, which would create an additional barrier to move the proof verification on-chain.

lheeger-jump · 2024-01-02T21:14:24Z

proposals/0095-xof-hashing.md

+Another alternative is to implement the BulletProof zero-knowledge library as a 
+native program entirely, however, this would limit the use cases that can 
+additionally be enabled by supporting the customable extendable output functin 
+cSHAKE, and the merlin transcripts. Though, supporting a native zero-knowledge 
+proof library would likely be more efficient. 


I will re-double this message. Adding new syscalls can hurt overall performance for all smart contracts. It would be useful to see an SBPF implementation as that may be sufficient, especially if only one or a handful of apps need this right now.

lheeger-jump · 2024-01-02T21:30:13Z

proposals/0095-xof-hashing.md

+
+#### Implementation Details
+
+The Strobe designers released an official implementation in C available at 


It is unlikely the FD will use this implementation and will roll our own.

lheeger-jump · 2024-01-02T21:31:19Z

proposals/0095-xof-hashing.md

+define_syscall!(fn sol_strobe128_ad(...) -> u64);
+define_syscall!(fn sol_strobe128_meta_ad(...) -> u64);
+define_syscall!(fn sol_strobe128_key(...) -> u64);
+define_syscall!(fn sol_strobe128_prf(...) -> u64);
+
+define_syscall!(fn sol_strobe256_ad(...) -> u64);
+define_syscall!(fn sol_strobe256_meta_ad(...) -> u64);
+define_syscall!(fn sol_strobe256_key(...) -> u64);
+define_syscall!(fn sol_strobe256_prf(...) -> u64);


Agreed. We need benchmarks to see how long the keccak portion takes vs the other work.

lheeger-jump · 2024-01-02T21:33:47Z

proposals/0095-xof-hashing.md

+define_syscall!(fn sol_strobe128_ad(...) -> u64);
+define_syscall!(fn sol_strobe128_meta_ad(...) -> u64);
+define_syscall!(fn sol_strobe128_key(...) -> u64);
+define_syscall!(fn sol_strobe128_prf(...) -> u64);
+
+define_syscall!(fn sol_strobe256_ad(...) -> u64);
+define_syscall!(fn sol_strobe256_meta_ad(...) -> u64);
+define_syscall!(fn sol_strobe256_key(...) -> u64);


Every new syscall adds overhead and hurts performance for the VM vs an SBPF implementation for some specific app. I would recommend finding an approach similar to what @ripatel-fd has mentioned.

Concur with @lheeger-jump. The technical reasons for this is the code footprint of a compiled interpreter. Right now, all the interpreter core and all its cheap syscalls probably fit in L2 cache. The more syscalls we add, the more instruction cache pressure is increased across every program execution.

On an FPGA implementation of the VM, this is even worse as you start running into hard physical constraints.

samkim-crypto · 2024-01-09T02:35:59Z

The main application for this would be the boomerang protocol for Brave (https://arxiv.org/pdf/2401.01353.pdf), which uses Bulletproofs for the zkp. The main components that currently make the bulletproof implementation difficult is the curve operations and the support for hash function needed for Fiat-Shamir. The curve operations can use the curve25519 syscalls, but we currently do not have support for XOF hash support to implement Fiat-Shamir.

In theory, the Fiat-Shamir heuristic can be implemented using any cryptographic hash function including BLAKE3 XOF.
However, many standard zk implementations use SHAKE as it is more standard in academic works. This is the case for bulletproofs as well. Re-implementing bulletproofs/other zk implementations using alternative hash functions would require quite a bit of effort and time for application builders.

Brave expects boomerang to have about 5 million users for the boomerang protocol (@ankeleralph ). Since token22 also uses bulletproofs, this proposal can also benefit future zkp additions to token22 as well. Given that SHAKE is used in many of the newer (non-bulletproof) zk systems for Fiat-Shamir as well, I would be in favor of this SIMD given that the above points are addressed.

lheeger-jump · 2024-01-31T20:05:37Z

Since token22 also uses bulletproofs, this proposal can also benefit future zkp additions to token22 as well

I think then instead, I would propose a replacement for ZkTokenProgram as these two would be redundant

samkim-crypto · 2024-02-01T07:59:02Z

Since token22 also uses bulletproofs, this proposal can also benefit future zkp additions to token22 as well

I think then instead, I would propose a replacement for ZkTokenProgram as these two would be redundant

Yes, ZkTokenProgram uses bulletproofs plus additional smaller proofs that are outside the scope of bulletproofs, but if this proposal is implemented, then the instructions in ZkTokenProgram could just be replaced with the curve25519 and XOF hash support.

ankeleralph · 2024-02-01T10:08:32Z

Hi @lheeger-jump, @ripatel-fd, to clarify my intended usecase, we are building a protocol (more details in our research paper, or blog post) that uses Bulletproof zero-knowledge proofs, which we would like to verify within a Solana program. Therefore, we would need support for the merlin transcripts, that are based on STROBE, and STROBE is using cSHAKE.
While I agree, that the STROBE functionality can be implemented in a eBPF program, I think it would be nice if Solana would provide an implementation within the VM, as it would simplify other developers to verify the widely used BulletProof zero-knowledge proofs within a Solana program.
Regarding, cSHAKE vs BLAKE3, as @samkim-crypto already mentioned this would require developers to change the proof generation as well, which would create an additional barrier to move the proof verification on-chain.
With regards to benchmarks of the STROBE framework, I think its obvious that most performance will be required by the underlying call to the sponge function (in this case cSHAKE or Keccak), as the metadata operations will not require a lot of overhead. Let me know if you want to see some more concrete numbers for benchmarks, happy to quickly do a few to support the adoption of this proposal.

ripatel-fd · 2024-02-01T13:44:25Z

@ankeleralph Thank you for the context.

While I agree, that the STROBE functionality can be implemented in a eBPF program, I think it would be nice if Solana would provide an implementation within the VM, as it would simplify other developers to verify the widely used BulletProof zero-knowledge proofs within a Solana program.

Solana provides facilities for sharing eBPF code across different programs either by statically linking it in via Cargo or by deploying a shared program invoked via cross-program-invocation. Would you be open to trying that first?

lheeger-jump · 2024-02-05T19:46:37Z

Doing an implementation on chain as @ripatel-fd mentioned is a good way to verify the feasibility of the feature.

ankeleralph added 4 commits December 6, 2023 17:51

add initial draft of SIMD for xof hashing

7ca5027

add implementation details for cShake

472499b

add some details for Strobe

7e04e83

update some details for Strobe

ceea618

ankeleralph changed the title ~~SIMD-XXXX: Extendable Output (XOF) Hashing Support~~ SIMD-0095: Extendable Output (XOF) Hashing Support Dec 14, 2023

ankeleralph added 2 commits December 14, 2023 12:11

replace SIMD number, fix linter errors

e75f18a

fix linter errors

9a320d2

ripatel-fd reviewed Dec 29, 2023

View reviewed changes

lheeger-jump reviewed Jan 2, 2024

View reviewed changes

sakridge added the core Standard SIMD with type Core label Feb 9, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

SIMD-0095: Extendable Output (XOF) Hashing Support #95

SIMD-0095: Extendable Output (XOF) Hashing Support #95

ankeleralph commented Dec 14, 2023

ripatel-fd commented Dec 29, 2023 •

edited

ripatel-fd Dec 29, 2023

lheeger-jump Jan 2, 2024

ankeleralph Feb 1, 2024

ripatel-fd Feb 1, 2024

ripatel-fd Dec 29, 2023

ripatel-fd Dec 29, 2023

lheeger-jump Jan 2, 2024

ankeleralph Feb 1, 2024

ripatel-fd Feb 1, 2024 •

edited

lheeger-jump left a comment

lheeger-jump Jan 2, 2024

ankeleralph Feb 1, 2024

lheeger-jump Jan 2, 2024

lheeger-jump Jan 2, 2024

lheeger-jump Jan 2, 2024

lheeger-jump Jan 2, 2024

ripatel-fd Feb 1, 2024 •

edited

samkim-crypto commented Jan 9, 2024

lheeger-jump commented Jan 31, 2024

samkim-crypto commented Feb 1, 2024

ankeleralph commented Feb 1, 2024

ripatel-fd commented Feb 1, 2024

lheeger-jump commented Feb 5, 2024


		This proposal introduces three new concepts to the Solana runtime:

		- Support extendable Output Functions (XOF) hashing, based on cSHAKE


		#### Implementation Details

		The Strobe designers released an official implementation in C available at

SIMD-0095: Extendable Output (XOF) Hashing Support #95

Are you sure you want to change the base?

SIMD-0095: Extendable Output (XOF) Hashing Support #95

Conversation

ankeleralph commented Dec 14, 2023

ripatel-fd commented Dec 29, 2023 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ripatel-fd Feb 1, 2024 • edited

Choose a reason for hiding this comment

lheeger-jump left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ripatel-fd Feb 1, 2024 • edited

Choose a reason for hiding this comment

samkim-crypto commented Jan 9, 2024

lheeger-jump commented Jan 31, 2024

samkim-crypto commented Feb 1, 2024

ankeleralph commented Feb 1, 2024

ripatel-fd commented Feb 1, 2024

lheeger-jump commented Feb 5, 2024

ripatel-fd commented Dec 29, 2023 •

edited

ripatel-fd Feb 1, 2024 •

edited

ripatel-fd Feb 1, 2024 •

edited