Add no-std support #100

elichai · 2019-02-18T12:38:57Z

Hi,
This make this library support no-std.
first I replaced everything from std to get it from core.
then I had 2 things that I had to change and would love you input:

the C types from std::os::raw, I see few options. one which I did is just declare it manually which I don't really love, another is to make a feature if it will be received from std or libc.
Replacing the Vec in the DER serialization. I talked with Maxwell and he told me that the signature can be anywhere between 8 and 72 bytes, so the only way I thought I can achieve this without breaking the API is using the ArrayVec crate which initializes the Vector on the stack,
A different approach will be accepting &mut [u8; 72] and returning a slice to the correct signature that will be a part of that mutable.

elichai · 2019-02-18T13:26:40Z

Ok, I don't like the types.rs approach.
I would prefer to have a feature to switch between libc and libstd, but it's up to you.

elichai · 2019-02-18T14:49:09Z

So this might influence your thoughts on types.rs vs libc feature,
core::ffi::c_void was introduced only in rust 1.30 rust-lang/rust#53910
so to support that we either need to add that enum to types.rs too or just declare c_uchar as c_void (as I did)

apoelstra · 2019-02-26T14:12:32Z

I also don't like types.rs; I don't see how it can be correct across multiple architectures. What would a feature-flag look like?
I also really don't want to add any more dependencies to this library, including arrayvec. We used arrayvec in the past and removed it because it was an extra dependency. I'd prefer we return a ([u8; 74], usize) or use an outptr.
Context creation still uses malloc in the C function secp256k1_context_create. Is this okay for nostd users or do we need to change to use the preallocation API?

elichai · 2019-02-26T15:08:31Z

I'll try to think of a better solution, maybe feature based and let you know what I come up with.
If you don't care about breaking the API I think the better solution would be to accept a &mut [u8; 74] into the function and return a &[u8] of the exact slice that contains the DER which will be part of the original mut slice,
this way all the allocations are on the user behalf and he receives back exactly the right thing without needing to slice it afterwards, what do you think?
I think malloc is support everywhere liballoc is supported. so in my case for example, I'm putting this in SGX so I need no-std but I can allow mallocs so it really depends on the use case. some embedded devices even have no support for liballoc and there it will be a problem(AFAIK).
After some testing there's a problem with the fact that the C code usesfprintf,
after I commented all the uses out I managed to create keys and sign a message successfully.

elichai · 2019-02-26T15:25:03Z

Ok, so about 1, I see a few possibilities.

get libc back with default-features=false
forget about these types or use the types.rs because there isn't different types on different architecture with the std::os::raw::* it's just that on some architectures these types are missing.
Just copy the whole std::os::raw file https://doc.rust-lang.org/src/std/os/raw/mod.rs.html#11-108

apoelstra · 2019-02-26T18:06:22Z

It's really surprising to me that there aren't different types for different architectures.

Regarding accepting an &mut [u8; 74] and returning an exact slice, sure, that sounds fine to me. But if std is enabled I'd like to keep the existing API, it's much more ergonomic :).

Regarding the printf calls, is the issue with compilation or can it be fixed by providing different error callbacks at runtime?

elichai · 2019-02-26T18:12:47Z

It's in runtime.
And I guess it's possible to provide two versions of the same function, although that will be hard to see in docs.

Maybe adding a new serliaze_mut or serialize_no_vec or something similiar and the regular one won't be available without std

elichai · 2019-02-26T21:57:05Z

Wait. Not on compilation, it's in runtime. Sorry my mistake.

apoelstra · 2019-02-27T18:18:11Z

Yeah, good idea naming the alternate serialize function something different.

And if there are runtime issues this is fine; during context creation we can use the FFI functions secp256k1_context_set_illegal_callback and secp256k1_context_set_error_callback to replace the offending printf-calling functions with different ones. This way we won't need to patch the C code.

elichai · 2019-02-27T18:52:33Z

@apoelstra Thanks for the feedback.
I did the first one, I can give the second one a try and see if I get it right.

BTW how does this work? https://github.com/rust-bitcoin/rust-secp256k1/blob/master/src/lib.rs#L338
does the C code do the allocations?
because AFAIK with_capacity() doesn't allocate anything, it just tells it that the first time there'll be an allocation it will be with that capacity

apoelstra · 2019-02-27T20:33:08Z

No, the C code does not do allocations. My understanding was that with_capacity would do the allocation -- I don't see how it could behave otherwise. Where in the Vec structure would the "amount to allocate on next allocation" be stored?

elichai · 2019-02-27T20:37:03Z

@apoelstra It's stored in the RawVec struct: https://doc.rust-lang.org/1.26.0/src/alloc/raw_vec.rs.html#52
that it's initialized here: https://doc.rust-lang.org/src/alloc/vec.rs.html#366-371

You can also see the docs about with_capacity(): https://doc.rust-lang.org/std/vec/struct.Vec.html#method.with_capacity

apoelstra · 2019-02-27T20:39:29Z

@elichai The cap field stores the amount of currently-allocated memory. The with_capacity constructor definitely allocates memory and the docs don't suggest otherwise.

apoelstra · 2019-02-27T20:40:33Z

If you're using "allocate" to refer to setting the vector's length, we do this manually after the FFI call by calling the unsafe set_len method on the vector.

elichai · 2019-02-27T20:51:17Z

@apoelstra Ok, I were wrong and you were right.
the docs were a bit confusing for me because it's written a bit through negation If capacity is 0, the vector will not allocate meaning that if it's not 0 it will allocate

I'm really Sorry for the confusion

elichai · 2019-03-03T08:56:08Z

So my problems appears neither in compilation nor in runtime.
To run a code in SGX I need to later link it with some other C files and it fails in linkage because it tries to link with stdio:

$ g++ enclave/Enclave_t.o -o enclave/enclave.so -m64 -O2 -Wl,--no-undefined -nostdlib -nodefaultlibs -nostartfiles -L/opt/intel/sgxsdk/lib64 -Wl,--whole-archive -lsgx_trts -lsgx_tservice -Wl,--no-whole-archive -Wl,--start-group -lsgx_tstdc -lsgx_tcxx -lsgx_tcrypto -L./lib -lcompiler-rt-patch -lenclave -Wl,--end-group -Wl,-Bstatic -Wl,-Bsymbolic -Wl,--no-undefined -Wl,-pie,-eenclave_entry -Wl,--export-dynamic -Wl,--defsym,__ImageBase=0 -Wl,--gc-sections -Wl,--version-script=enclave/Enclave.lds
./lib/libenclave.a(secp256k1.o): In function `fprintf':
/usr/include/x86_64-linux-gnu/bits/stdio2.h:97: undefined reference to `stderr'
/usr/include/x86_64-linux-gnu/bits/stdio2.h:97: undefined reference to `__fprintf_chk'
/usr/include/x86_64-linux-gnu/bits/stdio2.h:97: undefined reference to `stderr'
/usr/include/x86_64-linux-gnu/bits/stdio2.h:97: undefined reference to `__fprintf_chk'
/usr/include/x86_64-linux-gnu/bits/stdio2.h:97: undefined reference to `stderr'
/usr/include/x86_64-linux-gnu/bits/stdio2.h:97: undefined reference to `__fprintf_chk'
collect2: error: ld returned 1 exit status

elichai · 2019-03-03T12:32:55Z

@apoelstra how do you want to proceed with this?
Should I open an Issue on https://github.com/bitcoin-core/secp256k1 to see what sipa/gmaxwell think?

real-or-random · 2019-03-03T19:07:01Z

@elichai I'll create a simple PR to secp256k1 tomorrow that should make it possible to build it without stdio.h (I'm runing into similar problems with a different project).

elichai · 2019-03-03T19:10:59Z

@real-or-random Thanks! tag me in it so i'll follow when it gets merged :)

elichai · 2019-03-11T22:15:01Z

So I think it's best to put this on hold until these two are figured out:
bitcoin-core/secp256k1#566
bitcoin-core/secp256k1#595

elichai · 2019-04-11T09:04:43Z

I just thought of another interesting way to make the serialize_der() no-std,
We can make a struct that has a 72 byte array and a length,
then that struct can implement AsRef / Have a slice() function that returns &arr[..len],
then the user is invisible to what happens but he get's a struct that can be easily serializable into his signature.

apoelstra · 2019-04-11T16:39:48Z

Oh! Yeah, I like this idea. We could call it a SerSignature and give it slicing ability and the ability to go to/from a regular Signature.

elichai · 2019-04-11T17:42:33Z

Should that be as part of serialize_der_no_alloc() or the regular serialize_der()?
(I think it's a good enough solution but it will break the api.)

apoelstra · 2019-04-11T17:50:13Z

I think we should go ahead and break the API and change serialize_der. I hate that this function returns a Vec anyway.

elichai · 2019-04-11T18:18:04Z

BTW, is it using this: https://github.com/rust-bitcoin/rust-secp256k1/blob/master/src/ffi.rs#L419 or the one from the bindings? (https://github.com/rust-bitcoin/rust-secp256k1/blob/master/src/ffi.rs#L198)

If the rust one, what do you think about a re-implementation in safe rust?

apoelstra · 2019-04-11T18:26:17Z

We are using the FFI one. The other is something Matt wrote for fuzzing; I doubt that it actually serializes a signature (though it might); these implementations are there to provide low-complexity mock implementations of the crypto, to prevent the fuzzer getting lost trying to break cryptography.

I like the idea of a safe reimplementation, though it does create a layer violation: the types exposed by the underlying secp256k1 are considered opaque and subject to change without warning. On the other hand, since rust-secp bundles a fixed version of secp256k1, it won't change without us taking active steps..

apoelstra · 2019-04-12T19:03:50Z

Cool. I'm happy with the result. Can you rebase/squash so that we don't go through the ArrayVec attempt in the git history?

…eature

elichai · 2019-04-14T09:11:09Z

Done. (the changes from before the rebasing to after are here if you're interested: https://github.com/rust-bitcoin/rust-secp256k1/compare/0e0d8df84b001da2ba42c07205ad39e739c532ad..41af612ae277962b4b1747c5b58704bd40e1819d )

real-or-random

looks good to me overall :)

src/lib.rs

jonasnick · 2019-04-17T21:13:46Z

src/types.rs

+pub type c_int = i32;
+pub type c_uchar = u8;
+pub type c_uint = u32;
+pub type c_void = c_uchar;


Is it really correct to just use u8 for c_void? That comment makes me think otherwise https://doc.rust-lang.org/src/core/ffi.rs.html#21 (though I don't understand it).

In one of the comments here I think @apoelstra pointed out that *void is almost equivalent to *char, it's hard to look for resolved comments on the phone.

But wouldn't it be cleaner to copy the enum implementation with the two variants? It's just three lines of code, so it wouldn't hurt I guess.

Agree with real-or-random. I don't mind type-punning *c_void for *c_uchar but the base types c_void and c_uchar are very different.

So I see two options.
one is to remove the c_void type and replace the pointers to *c_uchar, another is what @real-or-random suggested and copy the enum implementation.

I'm really no expert in C and don't know the implications if someone later on misuse c_void with c_uchar so I don't have any opinion on this, I think it should be as foolproof as possible.

I think we should copy the enum implementation and link to the libc code, and mention that in Rust 1.30 we will be able to just use core::ffi.

Ok, FYI c_void != void.
but *const c_void == const void* and *mut c_void == void*

src/types.rs

apoelstra

ACK. Thank you for your patience!!

cc @real-or-random @jonasnick can you also approve since you've been guiding this PR?

src/lib.rs

real-or-random

ACK

See rust-bitcoin/rust-secp256k1#100 to review commit that caused failure

@elichai

e49f799 Add missing #(un)defines to base-config.h (Tim Ruffing) 77defd2 Add secp256k1_ prefix to default callback functions (Tim Ruffing) 908bdce Include stdio.h and stdlib.h explicitly in secp256k1.c (Tim Ruffing) 5db782e Allow usage of external default callbacks (Tim Ruffing) 6095a86 Replace CHECKs for no_precomp ctx by ARG_CHECKs without a return (Tim Ruffing) Pull request description: This is intended for environments without implementations for `abort()`, `fprintf()`, and `stderr`. e.g., embedded systems. Those can provide their own implementations of `default_illegal_callback_fn` and `default_error_callback_fn` at compile time. If you want to use your own default callback, things will be somewhat inconsistent unfortunately: We cannot make the callback data `extern` too, because then the initialization lists for `default_illegal_callback` won't contain only constants. (`const` variables are not compile-time constants). So you cannot take callback data in your own default callback function. As a more drastic/breaking alternative I suggest to remove the callback data entirely. I don't think it's a big loss and I would be surprised if anyone uses it. Additionally, we could even remove the possibility to set the callback function at runtime after this PR. This will simplify things a lot, and again I don't think it's a big loss. Note that `abort()`, `fprintf()`, and `stderr` are also used in `CHECK`, which is still used in production code if we rely on gmp for scalar and field inversions (e.g., https://github.com/bitcoin-core/secp256k1/blob/master/src/scalar_impl.h#L240). This is not an issue for embedded system which probably don't want to use gmp anyway, but it is probably an issue for the reasons explained in #566 (comment). (related downstream: rust-bitcoin/rust-secp256k1#100 @elichai) ACKs for commit e49f79: Tree-SHA512: 4dec0821eef4156cbe162bd8cdf0531c1fae8c98cd9db8438170ff1aa0e59b199739eeab293695bb582246812bea5309959f02f1fb74bb57872da54ebc52313f

elichai force-pushed the master branch from babc0f6 to d0e7a9c Compare February 27, 2019 19:45

real-or-random mentioned this pull request Mar 4, 2019

Allow to use external default callbacks bitcoin-core/secp256k1#595

Merged

TheBlueMatt mentioned this pull request Mar 22, 2019

Optional secp256k1 dependency with feature rust-bitcoin/rust-bitcoin#240

Closed

elichai force-pushed the master branch from d0e7a9c to 9427070 Compare April 11, 2019 17:55

elichai force-pushed the master branch from 0e0d8df to c673736 Compare April 14, 2019 09:06

elichai added 4 commits April 14, 2019 12:09

Replaced std with core, and made std::error::Error optional through f…

11ce866

…eature

Replaced the std::os::raw c types with types.rs

ab8066e

Updated travis.yml to test no-std too

6871567

Added backwards support without the crate keyword

e98975a

elichai force-pushed the master branch from c673736 to 41af612 Compare April 14, 2019 09:09

real-or-random reviewed Apr 15, 2019

View reviewed changes

src/lib.rs Show resolved Hide resolved

src/lib.rs Show resolved Hide resolved

src/lib.rs Outdated Show resolved Hide resolved

src/lib.rs Outdated Show resolved Hide resolved

src/lib.rs Show resolved Hide resolved

jonasnick reviewed Apr 17, 2019

View reviewed changes

elichai force-pushed the master branch 2 times, most recently from 5f04fcb to f42cfcb Compare April 24, 2019 12:35

jonasnick reviewed Apr 30, 2019

View reviewed changes

src/types.rs Outdated Show resolved Hide resolved

Added Error::description copied c_void impl from core::ffi::c_void

312b9a5

elichai force-pushed the master branch from f42cfcb to becb35a Compare April 30, 2019 19:08

apoelstra approved these changes May 1, 2019

View reviewed changes

real-or-random self-requested a review May 1, 2019 14:07

real-or-random reviewed May 2, 2019

View reviewed changes

src/lib.rs Show resolved Hide resolved

Added a new struct for the DER serialized signature

dfcf74f

elichai force-pushed the master branch from becb35a to dfcf74f Compare May 10, 2019 21:11

real-or-random approved these changes May 10, 2019

View reviewed changes

real-or-random merged commit ab4320f into rust-bitcoin:master May 10, 2019

vhnatyk mentioned this pull request May 14, 2019

secp256k1 no_std/wasm friendly ZenGo-X/emerald-city#4

Open

mattaudesse added a commit to AlacrisIO/mkb that referenced this pull request May 14, 2019

Pin secp256k1 to 90faf434 due to type mismatch during compilation

0e34b0d

See rust-bitcoin/rust-secp256k1#100 to review commit that caused failure

mattaudesse mentioned this pull request May 14, 2019

Pin secp256k1 to 90faf434 due to type mismatch during compilation AlacrisIO/mkb#2

Merged

elichai mentioned this pull request Apr 19, 2020

Cut verification/signing/proofs etc. time by half for 64bit machines mimblewimble/rust-secp256k1-zkp#71

Open

elichai mentioned this pull request Sep 7, 2021

Replace secp256k1 with libsecp256k1 pure rust implementation ZenGo-X/curv#135

Closed

apoelstra mentioned this pull request Jun 21, 2022

impl Default for SerializedSignature doesn't seem to make sense #454

Closed

apoelstra mentioned this pull request Mar 16, 2024

secp256k1-sys/src/lib.rs uses a wildcard import #681

Closed

Add no-std support #100

Add no-std support #100

Conversation

elichai commented Feb 18, 2019

elichai commented Feb 18, 2019 • edited

elichai commented Feb 18, 2019

apoelstra commented Feb 26, 2019 • edited

elichai commented Feb 26, 2019

elichai commented Feb 26, 2019 • edited

apoelstra commented Feb 26, 2019

elichai commented Feb 26, 2019 • edited

elichai commented Feb 26, 2019

apoelstra commented Feb 27, 2019

elichai commented Feb 27, 2019 • edited

apoelstra commented Feb 27, 2019

elichai commented Feb 27, 2019

apoelstra commented Feb 27, 2019

apoelstra commented Feb 27, 2019

elichai commented Feb 27, 2019 • edited

elichai commented Mar 3, 2019 • edited

elichai commented Mar 3, 2019

real-or-random commented Mar 3, 2019

elichai commented Mar 3, 2019 • edited

elichai commented Mar 11, 2019

elichai commented Apr 11, 2019

apoelstra commented Apr 11, 2019

elichai commented Apr 11, 2019 • edited

apoelstra commented Apr 11, 2019

elichai commented Apr 11, 2019

apoelstra commented Apr 11, 2019

apoelstra commented Apr 12, 2019

elichai commented Apr 14, 2019

real-or-random left a comment

Choose a reason for hiding this comment

jonasnick Apr 17, 2019

Choose a reason for hiding this comment

elichai Apr 17, 2019

Choose a reason for hiding this comment

real-or-random Apr 17, 2019

Choose a reason for hiding this comment

apoelstra Apr 20, 2019

Choose a reason for hiding this comment

elichai Apr 21, 2019

Choose a reason for hiding this comment

apoelstra Apr 23, 2019

Choose a reason for hiding this comment

elichai Apr 24, 2019 • edited

Choose a reason for hiding this comment

apoelstra left a comment

Choose a reason for hiding this comment

real-or-random left a comment

Choose a reason for hiding this comment

elichai commented Feb 18, 2019 •

edited

apoelstra commented Feb 26, 2019 •

edited

elichai commented Feb 26, 2019 •

edited

elichai commented Feb 26, 2019 •

edited

elichai commented Feb 27, 2019 •

edited

elichai commented Feb 27, 2019 •

edited

elichai commented Mar 3, 2019 •

edited

elichai commented Mar 3, 2019 •

edited

elichai commented Apr 11, 2019 •

edited

elichai Apr 24, 2019 •

edited