[WIP] Subxt-Core Crate #1411

tadeohepperle · 2024-02-02T16:00:48Z

relates to #1394

This PR does a few things:

make subxt-signer no-std compatible
create a new subxt-core crate that is no-std compatible by default and offers a subset of subxt's functionality.
use this subxt-core crate in subxt-signer and in subxt.

What is not working yet:

The critical_section feature of the once_cell crate used in subxt-signer results in linker errors when run in no-std.
Subxt compiles, but subxt examples are not all adjusted yet
Interface and imports need to be discucced and made nicer, some messiness has to be removed
Docs, especially book needs to be adjusted

…/subxt-core-2

tadeohepperle · 2024-02-02T16:03:11Z

signer/src/ecdsa.rs

+    fn deref(&self) -> &Self::Target {
+        static ONCE: once_cell::sync::OnceCell<Secp256k1<All>> = once_cell::sync::OnceCell::new();
+        ONCE.get_or_init(|| Secp256k1::new())
+    }


When in an std context we could also add a line here to seed the Secp256k1 by some Rng.
Will not be available in WASM and no-std though.

I'd copy what Substrate does here to be on the safe side, which is to do:

#[cfg(feature = "std")] let context = SECP256K1; #[cfg(not(feature = "std"))] let context = Secp256k1::signing_only();

In the sign/from_seed fns to select which context to use ie https://github.com/paritytech/polkadot-sdk/blob/d54412ce8bef113c7a515b3a08851e7d77553df5/substrate/primitives/core/src/ecdsa.rs#L390 and https://github.com/paritytech/polkadot-sdk/blob/d54412ce8bef113c7a515b3a08851e7d77553df5/substrate/primitives/core/src/ecdsa.rs#L460

Since only two places this context is used anyway, I think that's fine :)

tadeohepperle · 2024-02-07T14:03:31Z

I am having trouble getting no_std testing working properly.
I setup a crate testing/no-std-tests that pulls in subxt-metadata, subxt-signer and subxt-core. You can try to build it with either of these:

cargo run 
cargo build --target thumbv7em-none-eabi 
cargo build --target aarch64-unknown-none

With cargo run (the native default target), we get linker errors. The curious thing is that the linker errors on my linux machine are different from the erros James is getting on a macos target.

With the thumbv7em-none-eabi and aarch64-unknown-none targets the problem is, that parts of the rust core prelude (e.g. Some and None variants, Iterator trait, etc. are not found by the compiler).

jsdw · 2024-02-08T17:05:27Z

signer/src/ecdsa.rs

@@ -91,8 +103,9 @@ impl Keypair {
    /// keypair.sign(b"Hello world!");
    /// ```
    pub fn from_phrase(mnemonic: &bip39::Mnemonic, password: Option<&str>) -> Result<Self, Error> {
-        let big_seed = seed_from_entropy(&mnemonic.to_entropy(), password.unwrap_or(""))
-            .ok_or(Error::InvalidSeed)?;
+        let (arr, len) = mnemonic.to_entropy_array();


Nice catch to avoid an allocation anyway :)

jsdw · 2024-02-08T17:14:06Z

signer/src/ecdsa.rs

@@ -192,31 +206,35 @@ pub fn verify<M: AsRef<[u8]>>(sig: &Signature, message: M, pubkey: &PublicKey) -
    };
    let message_hash = sp_core_hashing::blake2_256(message.as_ref());
    let wrapped = Message::from_digest_slice(&message_hash).expect("Message is 32 bytes; qed");
-    signature.verify(&wrapped, &public).is_ok()
+    GlobalSecp256K1Context


Ah and here you can use https://docs.rs/secp256k1/latest/secp256k1/struct.Secp256k1.html#method.verification_only when in no-std rather than signing_only, I think!

jsdw · 2024-02-08T17:18:19Z

signer/Cargo.toml


 # The getrandom package is used via schnorrkel. We need to enable the JS
 # feature on it if compiling for the web.
-web = ["getrandom/js", "subxt?/web"]
-native = ["subxt?/native"]
+web = ["getrandom/js"]


Nice to be able to ditch the native feature flag!

Does everything work when "web" is enabled and "std" isn't, I wonder?

Oh that is a good question, I need to try that.

jsdw · 2024-02-08T17:36:29Z

signer/Cargo.toml

 secrecy = { workspace = true }

+# We only pull this in because `std::sync::OnceLock` is not available in no-std contexts.
+once_cell = { workspace = true, default-features = false, features = ["alloc", "critical-section"] }


critical-section uses this crate to provide the only-one-thread-at-a-time behaviour:

https://crates.io/crates/critical-section

This crate requires that you enable different features to support different no-std targets (perhaps this explains why you hit a compile error in no-std).

I'm not sure whether this is a good idea to use or not.

If we do use it, we should def enable critical-section/std when std feature is enabled though, so that it uses a standard mutex instead of anything strange in std cases. We'd also need to document to users to go read the critical-section docs (like once_cell does) because they would need to depend on critical-section themselves and add the relevant features for the targets they are trying to support.

But, all of this is just to enable the provided dev accounts at the end of the day, so perhaps the best route to take is revert to using std::sync::OnceLock, remove once_cell dep and just don't globally instantiate the dev accounts when in no-std mode. So we could alter the once_static macro thing to just use the OnceLock global when in std and recreate the account (run the $expr) every time when in no-std? (and then document on the macro that in no-std we run the expr every time)

jsdw · 2024-02-14T09:09:07Z

subxt/src/lib.rs

+// pub use subxt_core::config;
+// pub use subxt_core::config::{Config, PolkadotConfig, SubstrateConfig};
+// pub use subxt_core::dynamic;
+pub use subxt_core::*;


I'd prefer to be really explicit here so that it's obvious exactly what we're exporting :)

jsdw · 2024-02-14T09:13:30Z

core/src/tx/mod.rs

@@ -168,7 +170,8 @@ impl<CallData: EncodeAsFields> TxPayload for Payload<CallData> {
            .map(|f| scale_encode::Field::new(f.ty.id, f.name.as_deref()));

        self.call_data
-            .encode_as_fields_to(&mut fields, metadata.types(), out)?;
+            .encode_as_fields_to(&mut fields, metadata.types(), out)
+            .expect("The fields are valid types from the metadata, qed;");


There are various other reasons why the call data might not properly encode though I think; better to keep the ??

jsdw · 2024-02-14T09:16:07Z

core/src/tx/mod.rs

I guess in follow up PRs we can add more functionality; there's a bunch of TX stuff we could have in core too I think; most of the stuff around signing and encoding calls ready to send (just not the "online" bits eg actually sending them)

jsdw · 2024-02-14T09:19:37Z

core/README.md

@@ -0,0 +1,3 @@
+# Subxt-Core
+
+This library provides core functionality using in `subxt` and `subxt-signer`. It should be no-std compatible.


I think for now we should add a warning here and in the lib docs that this is not stable may still change significantly between releases, so better to rely on subxt where possible: there's a lot to look at and think about with this new crate, but at the same time it'd be good to feel like we can release it now and iterate on it without worrying too much about breaking things :)

core/src/client/mod.rs

jsdw · 2024-02-14T09:37:07Z

Interface and imports need to be discucced and made nicer, some messiness has to be removed

I'd be curious what you are currently thinking is messy and could be improved; perhaps we should have a walk through of this stuff at some point to get a better feeling for it all as there is a lot to think about in this PR :)

…/subxt-core-2

…ory. (#1431) * delete substrate and polkadot binaries * use nodes should move instead of copy and remove the archive

* add codgen mod to the module tree * port codegen tests to scale-typegen * remove test flag

* no-std tests and porting of subxt-metadata * update pipeline * fix generate custom metadata test * fix cargo run command * adjust pipeline * remove prelude from subxt-metadata * revert autoformatting of Cargo.toml * remove alloc::format! again, still causes linker errors * add no-std-build for thumbv7em-none-eabi target * remove std feature flag * remove libc and add small readme with test instructions * change ci for nightly no std

tadeohepperle · 2024-03-07T17:38:22Z

I started a new branch, because it was easier than trying to solve all conflicts, correctly. To be continued here: #1466

tadeohepperle added 27 commits January 26, 2024 15:34

no-std tests and porting of subxt-metadata

a262766

Merge branch 'master' into tadeohepperle/subxt-metadata-no-std

8aa4bec

update pipeline

55cfb23

fix generate custom metadata test

1b4e371

fix cargo run command

d679e95

adjust pipeline

7954815

Merge branch 'master' into tadeohepperle/subxt-metadata-no-std

fbee16d

init of subxt-core crate

a3ec38f

subxt-core compiles to no std again

4512ad7

subxt compiles with new dependency features

2e66b48

subxt: use utils from core

7b2cfdc

start porting dynamic value stuff

aed00e5

add storage address and custom error type to core crate

97aad71

remove prelude from subxt-metadata

7aa6989

Merge branch 'master' into tadeohepperle/subxt-metadata-no-std

addc4c8

revert autoformatting of Cargo.toml

87200b4

remove alloc::format! again, still causes linker errors

57743c3

Merge branch 'master' into tadeohepperle/subxt-metadata-no-std

408f81e

Merge branch 'tadeohepperle/subxt-metadata-no-std' into tadeohepperle…

6236c49

…/subxt-core-2

remove prelude from core, use alloc directly

e427529

constants and storage entries

e855aac

custom values and runtime apis

930f2c4

port subxt-signer to no-std (such a pain)

14b21ab

core integration into subxt, except for examples

4e2d3fd

start fixing examples and cleaning up interface

fce85c4

discover linker errors for subxt-signer critical section of once_cell

e5ee971

fmt for whole workspace

bc6cd9b

tadeohepperle commented Feb 2, 2024

View reviewed changes

tadeohepperle force-pushed the tadeohepperle/subxt-core-2 branch from f65d834 to bc6cd9b Compare February 2, 2024 16:04

add no-std-build for thumbv7em-none-eabi target

f41f090

jsdw reviewed Feb 8, 2024

View reviewed changes

jsdw reviewed Feb 14, 2024

View reviewed changes

core/src/client/mod.rs Show resolved Hide resolved

tadeohepperle and others added 17 commits February 14, 2024 10:42

remove std feature flag

ce17d82

Merge branch 'master' into tadeohepperle/subxt-metadata-no-std

5d6e0d5

Merge branch 'master' into tadeohepperle/subxt-metadata-no-std

cb0d35d

adjust subxt signer static once locks

64987fe

remove libc and add small readme with test instructions

d447f7c

Merge branch 'tadeohepperle/subxt-metadata-no-std' into tadeohepperle…

a0a79c4

…/subxt-core-2

remove unicode normalization and std references

cc33870

subxt signer linker errors but core compiles

b86b1ea

[Fix] Auto-generated PR: delete polkadot binaries from working direct…

1fe8ec3

…ory. (#1431) * delete substrate and polkadot binaries * use nodes should move instead of copy and remove the archive

Add missing codegen mod in integration tests (#1418)

db00212

* add codgen mod to the module tree * port codegen tests to scale-typegen * remove test flag

fix examples with core crate

ac3c7d7

formatting

e527687

Merge branch 'master' into tadeohepperle/subxt-core-2

eeca8f8

fix unused deps

6147ac9

fix doc tests and docs

fbf2cc9

fmt

aae7636

tadeohepperle closed this Mar 7, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[WIP] Subxt-Core Crate #1411

[WIP] Subxt-Core Crate #1411

tadeohepperle commented Feb 2, 2024

tadeohepperle Feb 2, 2024

jsdw Feb 8, 2024

tadeohepperle commented Feb 7, 2024 •

edited

Loading

jsdw Feb 8, 2024

jsdw Feb 8, 2024

jsdw Feb 8, 2024 •

edited

Loading

tadeohepperle Feb 8, 2024

jsdw Feb 8, 2024

jsdw Feb 14, 2024

jsdw Feb 14, 2024 •

edited

Loading

jsdw Feb 14, 2024

jsdw Feb 14, 2024

jsdw commented Feb 14, 2024

tadeohepperle commented Mar 7, 2024

		@@ -0,0 +1,3 @@
		# Subxt-Core

		This library provides core functionality using in `subxt` and `subxt-signer`. It should be no-std compatible.

[WIP] Subxt-Core Crate #1411

[WIP] Subxt-Core Crate #1411

Conversation

tadeohepperle commented Feb 2, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

tadeohepperle commented Feb 7, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jsdw Feb 8, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jsdw Feb 14, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jsdw commented Feb 14, 2024

tadeohepperle commented Mar 7, 2024

tadeohepperle commented Feb 7, 2024 •

edited

Loading

jsdw Feb 8, 2024 •

edited

Loading

jsdw Feb 14, 2024 •

edited

Loading