pb-rust: JSON schema compilation source #118

jleightcap · 2023-08-02T14:44:57Z

Summary

Note: blocked by #121.
Compiles Rust bindings from JSON schemas.

Release Note

Documentation

Closes #115.

includes the modules structure definitions needed have full package prefixes, like ```rust use sigstore_protobuf_specs::dev::sigstore::verification::v1::Input ``` Signed-off-by: Jack Leightcap <jack.leightcap@trailofbits.com>

jleightcap · 2023-08-02T16:55:31Z

Note: I believe JIT compiling the JSON schema locally to the Rust crate requires committing the schemas a second time: the packaging requires all of the source data to be self-contained in gen/pb-rust.

I've added relevant rules to the Makefile to sync the two, let me know if that automates this enough. Or if I'm totally off-base 🙃

jleightcap · 2023-08-02T17:05:08Z

This PR does not provide complete Rust bindings, instead binding just enough to unblock parent sigstore-rs issue (sigstore/sigstore-rs#280). For future work, see note:

protobuf-specs/gen/pb-rust/src/lib.rs

Lines 3 to 25 in e8f2d78

    
           /// NOTE(jleightcap): a method to include all JSON schemas is not immediately obvious to me: 
        
           /// 
        
           /// - `schemafy!` is a direct 1:1 compilation to Rust structures of each definition in "defintions", 
        
           /// - each schema is 'standalone': including definitions of all it's dependencies, 
        
           /// as a result, the Rust generated structures have name collisions if included in the same module scope. 
        
           /// 
        
           /// prefixing works, 
        
           /// 
        
           /// ```rust 
        
           /// mod CertificateIdentity { 
        
           ///   schemafy::schemafy!("schemas/CertificateIdentity.shema.json") 
        
           /// } 
        
           /// mod CertificateIdentities { 
        
           ///   schemafy::schemafy!("schemas/CertificateIdentities.schema.json") 
        
           /// } 
        
           /// ``` 
        
           /// but is clunky to use and manual to generate. 
        
           /// 
        
           /// a more general approach is to 'flatten' each JSON Schema, 
        
           /// unioning all of the "defintions" fields. 
        
           /// 
        
           /// Since standardized bundles is the singular motiviation for these Rust bindings, 
        
           /// for now we're punting this issue.

(per conversation with @woodruffw)

woodruffw · 2023-08-02T17:46:01Z

I've added relevant rules to the Makefile to sync the two, let me know if that automates this enough. Or if I'm totally off-base 🙃

I'm fine with that for now -- ideally we'd only have one copy here, but I'd rather we do this the "right" way with cargo and in-tree resources, which requires two copies.

jleightcap · 2023-08-02T19:43:43Z

there's a chance to investigate #115 as this crystalizes.

gen/pb-rust/tests/unit.rs

jleightcap · 2023-08-02T21:04:25Z

There's currently an annoying naming convention discrepancy when attempting to consume sigstore-python artifacts directly; see #67 (comment).

For the moment the integration tests uses a sigstore-python bundle but manually edited to use snake_case.

jleightcap · 2023-08-02T22:12:51Z

There's currently an annoying naming convention discrepancy when attempting to consume sigstore-python artifacts directly; see #67 (comment).

For the moment the integration tests uses a sigstore-python bundle but manually edited to use snake_case.

I must have just had some vestigital state around, doing a clean rebuild forcing the use of camelCase with both the schema and artifact seems to have resolved compatibility issues with other sigstore artifacts.

woodruffw · 2023-08-03T18:46:20Z

gen/pb-rust/src/lib.rs

+use serde::{Deserialize, Serialize};
+schemafy::schemafy!("schemas/Bundle.schema.json");


If I'm understanding correctly, this ends up in the current module namespace and not a nested one, right? Fine if so, just making sure.

Correct, see the use in the hand-written tests for usage.

https://github.com/sigstore/protobuf-specs/pull/118/files#diff-3e6cc2e74ad94dd29d10a8e8df25be38bd688d5e5f0105720695335005100ec6R1-R5

The names are compiled from the dev.sigstore.{...} strings into VeryLongCamelCaseStrings -- annoying, but with an as I don't think it's too prohibitive for use in sigstore-rs.

Signed-off-by: Jack Leightcap <jack.leightcap@trailofbits.com> pb-rust: schemafy construction Signed-off-by: Jack Leightcap <jack.leightcap@trailofbits.com>

adapted from sigstore@b7d905d Signed-off-by: Jack Leightcap <jack.leightcap@trailofbits.com>

Signed-off-by: Jack Leightcap <jack.leightcap@trailofbits.com>

woodruffw · 2023-08-04T17:03:23Z

gen/pb-rust/schemas/MessageSignature.schema.json

+                    "enum": [
+                        "HASH_ALGORITHM_UNSPECIFIED",
+                        0,
+                        "SHA2_256",
+                        1
+                    ],
+                    "oneOf": [
+                        {
+                            "type": "string"
+                        },
+                        {
+                            "type": "integer"
+                        }


Are we missing enums_as_strings? Or is this schema out-of-date possibly?

Silly bug. (f21f9ef#diff-fe6f382153307cf3287026377c362727d9ec012c8681090c32a1d93255a652d7R9)

Added the updated schemas to this PR.

Signed-off-by: Jack Leightcap <jack.leightcap@trailofbits.com>

woodruffw · 2023-08-04T21:20:00Z

gen/pb-rust/tests/integration.rs

+            }),
+            message_signature: Some(MessageSignature {
+                message_digest: Some(HashOutput{
+                    algorithm: Some(HashAlgorithm::Variant0(String::from("SHA2_256"))),


Is this test running? I'd expect it to fail now.

Yeah, wasn't caught -- root issue here is not running the test suite in CI. (1b22bff)

Noting:

$ cargo test (cut) Doc-tests sigstore_protobuf_specs running 3 tests test src/lib.rs - Serialize (line 9) ... FAILED test src/lib.rs - Deserialize (line 9) ... FAILED test src/lib.rs - (line 9) ... FAILED

The macro expansion seems to result in some failing doc tests? I can look into it if that's important, for the moment I have CI just run the test suite with cargo test --tests.

Now passing test suite: https://github.com/trail-of-forks/protobuf-specs/actions/runs/5789830447/job/15691699889

Signed-off-by: Jack Leightcap <jack.leightcap@trailofbits.com>

.github/workflows/rust-build.yml

woodruffw · 2023-08-07T21:25:43Z

LGTM. I'd like it if the doctests also pass, but I'll look at that in a follow-up.

codegen: init schemafy

df43a3b

includes the modules structure definitions needed have full package prefixes, like ```rust use sigstore_protobuf_specs::dev::sigstore::verification::v1::Input ``` Signed-off-by: Jack Leightcap <jack.leightcap@trailofbits.com>

jleightcap mentioned this pull request Aug 2, 2023

pb-rust: Serde via prost + pbjson #95

Closed

jleightcap force-pushed the jl/pb-rust-schemafy branch from 0421499 to e8f2d78 Compare August 2, 2023 16:56

jleightcap force-pushed the jl/pb-rust-schemafy branch 2 times, most recently from d6bcc8f to 00831d6 Compare August 2, 2023 19:30

jleightcap commented Aug 2, 2023

View reviewed changes

gen/pb-rust/tests/unit.rs Outdated Show resolved Hide resolved

jleightcap marked this pull request as ready for review August 2, 2023 20:42

jleightcap mentioned this pull request Aug 2, 2023

Generate JSON Schema from Protobuf definitions and use it to enforce fields? #67

Closed

jleightcap mentioned this pull request Aug 3, 2023

jsonschema: container fix, updated compilation options #121

Merged

jleightcap force-pushed the jl/pb-rust-schemafy branch from 00831d6 to 4baf1ac Compare August 3, 2023 18:40

woodruffw reviewed Aug 3, 2023

View reviewed changes

woodruffw mentioned this pull request Aug 3, 2023

Optimize the Rust codegen #115

Closed

jleightcap added 5 commits August 4, 2023 12:58

Merge branch 'main' of github.com:sigstore/protobuf-specs

ff07ab4

pb-rust: schemafy only Bundles

dcdd646

Signed-off-by: Jack Leightcap <jack.leightcap@trailofbits.com> pb-rust: schemafy construction Signed-off-by: Jack Leightcap <jack.leightcap@trailofbits.com>

pb-rust: port prost unit tests

6659242

adapted from sigstore@b7d905d Signed-off-by: Jack Leightcap <jack.leightcap@trailofbits.com>

pb-rust: bundle deserialize integration

d309c13

Signed-off-by: Jack Leightcap <jack.leightcap@trailofbits.com>

pb-rust: use camelCase consistently in schema/artifacts

31c3ad9

Signed-off-by: Jack Leightcap <jack.leightcap@trailofbits.com>

jleightcap force-pushed the jl/pb-rust-schemafy branch from 4baf1ac to 31c3ad9 Compare August 4, 2023 16:59

woodruffw reviewed Aug 4, 2023

View reviewed changes

pb-rust: schema generation bugs

f21f9ef

Signed-off-by: Jack Leightcap <jack.leightcap@trailofbits.com>

woodruffw reviewed Aug 4, 2023

View reviewed changes

pb-rust: re-enable test suite

1b22bff

Signed-off-by: Jack Leightcap <jack.leightcap@trailofbits.com>

jleightcap commented Aug 7, 2023

View reviewed changes

.github/workflows/rust-build.yml Show resolved Hide resolved

woodruffw approved these changes Aug 7, 2023

View reviewed changes

woodruffw merged commit 88fe2fb into sigstore:main Aug 7, 2023
7 checks passed

woodruffw deleted the jl/pb-rust-schemafy branch August 7, 2023 21:25

woodruffw mentioned this pull request Aug 29, 2023

Route away from experimental to 1.0 sigstore/sigstore-rs#274

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

pb-rust: JSON schema compilation source #118

pb-rust: JSON schema compilation source #118

jleightcap commented Aug 2, 2023 •

edited by woodruffw

Loading

jleightcap commented Aug 2, 2023

jleightcap commented Aug 2, 2023 •

edited

Loading

woodruffw commented Aug 2, 2023

jleightcap commented Aug 2, 2023 •

edited

Loading

jleightcap commented Aug 2, 2023

jleightcap commented Aug 2, 2023

woodruffw Aug 3, 2023

jleightcap Aug 3, 2023

jleightcap Aug 3, 2023

woodruffw Aug 4, 2023 •

edited

Loading

jleightcap Aug 4, 2023

woodruffw Aug 4, 2023

jleightcap Aug 7, 2023

jleightcap Aug 7, 2023

jleightcap Aug 7, 2023

woodruffw commented Aug 7, 2023

		use serde::{Deserialize, Serialize};
		schemafy::schemafy!("schemas/Bundle.schema.json");

pb-rust: JSON schema compilation source #118

pb-rust: JSON schema compilation source #118

Conversation

jleightcap commented Aug 2, 2023 • edited by woodruffw Loading

Summary

Release Note

Documentation

jleightcap commented Aug 2, 2023

jleightcap commented Aug 2, 2023 • edited Loading

woodruffw commented Aug 2, 2023

jleightcap commented Aug 2, 2023 • edited Loading

jleightcap commented Aug 2, 2023

jleightcap commented Aug 2, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

woodruffw Aug 4, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

woodruffw commented Aug 7, 2023

jleightcap commented Aug 2, 2023 •

edited by woodruffw

Loading

jleightcap commented Aug 2, 2023 •

edited

Loading

jleightcap commented Aug 2, 2023 •

edited

Loading

woodruffw Aug 4, 2023 •

edited

Loading