Skip to content

json2sbp: construct sbp from json fields #1183

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 15 commits into from
Jul 10, 2022
Merged

json2sbp: construct sbp from json fields #1183

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
15 commits
Select commit Hold shift + click to select a range
d9533f3
add deserialize derive
adrian-kong Jul 5, 2022
9347cee
Release v4.5.0
adrian-kong Jul 1, 2022
0daf244
Merge branch 'master' into adrian/fromfields
adrian-kong Jul 7, 2022
5261f5e
Merge remote-tracking branch 'origin/adrian/fromfields' into adrian/f…
adrian-kong Jul 7, 2022
6b19269
added deserialize, added from-fields for json2sbp
adrian-kong Jul 7, 2022
13d9051
cleanup
notoriaga Jul 7, 2022
f052f91
comment
notoriaga Jul 7, 2022
d871bdf
floating point err
adrian-kong Jul 8, 2022
bb581c2
added visitor for vec sbpstring
adrian-kong Jul 8, 2022
ffb5254
added visitor for array sbpstring
adrian-kong Jul 8, 2022
3bce67f
added roundtrip tests
adrian-kong Jul 8, 2022
2112826
added str visitor
adrian-kong Jul 8, 2022
1219563
add json test data without --float-compat
notoriaga Jul 8, 2022
c8405a8
update cargo toml's
Jul 8, 2022
3798304
remove some panics
notoriaga Jul 8, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 3 additions & 3 deletions generator/sbpg/targets/resources/rust/sbp2json_cargo.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,11 +18,11 @@ keywords = ["encoding", "parsing"]

[dependencies.sbp]
path = "../sbp" # TODO: replace with published `sbp` crate version
features = ["json"]
features = ["json", "float_roundtrip"]

[dependencies]
env_logger = "0.8"
serde_json = "1.0"
serde_json = "1.0.82"
clap = { version = "3.1.15", features = ["derive"] }
mimalloc = { version = "0.1", default-features = false }

Expand All @@ -34,7 +34,7 @@ default-features = false
sha2 = "0.8"
hex = "0.4"
assert_cmd = "1.0.1"
serde_json = "1.0"
serde_json = "1.0.82"
assert-json-diff = "2.0"

[profile.release]
Expand Down
5 changes: 3 additions & 2 deletions generator/sbpg/targets/resources/rust/sbp_cargo.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -21,6 +21,7 @@ readme = "../../README.md"
default = []
async = ["futures", "dencode/async"]
json = ["serde", "serde_json", "serde-big-array", "base64"]
float_roundtrip = ["serde_json/float_roundtrip"]
link = ["slotmap"]

[lib]
Expand All @@ -45,11 +46,11 @@ features = ["derive"]
optional = true

[dependencies.serde_json]
version = "1.0"
version = "1"
optional = true

[dependencies.serde-big-array]
version = "0.4.1"
version = "0.4"
optional = true

[dependencies.base64]
Expand Down
20 changes: 20 additions & 0 deletions generator/sbpg/targets/resources/rust/sbp_messages_mod.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -123,6 +123,26 @@ pub enum Sbp {
Unknown( Unknown ),
}

#[cfg(feature = "serde")]
impl<'de> serde::Deserialize<'de> for Sbp {
fn deserialize<D>(deserializer: D) -> Result<Self, D::Error>
where
D: serde::Deserializer<'de>,
{
let value = serde_json::Value::deserialize(deserializer)?;
Copy link
Contributor

@notoriaga notoriaga Jul 7, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

just to note there is probably a better way to do this. We can't use the derive + an untagged enum because some messages are subsets of other messages which causes messages to be deserialized as the wrong type. It might work if we added #[serde(deny_unknown_fields)]. But untagged still seems like the wrong approach as it just tries each variant in order which is inefficient as we know the type ahead of time.

typically you would not want to impl serde::Deserialize by using a specific data format like serde_json. But we don't actually expose a serde feature just a json one so any time we have serde we'll also have serde_json. I think there should be way to implement this with just serde. It'll look something like a deserializer that just grabs the id, and then a big match that forwards to the correct message deserializer. But we could look into that later if we ever decide to expose a plain serde feature

match value.get("msg_type").and_then(|v| v.as_u64()).and_then(|v| v.try_into().ok()) {
((*- for m in msgs *))
Some( (((m.msg_name)))::MESSAGE_TYPE) => {
serde_json::from_value::<(((m.msg_name)))>(value).map(Sbp::(((m.msg_name))) )
},
((*- endfor *))
_ => {
serde_json::from_value::<Unknown>(value).map(Sbp::Unknown)
},
}.map_err(serde::de::Error::custom)
}
}

impl Sbp {
/// Parse a message from a [Frame](crate::Frame).
///
Expand Down
8 changes: 4 additions & 4 deletions generator/sbpg/targets/resources/rust/sbp_messages_template.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -41,22 +41,22 @@ use crate::messages::(((i)))::*;
((*- elif m.short_desc *))
/// (((m.short_desc)))
((*- endif *))
#[cfg_attr(feature = "serde", derive(serde::Serialize))]
#[cfg_attr(feature = "serde", derive(serde::Serialize, serde::Deserialize))]
#[derive(Debug, PartialEq, Clone)]
pub struct (((m.msg_name))) {
((*- if m.is_real_message *))
/// The message sender_id
#[cfg_attr(feature = "serde", serde(skip_serializing))]
#[cfg_attr(feature = "serde", serde(skip_serializing, alias = "sender"))]
pub sender_id: Option<u16>,
((*- endif *))
((*- for f in m.fields *))
((*- if f.desc *))
/// (((f.desc | commentify(indent=2) )))
((*- endif *))
((*- if f.type_id == "array" and "size" in f.options and f.options["size"].value >= 32 *))
#[cfg_attr(feature = "serde", serde(with="BigArray", rename(serialize = "(((f.identifier)))")))]
#[cfg_attr(feature = "serde", serde(with="BigArray", rename = "(((f.identifier)))"))]
((*- else *))
#[cfg_attr(feature = "serde", serde(rename(serialize = "(((f.identifier)))")))]
#[cfg_attr(feature = "serde", serde(rename = "(((f.identifier)))"))]
((*- endif *))

pub (((f.field_name))): (((f.type))),
Expand Down
5 changes: 3 additions & 2 deletions rust/sbp/Cargo.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -21,6 +21,7 @@ readme = "../../README.md"
default = []
async = ["futures", "dencode/async"]
json = ["serde", "serde_json", "serde-big-array", "base64"]
float_roundtrip = ["serde_json/float_roundtrip"]
link = ["slotmap"]

[lib]
Expand All @@ -45,11 +46,11 @@ features = ["derive"]
optional = true

[dependencies.serde_json]
version = "1.0"
version = "1"
optional = true

[dependencies.serde-big-array]
version = "0.4.1"
version = "0.4"
optional = true

[dependencies.base64]
Expand Down
10 changes: 10 additions & 0 deletions rust/sbp/src/json/de.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,16 @@ pub fn iter_messages<R: io::Read>(input: R) -> impl Iterator<Item = Result<Sbp,
JsonDecoder::framed(input)
}

/// Deserialize the IO stream into an iterator of messages. Unlike [iter_messages], this function
/// will use the fields of the JSON message, rather than the base64 encoded payload.
pub fn iter_messages_from_fields<R: io::Read>(
input: R,
) -> impl Iterator<Item = Result<Sbp, JsonError>> {
Deserializer::from_reader(input)
.into_iter()
.map(|msg| msg.map_err(JsonError::SerdeJsonError))
}

/// Deserialize the IO stream into an iterator of [Json2JsonInput] messages.
pub fn iter_json2json_messages<R: io::Read>(
input: R,
Expand Down
2 changes: 1 addition & 1 deletion rust/sbp/src/json/mod.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,7 @@ pub use serde_json::ser::CompactFormatter;

#[cfg(feature = "async")]
pub use de::stream_messages;
pub use de::{iter_json2json_messages, iter_messages};
pub use de::{iter_json2json_messages, iter_messages, iter_messages_from_fields};

pub use ser::{to_vec, to_writer, Json2JsonEncoder, JsonEncoder};

Expand Down
Loading