Skip to content

Commit

Permalink
ARROW-3855: [Rust] Schema/Field/Datatype now have derived serde traits
Browse files Browse the repository at this point in the history
This PR makes `Schema`, `Field`, `DataType` serializable using the serde crate. This approach supports serialization to numerous binary and text formats supported by the serde crate.

The main benefit is to allow users of the Arrow crate to serialize structs that reference Arrow types (for example, allowing a logical query plan to be serialized and sent over the network).

Note that this does not change the custom JSON serialization that is already in place for serializing in the specific format specified in `format/Metadata.md`.

Author: Andy Grove <andygrove73@gmail.com>

Closes #3016 from andygrove/ARROW-3855 and squashes the following commits:

329da92 <Andy Grove> Merge branch 'master' into ARROW-3855
d988cc6 <Andy Grove> cargo fmt
bd8375d <Andy Grove> Schema/Field/Datatype now have derived serde traits
  • Loading branch information
andygrove committed Nov 28, 2018
1 parent 39d1e86 commit 0066af8
Show file tree
Hide file tree
Showing 3 changed files with 46 additions and 3 deletions.
2 changes: 2 additions & 0 deletions rust/Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -36,6 +36,8 @@ path = "src/lib.rs"
[dependencies]
bytes = "0.4"
libc = "0.2"
serde = { version = "1.0.80", features = ["alloc", "rc"] }
serde_derive = "1.0.80"
serde_json = "1.0.13"
rand = "0.5"
csv = "1.0.0"
Expand Down
42 changes: 39 additions & 3 deletions rust/src/datatypes.rs
Original file line number Diff line number Diff line change
Expand Up @@ -39,7 +39,7 @@ use serde_json::Value;
/// Nested types can themselves be nested within other arrays.
/// For more information on these types please see
/// [here](https://arrow.apache.org/docs/memory_layout.html).
#[derive(Debug, Clone, PartialEq, Eq)]
#[derive(Serialize, Deserialize, Debug, Clone, PartialEq, Eq)]
pub enum DataType {
Boolean,
Int8,
Expand All @@ -61,7 +61,7 @@ pub enum DataType {
/// Contains the meta-data for a single relative type.
///
/// The `Schema` object is an ordered collection of `Field` objects.
#[derive(Debug, Clone, PartialEq, Eq)]
#[derive(Serialize, Deserialize, Debug, Clone, PartialEq, Eq)]
pub struct Field {
name: String,
data_type: DataType,
Expand Down Expand Up @@ -300,7 +300,7 @@ impl fmt::Display for Field {
///
/// Note that this information is only part of the meta-data and not part of the physical memory
/// layout.
#[derive(Debug, Clone)]
#[derive(Serialize, Deserialize, Debug, Clone)]
pub struct Schema {
fields: Vec<Field>,
}
Expand Down Expand Up @@ -382,6 +382,42 @@ mod tests {
]);
}

#[test]
fn serde_struct_type() {
let person = DataType::Struct(vec![
Field::new("first_name", DataType::Utf8, false),
Field::new("last_name", DataType::Utf8, false),
Field::new(
"address",
DataType::Struct(vec![
Field::new("street", DataType::Utf8, false),
Field::new("zip", DataType::UInt16, false),
]),
false,
),
]);

let serialized = serde_json::to_string(&person).unwrap();

// NOTE that this is testing the default (derived) serialization format, not the
// JSON format specified in metadata.md

assert_eq!(
"{\"Struct\":[\
{\"name\":\"first_name\",\"data_type\":\"Utf8\",\"nullable\":false},\
{\"name\":\"last_name\",\"data_type\":\"Utf8\",\"nullable\":false},\
{\"name\":\"address\",\"data_type\":{\"Struct\":\
[{\"name\":\"street\",\"data_type\":\"Utf8\",\"nullable\":false},\
{\"name\":\"zip\",\"data_type\":\"UInt16\",\"nullable\":false}\
]},\"nullable\":false}]}",
serialized
);

let deserialized = serde_json::from_str(&serialized).unwrap();

assert_eq!(person, deserialized);
}

#[test]
fn struct_field_to_json() {
let f = Field::new(
Expand Down
5 changes: 5 additions & 0 deletions rust/src/lib.rs
Original file line number Diff line number Diff line change
Expand Up @@ -19,9 +19,14 @@ extern crate bytes;
extern crate csv as csv_crate;
extern crate libc;

#[macro_use]
extern crate serde_derive;

#[macro_use]
extern crate serde_json;

extern crate serde;

extern crate rand;

pub mod array;
Expand Down

0 comments on commit 0066af8

Please sign in to comment.