Support big arrays #631
Comments
|
One relatively easy workaround for serialization is coercing to a slice: struct S {
#[serde(serialize_with = "<[_]>::serialize")]
arr: [u8; 256],
}Deserialization is still annoying I think. |
|
Hey folks, this feature is important to me because I'd like to be able to serialize a 512-bit hash (so, 64 bytes) and because the serde impls necessarily only go up to As workarounds I'm considering using Would it be okay to add impls up to 64? Or is there some reason that hasn't been done? |
|
In the meantime, perhaps we should add impls for the sizes that |
|
@clarcharr I would prefer to stick with what the standard library does, which is 0 to 32 (inclusive). |
|
Here is a workaround for deserializing. #[macro_use]
extern crate serde_derive;
extern crate serde;
extern crate serde_json;
use std::fmt;
use std::marker::PhantomData;
use serde::ser::{Serialize, Serializer, SerializeTuple};
use serde::de::{Deserialize, Deserializer, Visitor, SeqAccess, Error};
trait BigArray<'de>: Sized {
fn serialize<S>(&self, serializer: S) -> Result<S::Ok, S::Error>
where S: Serializer;
fn deserialize<D>(deserializer: D) -> Result<Self, D::Error>
where D: Deserializer<'de>;
}
macro_rules! big_array {
($($len:expr,)+) => {
$(
impl<'de, T> BigArray<'de> for [T; $len]
where T: Default + Copy + Serialize + Deserialize<'de>
{
fn serialize<S>(&self, serializer: S) -> Result<S::Ok, S::Error>
where S: Serializer
{
let mut seq = serializer.serialize_tuple(self.len())?;
for elem in &self[..] {
seq.serialize_element(elem)?;
}
seq.end()
}
fn deserialize<D>(deserializer: D) -> Result<[T; $len], D::Error>
where D: Deserializer<'de>
{
struct ArrayVisitor<T> {
element: PhantomData<T>,
}
impl<'de, T> Visitor<'de> for ArrayVisitor<T>
where T: Default + Copy + Deserialize<'de>
{
type Value = [T; $len];
fn expecting(&self, formatter: &mut fmt::Formatter) -> fmt::Result {
formatter.write_str(concat!("an array of length ", $len))
}
fn visit_seq<A>(self, mut seq: A) -> Result<[T; $len], A::Error>
where A: SeqAccess<'de>
{
let mut arr = [T::default(); $len];
for i in 0..$len {
arr[i] = seq.next_element()?
.ok_or_else(|| Error::invalid_length(i, &self))?;
}
Ok(arr)
}
}
let visitor = ArrayVisitor { element: PhantomData };
deserializer.deserialize_tuple($len, visitor)
}
}
)+
}
}
big_array! {
40, 48, 50, 56, 64, 72, 96, 100, 128, 160, 192, 200, 224, 256, 384, 512,
768, 1024, 2048, 4096, 8192, 16384, 32768, 65536,
}
#[derive(Serialize, Deserialize)]
struct S {
#[serde(with = "BigArray")]
arr: [u8; 64],
}
fn main() {
let s = S { arr: [1; 64] };
let j = serde_json::to_string(&s).unwrap();
println!("{}", j);
serde_json::from_str::<S>(&j).unwrap();
} |
|
As long as you're not working with primes: #[derive(Serialize, Deserialize, Debug)]
struct MyStruct {
data: [[u8; 32]; 16],
}
impl MyStruct {
fn data(&self) -> &[u8; 512] {
use std::mem::transmute;
unsafe { transmute(&self.data) }
}
}This is a pretty neat workaround for when never expect a human to use the serialised version (e.g. bincode), as it creates a nested array. Added bonus: it also works for |
|
FWIW, I use this: use serde::{Serialize, Serializer};
pub fn serialize_array<S, T>(array: &[T], serializer: S) -> Result<S::Ok, S::Error>
where S: Serializer, T: Serialize {
array.serialize(serializer)
}
#[macro_export]
macro_rules! serde_array { ($m:ident, $n:expr) => {
pub mod $m {
use std::{ptr, mem};
use serde::{Deserialize, Deserializer, de};
pub use $crate::serialize_array as serialize;
use super::*;
pub fn deserialize<'de, D, T>(deserializer: D) -> Result<[T; $n], D::Error>
where D: Deserializer<'de>, T: Deserialize<'de> + 'de {
let slice: Vec<T> = Deserialize::deserialize(deserializer)?;
if slice.len() != $n {
return Err(de::Error::custom("input slice has wrong length"));
}
unsafe {
let mut result: [T; $n] = mem::uninitialized();
for (src, dst) in slice.into_iter().zip(&mut result[..]) {
ptr::write(dst, src);
}
Ok(result)
}
}
}
}}
serde_array!(a64, 64);
serde_array!(a120, 120);
serde_array!(a128, 128);
serde_array!(a384, 384);And then struct Foo {
#[serde(with = "a128")]
bar: [f32; 128],
} |
|
I do not plan to implement the workaround from heapsize_derive. I would prefer to see something like #631 (comment) provided in a crate. |
|
Yes go for it! Thanks. |
|
Thanks! Published: https://github.com/est31/serde-big-array | https://crates.io/crates/serde-big-array |
|
@dtolnay what do you think, does moving it into the serde-rs org make sense? |
|
+1 on moving this into serde-rs. Ability to serialize/de-serialize arrays larger than 32 should be a core feature. I'd use it for sure. @dtolnay I do think we should consider changing the derive macro to support it instead. I'd rather have it work out of the box if possible. |
|
I posted a request for implementation of a slightly different approach: dtolnay/request-for-implementation#17. |
|
To all the people in this thread hoping that const generics will resolve this: When trying to port serde to using const generics, I came across the problem that This prolongs the lifetime of the |
Servo does this to support big arrays in one of their proc macros:
https://github.com/servo/heapsize/blob/44e86d6d48a09c9cbc30a122bc8725b188d017b2/derive/lib.rs#L36-L41
Let's do the same but only if the size of the array exceeds our biggest builtin impl.
Thanks @nox.
The text was updated successfully, but these errors were encountered: