Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Replace SerdeDeDataProvider with BufferProvider #1369

Merged
merged 51 commits into from
Dec 9, 2021
Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
51 commits
Select commit Hold shift + click to select a range
8c7c922
Add serde_format to DataResponseMetadata
sffc Nov 25, 2021
7a0ccfe
Review feedback from previous PR
sffc Nov 25, 2021
84e151e
serde_format --> buffer_format; add attributes
sffc Nov 25, 2021
c5930db
rm attributes and make it non_exhaustive instead
sffc Nov 30, 2021
78497e6
fmt
sffc Nov 30, 2021
9464db3
Checkpoint
sffc Nov 30, 2021
d3c7596
Checkpoint
sffc Nov 30, 2021
eda0521
Checkpoint
sffc Nov 30, 2021
db58d37
Checkpoint: migrate fs provider to BufferFormat enum
sffc Nov 30, 2021
22b981b
Checkpoint: all of ICU4X building again
sffc Nov 30, 2021
ed238c3
fmt
sffc Dec 1, 2021
21b58ea
Checkpoint
sffc Dec 1, 2021
f3a4bdb
impl BufferProvider for FsDataProvider
sffc Dec 1, 2021
c59b20b
Migrate FsDataProvider to use SerdeBufferProvider for deserialization
sffc Dec 1, 2021
1697c85
Remove obsolete comment
sffc Dec 1, 2021
c4a79de
Small fix
sffc Dec 8, 2021
bc57f68
Merge remote-tracking branch 'upstream/main' into dp-work
sffc Dec 8, 2021
2bea550
More small fixes
sffc Dec 8, 2021
9767307
Checkpoint
sffc Dec 8, 2021
3e0a9a8
rm SerdeDeDataProvider
sffc Dec 8, 2021
a170c86
Rename method
sffc Dec 8, 2021
a868486
Small doc
sffc Dec 8, 2021
409c7ba
Checkpoint
sffc Dec 8, 2021
f061785
Checkpoint
sffc Dec 8, 2021
3f7a832
Checkpoint
sffc Dec 8, 2021
c0d355f
Checkpoint
sffc Dec 8, 2021
e740fae
Checkpoint
sffc Dec 8, 2021
a26a57b
fmt
sffc Dec 8, 2021
eb9f504
Update provider/core/src/buffer_provider.rs
sffc Dec 8, 2021
5563762
Resolve TODO in AbstractSerializer
sffc Dec 8, 2021
ad6e666
Comment on catchall match
sffc Dec 9, 2021
79a9abc
Fix test
sffc Dec 9, 2021
782c458
Checkpoint: add SerializingDataProvider
sffc Dec 9, 2021
a6af6bb
Checkpoint
sffc Dec 9, 2021
ffc21ae
Checkpoint
sffc Dec 9, 2021
d9796d4
Undo SerializingDataProvider
sffc Dec 9, 2021
18219d4
Checkpoint: serializable helper methods
sffc Dec 9, 2021
b9ecbe5
Rename serialization primitives
sffc Dec 9, 2021
67cb8cb
Update docs
sffc Dec 9, 2021
2a6f37b
Error refactoring
sffc Dec 9, 2021
613c76b
Rename features
sffc Dec 9, 2021
6a3a244
Everything building again
sffc Dec 9, 2021
9b062fb
Fix features build
sffc Dec 9, 2021
db1ad70
fmt
sffc Dec 9, 2021
cec8aed
Fix features in blob provider
sffc Dec 9, 2021
0d6be7e
Re-gen readme
sffc Dec 9, 2021
9d9af9e
Test fixes
sffc Dec 9, 2021
cb8b13c
A few more feature fixes
sffc Dec 9, 2021
bd5b889
Error message feedback
sffc Dec 9, 2021
2b83989
Merge remote-tracking branch 'upstream/main' into dp-work
sffc Dec 9, 2021
5619631
Merge remote-tracking branch 'sffc/dp-work' into dp-work
sffc Dec 9, 2021
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
17 changes: 8 additions & 9 deletions provider/core/src/error.rs
Original file line number Diff line number Diff line change
Expand Up @@ -59,10 +59,10 @@ pub enum Error {
/// An error occured during serialization or deserialization.
#[cfg(feature = "erased-serde")]
#[displaydoc("Serde error: {0}")]
Serde(erased_serde::Error),
Serialize(crate::serde::SerializeError),

#[displaydoc("Error while deserializing: {0}")]
ProviderSerde(crate::serde::Error),
Deserialize(crate::serde::DeserializeError),

/// The data provider encountered some other error when loading the resource, such as I/O.
#[displaydoc("Failed to load resource: {0}")]
Expand All @@ -72,16 +72,15 @@ pub enum Error {
#[cfg(feature = "std")]
impl std::error::Error for Error {}

#[cfg(feature = "erased-serde")]
impl From<erased_serde::Error> for Error {
fn from(e: erased_serde::Error) -> Self {
Error::Serde(e)
impl From<crate::serde::SerializeError> for Error {
fn from(e: crate::serde::SerializeError) -> Self {
Error::Serialize(e)
}
}

impl From<crate::serde::Error> for Error {
fn from(e: crate::serde::Error) -> Self {
Error::ProviderSerde(e)
impl From<crate::serde::DeserializeError> for Error {
fn from(e: crate::serde::DeserializeError) -> Self {
Error::Deserialize(e)
}
}

Expand Down
70 changes: 69 additions & 1 deletion provider/core/src/serde/de.rs
Original file line number Diff line number Diff line change
Expand Up @@ -3,14 +3,82 @@
// (online at: https://github.com/unicode-org/icu4x/blob/main/LICENSE ).

use super::BufferFormat;
use super::Error;
use crate::buffer_provider::BufferProvider;
use crate::prelude::*;
use core::marker::PhantomData;
use serde::de::Deserialize;
use yoke::trait_hack::YokeTraitHack;
use yoke::Yokeable;

/// Error type for deserialization.
#[derive(displaydoc::Display, Debug)]
pub enum Error {
/// An error originating in [`serde_json`].
#[cfg(feature = "provider_json")]
#[displaydoc("{0}")]
Json(serde_json::error::Error),

/// An error originating in [`bincode`].
#[cfg(feature = "provider_bincode1")]
#[displaydoc("{0}")]
Bincode1(bincode::Error),

/// An error originating in [`postcard`].
#[cfg(feature = "provider_postcard07")]
#[displaydoc("{0}")]
Postcard07(postcard::Error),

/// An error indicating that the desired buffer format is not available. This usually
/// means that a required feature was not enabled
#[allow(dead_code)]
#[displaydoc("Unavailable buffer format: {0:?} (do you need to enable a feature?)")]
sffc marked this conversation as resolved.
Show resolved Hide resolved
UnavailableFormat(BufferFormat),

/// An error indicating that the buffer format could not be deduced. This is usually
/// unexpected and could indicate a problem with the data pipeline setup.
#[displaydoc("Buffer format not specified")]
FormatNotSpecified,
}

#[cfg(feature = "provider_json")]
impl From<serde_json::error::Error> for Error {
fn from(e: serde_json::error::Error) -> Self {
Error::Json(e)
}
}

#[cfg(feature = "provider_bincode1")]
impl From<bincode::Error> for Error {
fn from(e: bincode::Error) -> Self {
Error::Bincode1(e)
}
}

#[cfg(feature = "provider_postcard07")]
impl From<postcard::Error> for Error {
fn from(e: postcard::Error) -> Self {
Error::Postcard07(e)
}
}

/// Returns an error if the buffer format is not enabled.
pub fn check_format_supported(buffer_format: BufferFormat) -> Result<(), Error> {
match buffer_format {
#[cfg(feature = "provider_json")]
BufferFormat::Json => Ok(()),

#[cfg(feature = "provider_bincode1")]
BufferFormat::Bincode1 => Ok(()),

#[cfg(feature = "provider_postcard07")]
BufferFormat::Postcard07 => Ok(()),

// Allowed for cases in which all features are enabled
#[allow(unreachable_patterns)]
_ => Err(Error::UnavailableFormat(buffer_format)),
}
}

/// A [`BufferProvider`] that deserializes its data using Serde.
pub struct DeserializingBufferProvider<'a, P: ?Sized>(&'a P);

Expand Down
72 changes: 3 additions & 69 deletions provider/core/src/serde/mod.rs
Original file line number Diff line number Diff line change
Expand Up @@ -25,9 +25,12 @@
mod de;
mod ser;

pub use ser::Error as SerializeError;
pub use ser::SerializeBox;
pub use ser::SerializeMarker;

pub use de::Error as DeserializeError;
pub use de::check_format_supported;
pub use de::AsDeserializingBufferProvider;
pub use de::DeserializingBufferProvider;

Expand All @@ -42,72 +45,3 @@ pub enum BufferFormat {
/// Serialize using Postcard version 0.7.
Postcard07,
}

/// An Error type specifically for the [`Deserializer`](serde::Deserializer)
#[derive(displaydoc::Display, Debug)]
pub enum Error {
/// An error originating in [`serde_json`].
#[cfg(feature = "provider_json")]
#[displaydoc("{0}")]
Json(serde_json::error::Error),

/// An error originating in [`bincode`].
#[cfg(feature = "provider_bincode1")]
#[displaydoc("{0}")]
Bincode1(bincode::Error),

/// An error originating in [`postcard`].
#[cfg(feature = "provider_postcard07")]
#[displaydoc("{0}")]
Postcard07(postcard::Error),

/// An error indicating that the desired buffer format is not available. This usually
/// means that a required feature was not enabled
#[allow(dead_code)]
#[displaydoc("Unavailable buffer format: {0:?} (do you need to enable a feature?)")]
UnavailableFormat(BufferFormat),

/// An error indicating that the buffer format could not be deduced. This is usually
/// unexpected and could indicate a problem with the data pipeline setup.
#[displaydoc("Buffer format not specified")]
FormatNotSpecified,
}

#[cfg(feature = "provider_json")]
impl From<serde_json::error::Error> for Error {
fn from(e: serde_json::error::Error) -> Self {
Error::Json(e)
}
}

#[cfg(feature = "provider_bincode1")]
impl From<bincode::Error> for Error {
fn from(e: bincode::Error) -> Self {
Error::Bincode1(e)
}
}

#[cfg(feature = "provider_postcard07")]
impl From<postcard::Error> for Error {
fn from(e: postcard::Error) -> Self {
Error::Postcard07(e)
}
}

/// Returns an error if the buffer format is not enabled.
pub fn check_format_supported(buffer_format: BufferFormat) -> Result<(), Error> {
match buffer_format {
#[cfg(feature = "provider_json")]
BufferFormat::Json => Ok(()),

#[cfg(feature = "provider_bincode1")]
BufferFormat::Bincode1 => Ok(()),

#[cfg(feature = "provider_postcard07")]
BufferFormat::Postcard07 => Ok(()),

// Allowed for cases in which all features are enabled
#[allow(unreachable_patterns)]
_ => Err(Error::UnavailableFormat(buffer_format)),
}
}
18 changes: 17 additions & 1 deletion provider/core/src/serde/ser.rs
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,22 @@ use alloc::boxed::Box;
use core::ops::Deref;
use crate::dynutil::UpcastDataPayload;

/// Error type for serialization.
#[derive(displaydoc::Display, Debug)]
pub enum Error {
/// An error originating in [`erased_serde`].
#[cfg(feature = "provider_json")]
#[displaydoc("{0}")]
Serde(erased_serde::Error),
}

#[cfg(feature = "erased-serde")]
impl From<erased_serde::Error> for Error {
fn from(e: erased_serde::Error) -> Self {
Error::Serde(e)
}
}

/// A wrapper around `Box<erased_serde::Serialize>` for integration with DataProvider.
#[derive(yoke::Yokeable)]
pub struct SerializeBox(Box<dyn erased_serde::Serialize>);
Expand Down Expand Up @@ -64,7 +80,7 @@ impl DataPayload<SerializeMarker> {
/// ).expect("Serialization should succeed");
/// assert_eq!("{\"message\":\"(und) Hello World\"}".as_bytes(), buffer);
/// ```
pub fn serialize(&self, mut serializer: &mut dyn erased_serde::Serializer) -> Result<(), DataError> {
pub fn serialize(&self, mut serializer: &mut dyn erased_serde::Serializer) -> Result<(), Error> {
self.get().erased_serialize(&mut serializer)?;
Ok(())
}
Expand Down