[question] How to perform type conversions while populating

[djkoloski]

Hi! Thanks for all your work on bebop, it's a really exciting new framework.

I'm working on integrating bebop into the rust serialization benchmark. The code is structured like this:

• Data starts as native types (think user-defined structs).
• Then some frameworks like prost (protobuf) may convert the native types to their generated types (this is usually called the "populate" step).
• The data (native or populated) is then serialized.
• The process runs backwards for deserializing

The populate and serialize (also called encode) steps are separated out to time them separately. That way, users can pick which timing is appropriate for their use case.

I made a trait for bebop that looks like this:

pub trait Serialize<'a> {
    type Populated: 'a + Record<'a>;

    fn populate_bb(&'a self) -> Self::Populated;
    fn depopulate_bb(target: Self::Populated) -> Self;
}

This is intended to convert a native type to a bebop type so that it can be serialized. It works great, except for the following example:

lib.rs:

struct MyStruct {
    my_int: i32,
    my_float: f32,
}

struct MyData {
    my_meta: i32,
    my_data: Vec<MyStruct>,
}

schema.bop:

struct MyStruct {
    int32 my_int;
    float32 my_float;
}

struct MyData {
    int32 my_meta;
    MyStruct[] my_data;
}

In this case, there's an array of user-defined structs that needs to be serialized. If there's a native counterpart to this, then I have to convert a &'a crate::MyData that holds a Vec<crate::MyStruct> to a bebop_generated::MyData<'a> that holds a bebop::SliceWrapper<'a, bebop_generated::MyStruct>.

I used this pattern successfully to serialize strings because I can borrow a &str from any String. I can't use this pattern to serialize collections of native types because I can't borrow a &bebop_generated::Type from any crate::Type.

One option is to define a second intermediate type that uses bebop structs but not bebop containers:

struct Prepopulated {
    my_int: i32,
    my_data: Vec<bebop_generated::MyStruct>,
}

However this really increases the amount of work needed to get things up and running and will be very complicated later on with highly-structured data.

Is there a better way to approach this problem? Or a location in the serialization flow where I could define some custom behavior?

[mattiekat]

Welcome to early adoption 😄

I have been aware of some of these sharp edges but have been holding off on making any changes until I could play with it more myself, ideally in a real-world use case. To the best of my knowledge there is not a super easy way to do the conversion right now and if you look at my benchmark code you can see I define a very manual conversion here.

My thought while I was writing things out was to implement something like the SliceWrapper but as more of a ListWrapper or something. (If you have a name suggestion please share).

enum ListWrapper<'raw, T> {
  OwnedVecOwnedData(Vec<T>),
  BorrowedSliceOwnedData(&'raw [T]),
  OwnedVecBorrowedData(Vec<&'raw T>),
  BorrowedSliceBorrowedData(&'raw [&'raw T]),
  Wrapper(SliceWrapper<'raw, T>)
}

This would allow us to handle (I think all) of the different ownership cases more easily, and we can implement From quite easily.

Though taking this one step further, we could do a

enum OwnershipWrapper<'raw, O, B> {
  Owned(O),
  Borrowed(&'raw B)
}

which allows for

type WrappedVec<'a, T> = OwnershipWrapper<'a, Vec<OwnershipWrapper<'a, T, T>, [OwnershipWrapper<'a, T, T>]>;
struct MyData<'raw> {
  my_meta: i32,
  my_data: WrappedVec<'raw, MyStruct>
}

(well roughly, forgive me if this does not compile but I think the idea should work even if it takes a few tweaks)

This looks like it might be more painful to use than the former suggestion though, but with deref it might be okay. We also have to keep in mind that the map type has a similar but less extreme version of this issue.

Option "C" might be to just implement some From traits or something so you don't have to do it by hand. Not sure how this would work yet though or if it is possible.

Let me know if this seems like it would solve the issue you are describing and what you think of it.