New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Serializing map types #4115

Closed
erickt opened this Issue Dec 5, 2012 · 11 comments

Comments

Projects
None yet
4 participants
@erickt
Contributor

erickt commented Dec 5, 2012

std::serialization does not directly support serializing a map structure like LinearMap. Should we support it as a top-level construct? Records and structs are similar, but this map would allow for serializing non-string keys.

@nikomatsakis

This comment has been minimized.

Show comment
Hide comment
@nikomatsakis

nikomatsakis Dec 6, 2012

Contributor

This seems like a good idea to me. I presume what you mean is to add methods to the Serializer interface for maps, even though #[auto_serialize] would never generate code that calls them?

Contributor

nikomatsakis commented Dec 6, 2012

This seems like a good idea to me. I presume what you mean is to add methods to the Serializer interface for maps, even though #[auto_serialize] would never generate code that calls them?

@nikomatsakis

This comment has been minimized.

Show comment
Hide comment
@nikomatsakis

nikomatsakis Dec 6, 2012

Contributor

Thinking about this a bit more: one tricky issue that comes to mind is how to manage deserialization. I guess it'd be up to the deserializer to select an appropriate map implementation? Maybe we can supply a hint as part of the serialization as to what kind of map this originally was? Maybe we want a base serializer/deserializer interface that is more-or-less what we have now, and then some extended variants?

Contributor

nikomatsakis commented Dec 6, 2012

Thinking about this a bit more: one tricky issue that comes to mind is how to manage deserialization. I guess it'd be up to the deserializer to select an appropriate map implementation? Maybe we can supply a hint as part of the serialization as to what kind of map this originally was? Maybe we want a base serializer/deserializer interface that is more-or-less what we have now, and then some extended variants?

@erickt

This comment has been minimized.

Show comment
Hide comment
@erickt

erickt Dec 6, 2012

Contributor

An example of how this would be useful is how do we encode a map to and from json? Say we wanted to write this:

fn main() {
    let map = LinearMap();
    map.insert(~"a", 1);
    map.insert(~"b", 2);
    map.insert(~"c", 3);
    map.serialize(&json::Serializer(io::stdout()));
}

We could make a serializer like this:

impl<S, V: Serializable> LinearMap<~str, V>: Serializable<S> {
    fn serialize(&self, s: &S) {
        let mut i = 0;
        do s.emit_rec {
            for self.each |key, value| {
                s.emit_field(key, i, || value.serialize(s));
                i += 1;
            }
        }
    }
}

However, serializers support maps with non-string keys, like MessagePack (http://msgpack.org/). So to support that, we would have to write our impl to emit a vector of tuples, like this:

impl<S, K: Serializable, V: Serializable> LinearMap<K, V>: Serializable<S> {
    fn serialize(&self, s: &S) {
        let mut i = 0;
        do s.emit_owned_vec(self.len()) {
            for self.each |key, value| {
                do s.emit_vec_elt(i) {
                    do s.emit_tup(2) {
                        s.emit_tup_elt(0, || key.serialize(s));
                        s.emit_tup_elt(1, || value.serialize(s));
                    }
                }
                i += 1;
            }
        }
    }
}

It would take some effort to write serializers to infer that a vector of 2-tuples is actually a map, and generate a proper native map. Adding std::serialization could really simplify that code, and make serialization even more broadly applicable.

Contributor

erickt commented Dec 6, 2012

An example of how this would be useful is how do we encode a map to and from json? Say we wanted to write this:

fn main() {
    let map = LinearMap();
    map.insert(~"a", 1);
    map.insert(~"b", 2);
    map.insert(~"c", 3);
    map.serialize(&json::Serializer(io::stdout()));
}

We could make a serializer like this:

impl<S, V: Serializable> LinearMap<~str, V>: Serializable<S> {
    fn serialize(&self, s: &S) {
        let mut i = 0;
        do s.emit_rec {
            for self.each |key, value| {
                s.emit_field(key, i, || value.serialize(s));
                i += 1;
            }
        }
    }
}

However, serializers support maps with non-string keys, like MessagePack (http://msgpack.org/). So to support that, we would have to write our impl to emit a vector of tuples, like this:

impl<S, K: Serializable, V: Serializable> LinearMap<K, V>: Serializable<S> {
    fn serialize(&self, s: &S) {
        let mut i = 0;
        do s.emit_owned_vec(self.len()) {
            for self.each |key, value| {
                do s.emit_vec_elt(i) {
                    do s.emit_tup(2) {
                        s.emit_tup_elt(0, || key.serialize(s));
                        s.emit_tup_elt(1, || value.serialize(s));
                    }
                }
                i += 1;
            }
        }
    }
}

It would take some effort to write serializers to infer that a vector of 2-tuples is actually a map, and generate a proper native map. Adding std::serialization could really simplify that code, and make serialization even more broadly applicable.

@erickt

This comment has been minimized.

Show comment
Hide comment
@erickt

erickt Dec 6, 2012

Contributor

@nikomatsakis: Good point. Maybe we just provide deserializers for concrete types?

Contributor

erickt commented Dec 6, 2012

@nikomatsakis: Good point. Maybe we just provide deserializers for concrete types?

@erickt

This comment has been minimized.

Show comment
Hide comment
@erickt

erickt Dec 6, 2012

Contributor

@nikomatsakis: Yes, these Serializer::emit_map and etc methods would not be used by auto_serialize, just for types that opt into using them.

Contributor

erickt commented Dec 6, 2012

@nikomatsakis: Yes, these Serializer::emit_map and etc methods would not be used by auto_serialize, just for types that opt into using them.

@erickt

This comment has been minimized.

Show comment
Hide comment
@erickt

erickt Dec 6, 2012

Contributor

Ironically, this doesn't necessarily help out serializing maps to json. If we do support non-string-key maps, then if you want to serialize LinearMap<int, int> to json, either you:

  1. fail on the first non-string keys
  2. detect that the keys are non-strings and treat the map as [(int, int)]
  3. always emit maps as [(K, V)] types

I'd love to have trait specialization for this case, but I'm sure that adds a whole host of other problems.

Contributor

erickt commented Dec 6, 2012

Ironically, this doesn't necessarily help out serializing maps to json. If we do support non-string-key maps, then if you want to serialize LinearMap<int, int> to json, either you:

  1. fail on the first non-string keys
  2. detect that the keys are non-strings and treat the map as [(int, int)]
  3. always emit maps as [(K, V)] types

I'd love to have trait specialization for this case, but I'm sure that adds a whole host of other problems.

@erickt

This comment has been minimized.

Show comment
Hide comment
@erickt

erickt Dec 6, 2012

Contributor

(Responding to myself again) I do remember some talk of making a StrMap<V> type that has an interface better oriented toward working with strings. If we had that, then it'd be easy to serialize StrMap<V> to a json map.

Contributor

erickt commented Dec 6, 2012

(Responding to myself again) I do remember some talk of making a StrMap<V> type that has an interface better oriented toward working with strings. If we had that, then it'd be easy to serialize StrMap<V> to a json map.

@catamorphism

This comment has been minimized.

Show comment
Hide comment
@catamorphism

catamorphism May 23, 2013

Contributor

Added "far future" milestone

Contributor

catamorphism commented May 23, 2013

Added "far future" milestone

@nikomatsakis

This comment has been minimized.

Show comment
Hide comment
@nikomatsakis

nikomatsakis May 23, 2013

Contributor

(Deserialization, I realize now, is a non-issue: when deserializing, you always have the type you are deserializing into.)

Contributor

nikomatsakis commented May 23, 2013

(Deserialization, I realize now, is a non-issue: when deserializing, you always have the type you are deserializing into.)

@catamorphism

This comment has been minimized.

Show comment
Hide comment
@catamorphism

catamorphism Jul 29, 2013

Contributor

Revisiting for bug triage. I agree with my earlier self.

Contributor

catamorphism commented Jul 29, 2013

Revisiting for bug triage. I agree with my earlier self.

@alexcrichton

This comment has been minimized.

Show comment
Hide comment
@alexcrichton

alexcrichton Jul 30, 2013

Member

Currently there's a serialization/deserialization for HashMap/HashSet/TreeMap/TrieMap. Also, there's a function in the Encoder interface for emit_map/emit_map_elt_key/emit_map_elt_value.

I think that this satisfies what the bug was originally for, but feel free to reopen if it's intended for something else!

Member

alexcrichton commented Jul 30, 2013

Currently there's a serialization/deserialization for HashMap/HashSet/TreeMap/TrieMap. Also, there's a function in the Encoder interface for emit_map/emit_map_elt_key/emit_map_elt_value.

I think that this satisfies what the bug was originally for, but feel free to reopen if it's intended for something else!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment