[Question] std::shared_ptr and std::unique_ptr support

[frol]

Is there a specific reason not to support std::shared_ptr and std::unique_ptr?

I want to be able to serialize std::unordered_map<std::string, std::shared_ptr<MyClass>>, but it seems that msgpack-c doesn't support such things.

Here is the minimal example with std::vector for simplicity:

#include <msgpack.hpp>
#include <vector>
#include <string>
#include <memory>
#include <iostream>

class myclass {
private:
    std::string m_str;
    std::vector<int> m_vec;
public:
    myclass() {}
    myclass(std::string name) {
        m_str = name;
        m_vec.push_back(1);
    }
    std::string name() {
        return this->m_str;
    }
    MSGPACK_DEFINE(m_str, m_vec);
};

int main(void) {
        std::vector<std::shared_ptr<myclass> > vec;
        // add some elements into vec...
        vec.push_back(std::make_shared<myclass>("asd"));

        // serialize
        msgpack::sbuffer sbuf;
        msgpack::pack(sbuf, vec);

        // debug output
        std::cout << std::string(sbuf.data(), sbuf.size()) << std::endl;

        // deserialize
        msgpack::unpacked msg;
        msgpack::unpack(&msg, sbuf.data(), sbuf.size());

        msgpack::object obj = msg.get();

        // you can convert object to myclass directly
        std::vector<std::shared_ptr<myclass> > rvec;
        obj.convert(&rvec);

        // debug output
        std::cout << rvec[0]->name() << std::endl;
}

I compile it with:

$ c++ -std=c++11 example.cpp -o example

The error I'm getting is:

In file included from /usr/local/include/msgpack.hpp:18:0,
                 from qq.cpp:1:
/usr/local/include/msgpack/object.hpp: In instantiation of ‘const msgpack::v1::object& msgpack::v1::adaptor::convert<T, Enabler>::operator()(const msgpack::v1::object&, T&) const [with T = std::shared_ptr<myclass>; Enabler = void]’:
/usr/local/include/msgpack/adaptor/adaptor_base.hpp:64:33:   required from ‘const msgpack::v1::object& msgpack::v1::operator>>(const msgpack::v1::object&, T&) [with T = std::shared_ptr<myclass>]’
/usr/local/include/msgpack/object.hpp:503:24:   required from ‘T& msgpack::v1::object::convert(T&) const [with T = std::shared_ptr<myclass>]’
/usr/local/include/msgpack/adaptor/vector.hpp:45:17:   required from ‘const msgpack::v1::object& msgpack::v1::adaptor::convert<std::vector<_RealType> >::operator()(const msgpack::v1::object&, std::vector<_RealType>&) const [with T = std::shared_ptr<myclass>]’
/usr/local/include/msgpack/adaptor/adaptor_base.hpp:64:33:   required from ‘const msgpack::v1::object& msgpack::v1::operator>>(const msgpack::v1::object&, T&) [with T = std::vector<std::shared_ptr<myclass> >]’
/usr/local/include/msgpack/object.hpp:503:24:   required from ‘T& msgpack::v1::object::convert(T&) const [with T = std::vector<std::shared_ptr<myclass> >]’
/usr/local/include/msgpack/object.hpp:510:5:   required from ‘T* msgpack::v1::object::convert(T*) const [with T = std::vector<std::shared_ptr<myclass> >]’
qq.cpp:42:26:   required from here
/usr/local/include/msgpack/object.hpp:177:5: error: ‘class std::shared_ptr<myclass>’ has no member named ‘msgpack_unpack’
     v.msgpack_unpack(o.convert());
     ^
/usr/local/include/msgpack/object.hpp: In instantiation of ‘static msgpack::v1::packer<Stream>& msgpack::v1::detail::packer_serializer<Stream, T>::pack(msgpack::v1::packer<Stream>&, const T&) [with Stream = msgpack::v1::sbuffer; T = std::shared_ptr<myclass>]’:
/usr/local/include/msgpack/object.hpp:186:54:   required from ‘msgpack::v1::packer<Stream>& msgpack::v1::adaptor::pack<T, Enabler>::operator()(msgpack::v1::packer<Stream>&, const T&) const [with Stream = msgpack::v1::sbuffer; T = std::shared_ptr<myclass>; Enabler = void]’
/usr/local/include/msgpack/adaptor/adaptor_base.hpp:70:30:   required from ‘msgpack::v1::packer<Stream>& msgpack::v1::operator<<(msgpack::v1::packer<Stream>&, const T&) [with Stream = msgpack::v1::sbuffer; T = std::shared_ptr<myclass>]’
/usr/local/include/msgpack/object.hpp:392:24:   required from ‘msgpack::v1::packer<Stream>& msgpack::v1::packer<Stream>::pack(const T&) [with T = std::shared_ptr<myclass>; Stream = msgpack::v1::sbuffer]’
/usr/local/include/msgpack/adaptor/vector.hpp:62:13:   required from ‘msgpack::v1::packer<Stream>& msgpack::v1::adaptor::pack<std::vector<_RealType> >::operator()(msgpack::v1::packer<Stream>&, const std::vector<_RealType>&) const [with Stream = msgpack::v1::sbuffer; T = std::shared_ptr<myclass>]’
/usr/local/include/msgpack/adaptor/adaptor_base.hpp:70:30:   required from ‘msgpack::v1::packer<Stream>& msgpack::v1::operator<<(msgpack::v1::packer<Stream>&, const T&) [with Stream = msgpack::v1::sbuffer; T = std::vector<std::shared_ptr<myclass> >]’
/usr/local/include/msgpack/object.hpp:392:24:   required from ‘msgpack::v1::packer<Stream>& msgpack::v1::packer<Stream>::pack(const T&) [with T = std::vector<std::shared_ptr<myclass> >; Stream = msgpack::v1::sbuffer]’
/usr/local/include/msgpack/pack.hpp:148:5:   required from ‘void msgpack::v1::pack(Stream&, const T&) [with Stream = msgpack::v1::sbuffer; T = std::vector<std::shared_ptr<myclass> >]’
qq.cpp:32:32:   required from here
/usr/local/include/msgpack/object.hpp:166:9: error: ‘const class std::shared_ptr<myclass>’ has no member named ‘msgpack_pack’
         v.msgpack_pack(o);
         ^

[redboltz]

Is there a specific reason not to support std::shared_ptr and std::unique_ptr?

There are two reasons. One is C++11 types are just started supporting. std::unique_ptr and std::shared_ptr is not implemented yet. The other is about sharing information of std::shared_prtr.

Consider the following example:

struct A {
    std::shared_ptr<int> a;
    std::shared_ptr<int> b;
    MSGPACK_DEFINE(a, b);
};

A obj;
obj.a.reset(new int(1));
obj.b = obj.a;
// now obj.a and obj.b are sharing the same object.

If I choose simple implementation that packing as T, int, and converting from msgpack::object to T using std::make_shared, after packing and unpacking obj, sharing information is lost. So, I think that is acceptable but not 100% sure yet.

What do you think?

BTW, Boost.Serialization preserves sharing information using special meta data.

[frol]

It would be great if we could preserve sharing information, but even without this it will be still quite useful.

I have an idea of how to preserve sharing information, but I'm not sure how well it will work as the thought just came to me. I am sure that we don't want to hack msgpack protocol, so I suggest to pack everything with duplicates, but use extra map wrapper to save unique id (for example, pointer hash), which can be used on unpacking step to deduplicate data.

[redboltz]

It would be great if we could preserve sharing information, but even without this it will be still quite useful.

Thanks, I will implement simpler, not preserve sharing information version.

I have an idea of how to preserve sharing information, but I'm not sure how well it will work as the thought just came to me. I am sure that we don't want to hack msgpack protocol, so I suggest to pack everything with duplicates, but use extra map wrapper to save unique id (for example, pointer hash), which can be used on unpacking step to deduplicate data.

I realized that we need to care different programming languages. Some of them don't have sharing object semantics. It is difficult to define a data structure that has sharing information and cope with those programming languages. Now, I started thinking that sharing object information should be implemented by application domain not msgpack-c library.

[frol]

Thanks, I will implement simpler, not preserve sharing information version.

This sounds great!

Now, I started thinking that sharing object information should be implemented by application domain not msgpack-c library.

As I said, it would be great if it is provided as extended functionality which user could enable if he/she wishes, it might be not the default behaviour. I see this shared_ptr/unique_ptr as a very common use-case and think it would be great to deal with it nicely.

[redboltz]

I implemented it as #329. @frol, could you try it?