Implement framework::Variable

[wangkuiyi]

I am using a C++ technique called type-information hiding in this implementation.

Compared with this design, my implementation doesn't use boost::any and thus doesn't rely on boost. Actually, this design relies nothing other than #include <typeinfo>.

Compare with Caffe2's implementation, mine doesn't use self-defined type inference mechanism like caffe2::TypeMeta, which is defined in caffe2/core/typeid.{h,cc,test_cc}, and saved about 500 lines C++ code:

$ wc -l ~/work/caffe2/caffe2/core/typeid*
      56 /Users/yiwang/work/caffe2/caffe2/core/typeid.cc
     300 /Users/yiwang/work/caffe2/caffe2/core/typeid.h
     136 /Users/yiwang/work/caffe2/caffe2/core/typeid_test.cc

[emailweixu]

std::type_index(typeid(T)) == type_index(holder_->Type())

[wangkuiyi]

Done

[emailweixu]

why not unique_ptr?

[wangkuiyi]

done. using unique_ptr

[emailweixu]

just unique_ptr is enough? no need for deleter.

[wangkuiyi]

done

[emailweixu]

need to check whether T is same as holder_->Type().

[wangkuiyi]

done

[qingqing01]

In the variable design, we want to add Reset(T* ptr) function, which can delete the hosted pointer and set a new ptr. This interface is useful when we want to reuse a Variable and do not need to new another empty Variable. Should we add this function?

[wangkuiyi]

I had Reset but deleted it later, because it is out of the imagined usage in the design doc (README.md). I think we can add it at any time in the future if we believe we do need it.

[qingqing01]

We also want to use std::unique_ptr for boost::any and do not manage the delete method in the variable design at first. But it needs to use get() method when returning the hosted pointer. I'm not sure whether it is good by returning the value using get() method.

[wangkuiyi]

Since cuDNN and other BLAS libraries would anyway need a raw pointer from unique_ptr or other smart pointers, I think it is OK (and indeed a mandatory) to call get().

---

BTW, I guess you wanted to say "We had thought about using std::unique_ptr ..."? :-)

[qingqing01]

You're right. Thanks! :)

[qingqing01]

There is a unique id for the different type in the Caffe2's TypeMeta. The id is also used for type check in run-time. And the serializer method is created base on this id.

Here, we store the std::type_info to make check in run-time. There is a hash_code function in std:: type_info. This is equivalent to the unique id in the TypeMeta. Just comment here.

[hedaoyuan]

LGTM.
This pr is better than our previous design. Since it does not depend on boost::any. And remove the destructor with unique_ptr. I think this is the smallest implementation of Variable.

[JiayiFeng]

It seems that 'lazy allocation' is split into two parts: the part in Variable and the part in Tensor. I am wondering whether following design will be a little better:

In Tensor:

Tensor<Device, T, D> {
  // return an empty tensor with ptr_ pointed to nullptr
  Tensr();
  // return a tensor with newly allocated memory
  Tensor(Dim<D> size);
  // check whether ptr_ points to nullptr
  bool is_empty();
}

In Variable:
before GetMutable<Tensor>() is called, Variable is assigned with an empty Tenosr which is constructed by Tensor::Tensor(). When GetMutable<Tensor>() is called, it checks whether the tensor is empty, if so, construct a new tensor by Tensor::Tensor(Dim<D> size) and then replace the empty one with it.

The replacement costs little time because the empty tensor holds no memory.

The benefit of this design is that all implement of 'lazy allocation' is now limited in Variable::GetMutable<Tensor>(), and we have no need to think about it during tensor implementation. It is a little like Majel's approach.

[wangkuiyi]

I think Variable can hold anything -- it doens't have to be a Tensor, it could be a Scope even. Thus we might not initialize a Variable with an empty tensor.

[JiayiFeng]

Got it, thanks!