Feature/shuffle reader #8991
Feature/shuffle reader #8991
Conversation
… feature/shuffle_reader
… feature/shuffle_reader
|
|
||
| namespace paddle { | ||
| namespace framework { | ||
|
|
||
| class ReaderBase { | ||
| public: | ||
| explicit ReaderBase(const std::vector<DDim>& shapes) : shapes_(shapes) { | ||
| PADDLE_ENFORCE(!shapes_.empty()); | ||
| } | ||
| virtual void ReadNext(std::vector<LoDTensor>* out) = 0; | ||
|
|
||
| virtual void ReInit() = 0; |
There was a problem hiding this comment.
Sorry not related for this PR, just for discussion:
Do we need ReInit for ReaderBase? Maybe readers like network reader is a stream of data, can not be reinitialized back to the beginning.
What is the use case for ReInit? If we don't have a clear use case we probably should drop this method from the base class.
There was a problem hiding this comment.
For a network reader, we can just throw an exception.
The reader module is in flux. I just discussed the design doc with @JiayiFeng yesterday. Whole design doc will be written soon, maybe within 1-2 days.
There was a problem hiding this comment.
Thanks for the prompt reply!
| DDim shape(size_t idx) const; | ||
| std::vector<DDim> shapes() const { return shapes_; } | ||
| void set_shapes(const std::vector<DDim>& shapes) { shapes_ = shapes; } | ||
|
|
||
| virtual bool HasNext() const = 0; |
There was a problem hiding this comment.
How do you think of replacing HasNext with pre-defined error code returned from ReadNext (e.g., EOF indicates does not have next).
| enum_range.insert("CPU"); | ||
| AddAttr<std::string>("place", "The double buffer place, default is CPU") | ||
| .SetDefault("CPU") | ||
| .InEnum({enum_range}); |
There was a problem hiding this comment.
Shouldn't 'CPU' be in this enum?
| Item() : ctx_(nullptr) {} | ||
|
|
||
| std::vector<framework::LoDTensor> payloads_; | ||
| platform::DeviceContext* ctx_; |
There was a problem hiding this comment.
It seems that this ctx_ will only hold gpu_ctx. So gpu_ctx_ might be a better name.
| framework::Channel<std::vector<framework::LoDTensor>>* buffer_; | ||
| framework::Channel<Item>* buffer_; | ||
| platform::Place place_; | ||
| std::vector<std::unique_ptr<platform::DeviceContext>> ctxs_; |
There was a problem hiding this comment.
The DoubleBufferReader only takes one Place and all ctx would be the same. So why do we need a vector here?
No description provided.