-
Notifications
You must be signed in to change notification settings - Fork 5.5k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Modify Pybind LoDTensor API according to length-based LoD #11106
Conversation
077235f
to
ae366a8
Compare
77fd3d5
to
75816da
Compare
75816da
to
6b11525
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Looks Good
paddle/fluid/pybind/pybind.cc
Outdated
std::back_inserter(new_lod)); | ||
LoD new_offset_lod = ConvertToOffsetBasedLoD(new_lod); | ||
PADDLE_ENFORCE(CheckLoD(new_offset_lod, -1), | ||
"the provided lod info is invalid"); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
lod
to reursive_sequence_lengths
?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Done.
Those failed tests. I'm wondering if the test can |
61cc22a
to
d7fb929
Compare
d7fb929
to
051d7e6
Compare
@panyx0718 I have catch the exceptions in the |
Some background can be found here.
Essentially, our level of detail info (LoD) is currently both implemented in C++ and exposed in Python using "offsets". We want to encapsulate the
offset
implementation detail in C++ and only exposelength
based interface in Python.What this PR does (See the main change in
pybind.cc
here):Previously, the
set_lod
andlod
API inpybind.cc
uses offset_based LoD.This PR modifies the LoDTensor API in
pybind.cc
so that the user on the python side can provide or obtain length-based LoD usingset_lod
andlod
, respectively.The
init
API is also modified so that user can create a LoDTensor object in Python and initialize it with a length-based LoD.A
has_valid_lod
API is added inpybind.cc
so that user can check if the LoD info in a tensor is valid and matches the its data dimension on the python side.Utility functions to create LoDTensor in the python side defined here is largely simplified using the updated
set_lod
andlod
in pybind. (See changes here)Over 30 tests that uses
set_lod
andlod
has been modified to reflect the updated LoDTensor API inpybind.cc
.