Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Memory]More memory optimization policy #8690

Merged
merged 20 commits into from
Mar 12, 2018
Merged

[Memory]More memory optimization policy #8690

Show file tree
Hide file tree
Changes from 2 commits
Commits
Show all changes
20 commits
Select commit Hold shift + click to select a range
3eaa820
add memopt level
QiJune Mar 1, 2018
261571f
add opt level for image classification demo
QiJune Mar 1, 2018
5ca9615
clean code
QiJune Mar 1, 2018
15b40db
add delete op
QiJune Mar 2, 2018
6b5456e
clean code
QiJune Mar 5, 2018
cf12a01
test machine translation demo
QiJune Mar 5, 2018
7fa1d98
merge baidu/develop
QiJune Mar 5, 2018
c9cbfb3
clean code
QiJune Mar 5, 2018
60e49e8
clean code
QiJune Mar 5, 2018
2d0f3b9
skip fill constant with force cpu
QiJune Mar 6, 2018
8adba60
clean code
QiJune Mar 6, 2018
eb7f763
Merge remote-tracking branch 'baidu/develop' into fix_memopt_decay
QiJune Mar 6, 2018
674d04c
clean code
QiJune Mar 6, 2018
c3217a9
Merge remote-tracking branch 'baidu/develop' into more_level_memopt
QiJune Mar 6, 2018
0ec9860
Merge branch 'fix_memopt_decay' into more_level_memopt
QiJune Mar 6, 2018
066cb87
refine code
QiJune Mar 6, 2018
abad0a1
clean code
QiJune Mar 7, 2018
f8e6488
merge baidu/develop
QiJune Mar 7, 2018
84829d4
fix bug
QiJune Mar 7, 2018
084484c
merge baidu/develop
QiJune Mar 8, 2018
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
21 changes: 17 additions & 4 deletions python/paddle/fluid/memory_optimization_transpiler.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -118,7 +118,7 @@ def _find_var(self, block_desc, var_name, is_forward):
else:
return block_desc.find_var_recursive(str(var_name))

def memory_optimize(self):
def memory_optimize(self, level=0):
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The code style says that a function name should be a verb-subject phrase, like, optimize_memory, instead of memory_optimize.

Also, it seems that we cannot optimize the memory; what we could is to optimize the usage of the memory.

For this case, does it mean reuse_memory and should we rename level into reuse_tensor_with_the_same_size?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, you are right. It's mainly reuse memory. And level 0 means that we can reuse tensor with the same size. Level 1 means that we can reuse tensor if current tensor size is the same or less than cache pool tensor.

I will refine these codes accordingly. Thanks!

def check_var_validity(block_desc, x, is_forward):
if str(x) == "@EMPTY@":
return False
Expand All @@ -136,6 +136,18 @@ def check_var_validity(block_desc, x, is_forward):
return False
return True

def compare_shape(x_shape, cache_shape, opt_level):
if opt_level == 0:
return x_shape == cache_shape
if opt_level == 1:
if (x_shape[0] == -1) ^ (cache_shape[0] == -1):
return False
x_size = abs(reduce(lambda x, y: x * y, x_shape))
cache_size = abs(reduce(lambda x, y: x * y, cache_shape))
if x_size <= cache_size:
return True
return False

self._build_graph()
self._dataflow_analyze()
self.pool = []
Expand All @@ -160,7 +172,8 @@ def check_var_validity(block_desc, x, is_forward):
for index, cache_pair in enumerate(self.pool):
cache_var = cache_pair[0]
cache_shape = cache_pair[1]
if x_shape == cache_shape:
if compare_shape(x_shape, cache_shape, level):
# if x_shape == cache_shape:
if self._has_var(block_desc, cache_var, is_forward):
x_dtype = self._find_var(block_desc, x,
is_forward).dtype()
Expand Down Expand Up @@ -267,7 +280,7 @@ def get_cfgs(input_program):
return cfgs


def memory_optimize(input_program):
def memory_optimize(input_program, level=0):
cfgs = get_cfgs(input_program)
for cfg in cfgs:
cfg.memory_optimize()
cfg.memory_optimize(level)
6 changes: 3 additions & 3 deletions python/paddle/fluid/tests/book_memory_optimization/test_memopt_image_classification_train.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -124,9 +124,9 @@ def conv_block(input, num_filter, groups, dropouts):

accuracy = fluid.evaluator.Accuracy(input=predict, label=label)

fluid.memory_optimize(fluid.default_main_program())
fluid.memory_optimize(fluid.default_main_program(), level=1)

BATCH_SIZE = 128
BATCH_SIZE = 16
PASS_NUM = 1

# fix the order of training data
Expand Down Expand Up @@ -155,7 +155,7 @@ def conv_block(input, num_filter, groups, dropouts):
pass_acc))
# this model is slow, so if we can train two mini batch, we think it works properly.

if i > 2:
if i > 0:
exit(0)
if math.isnan(float(loss)):
sys.exit("got NaN loss, training failed.")
Expand Down