Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Reduce op compile slowly #10874

Closed
wanghaoshuang opened this issue May 23, 2018 · 3 comments · Fixed by #11029 or #11113
Closed

Reduce op compile slowly #10874

wanghaoshuang opened this issue May 23, 2018 · 3 comments · Fixed by #11029 or #11113
Assignees

Comments

@wanghaoshuang
Copy link
Contributor

wanghaoshuang commented May 23, 2018

为了支持对多个dims做reduce, reduce op 使用了大量的eigen template function, 因此导致reduce op编译变慢。

  1. 需要讨论下是否可以减少reduce支持的dims?
  2. 是否有更优的替代方案?
CXX_FLAGS="-m64 -std=c++11 -O3 -g -DNDEBUG --expt-relaxed-constexpr"

CXX_DEFINES="-DANY_IMPL_ANY_CAST_MOVEABLE -DPADDLE_DISABLE_PROFILER -DPADDLE_DISABLE_RDMA -DPADDLE_DISABLE_TIMER -DPADDLE_USE_DSO -DPADDLE_USE_PTHREAD_BARRIER -DPADDLE_USE_PTHREAD_SPINLOCK -DPADDLE_VERSION=0.12.0 -DPADDLE_WITHOUT_GOLANG -DPADDLE_WITH_CUDA -DPADDLE_WITH_CUPTI"

CXX_INCLUDES="..."

time nvcc /home/disk1/wanghaoshuang/paddle/debug/paddle/fluid/operators/reduce_op.cu -c -o /home/disk1/wanghaoshuang/paddle/debug/build/paddle/fluid/operators/CMakeFiles/reduce_op.dir/reduce_op_generated_reduce_op.cu.o -ccbin /home/work/.jumbo/opt/gcc48/bin/g++ $CXX_FLAGS $CXX_DEFINES $CXX_INCLUDES

用上述命令,编译时间如下:

real	5m49.639s
user	5m38.698s
sys	0m11.201s

删去L138-L146后编译时间如下:

real	2m23.974s
user	2m16.771s
sys	0m7.076s

删去L138-L146L290-L293后,编译时间如下:

real	0m39.780s
user	0m35.981s
sys	0m3.745s
@wanghaoshuang
Copy link
Contributor Author

reduce_op现在编译特别慢,GPU模式下等了快5分钟才编译完这个op。请问是不是138行-153行支持的维度太多了呢?能否把维度5和6删去,这样就从15种减少到6种了。

https://github.com/PaddlePaddle/Paddle/pull/10708/files#r190174087

@wanghaoshuang
Copy link
Contributor Author

#5491

@luotao1
Copy link
Contributor

luotao1 commented May 28, 2018

related #9306

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
5 participants