Join GitHub today
GitHub is home to over 31 million developers working together to host and review code, manage projects, and build software together.Sign up
[Experiment] Squeeze and Excitation #683
Trying out Squeeze and Excitation.
Looks really good.
Code: #673 (Brian would be nice for you to take a look)
Trained 6 networks, 2 each of baseline, Squeeze And Excitation (SE) and SE + bias
tensorflow code was very slow in inference. It was mentioned that averagepool is slower than reduce_average, I investigated but it appeared and disappeared unclear why.
Some links I used to profile performance
chrome tracing from:
(was .json renamed for git)timeline_01.txt
@sethtroisi From a quick survey of the paper, the Squeeze-Excite (SE) approach looks very similar to what @lightvector has been doing with global properties. The main difference seems to be that SE only considers average pooling (though they suggest other aggregations), while the later suggests that max pooling might also be useful.
See https://github.com/lightvector/GoNN#update-oct-2018 for some further reading of his research into the topic. He also discuss a bunch of other topics you might find inspiring for similar enhancements.