-
Notifications
You must be signed in to change notification settings - Fork 9
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Preallocation in activations #25
Comments
I really like this, there are lots of places we can be doing things in place and save some GC time. This, and the sparse kernels, are also a really good place to play around with mulithreading the for loops. I'll try to whip up an example before the call today, but I tested it quickly yesterday and got great results |
Multithreading example:
Before starting Julia you need to give it more threads like
|
sorry, I only saw this now. It looks really promising, I agree! Question: How stable are the threads? I remember this to being an experimental feature. As long as this works in a stable way also for more complicated functions, I'm fine using it. |
I have the same question/concern as you. I've only used it before for simple loops, so we will need to do some testing to see how it handles more complicated loops. It also seems to be a little finicky about your hardware. I've had no problems on Macs, but I had some problems yesterday on my Ubuntu machine |
another question is how Threads performs when its being used at different layers. Say, threading over all the examples in the forward propagation and then also using threading in the activation might cause problems, right? |
Yes you need to be careful how you layer them. If you end up using more than you have they block each other and the program grinds to a halt. The same is also true for making sure that if you're making BLAS calls inside the loop, that the number of BLAS threads times the number of Julia threads doesn't exceed how many threads you have. |
On a similar note, couldn't we also create in place https://github.com/XtractOpen/Meganet.jl/blob/master/src/layers/singleLayer.jl#L37 |
Maybe generate apply!
… On Feb 8, 2018, at 5:44 PM, Keegan Lensink ***@***.***> wrote:
On a similar note, couldn't we also create in place apply functions for the normalization layers? I tried adding this and the derivative tests failed, so I assume it isn't always going to be the case. But in some places, like the example below, it seems like it shouldn't be an issue because the data is being overwritten anyways.
https://github.com/XtractOpen/Meganet.jl/blob/master/src/layers/singleLayer.jl#L37
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub, or mute the thread.
|
I think an inplace apply function would be good to have for other Meganet elements as well. In particular, I'm talking about those that do not change the dimension of the input features. Why don't we rename the current method to I'm not sure if this works, but what we could also be doing is that |
Hi Guys, I wasn't aware of the conversations here, but here is a suggestion for you to check:
The .= operator does stuff in place (no allocation), and internally it uses vectorization and some multi-threading. I think that this is the best way to go - I had a really good experience with it so far. I only found out about that recently. If you wish to work with Threads, that's an option, but as far as my experience goes - they are far less efficient than openMP threads in C. I still think that Julia workers is the best way (maybe not use all possible workers and leave some for automatic internal multithreading in BLAS and stuff). I can explain this further on Skype if you wish. |
Maybe we should write two versions of activation functions (and other ones). One that does allocation and one that operates in place. See the following example that Eran and I have put together.
The text was updated successfully, but these errors were encountered: