Join GitHub today
GitHub is home to over 40 million developers working together to host and review code, manage projects, and build software together.Sign up
Performance of Simple Use Case - PyTorch vs Plain Python with Numpy #1630
Summary - PyTorch seems to be half as fast as plain Python for very simple networks.
I am learning PyTorch because it promises convenience and easy access to GPU acceleration.
In learning, I re-implemented my previous very simple tutorial 3-layer neural network created from scratch in Python with Numpy.
The two are here:
They are built to be as similar as possible - same size, same loss function, same training data, etc
The result of 2 timing tests show that the plain python version is about twice as fast as the PyTorch version:
Here's a blog which details the experiment conditions in more detail (near the bottom of the post).
What Am I Doing Wrong?
Or does PyTorch have too much overhead for simple scenarios, and only really shines for larger or GPU accelerated cases?
Right now our GPU libs are very optimized, but we can't say the same about the CPU backend. It's getting better, but we still aren't there where numpy is, and this is probably why are you seeing this difference. Can you try running your script with
@apaszke thanks for your helpful reply
My MacBook Pro 13 early 2015 has only 1 cpu with 2 cores. I therefore set
The results were:
I did this the comparison again as I am now running a browser showing live tv news, so need to make the environment equivalent to aid comparison. The ratio is 1.6 not in favour of PyTorch.
Interestingly, setting the
The environment variable was set in the notebook using:
and was tested with
Sorry, this was my mistake. I've tested your code with smaller datasets and PyTorch implementation was 2 times faster
The readme.md on pytorch's GitHub page says:
But the above comments suggest this isn't true.
I am now setting up a Google Cloud Compute GPU instance to test performance with CUDA. Weill update this issue if interesting.
I've updated my blog .. the GPU results are a bit faster .. about 25%.
I will do 2 more tests .. one using the environment variable ... and one scaling the hidden layer to about 1000, to see if the relative performance improves.
Update - I did the tests comparing PyTorch in CPU vs GPU mode. The results are interesting and positive.
In GPU mode, the duration to complete MNIST training rises slowly, but in CPU mode it grows quickly, as the size of the network grows (hidden nodes).
More detail at the bottom of this post: http://makeyourownneuralnetwork.blogspot.co.uk/2017/05/learning-mnist-with-gpu-acceleration.html
So this is good. I think PyTorch is nice to work with .. I understand that the tutorials and beginner-friendly guides still need people to work on them... would be great when they're out. Also I will do an intro talk at the PyData monthly meetup in London on my experience beginning PyTorch.
Also the environment variable didn't seem to have an effect during my comparative test ?! @apaszke
Now if only it was faster than numpy .....