convolutional network memory usage

[segalinc]

Hi !
I am new in tensorflow and I am following this nice tutorial.
I am trying to run the convolutional network tutorial on my GTX 1080 4 Gb but seems I have memory problem. I already set up the configuration to allow memory growth and per process memory fraction but I still find out of memory problem. I tried to reduce the batch size and run it on python instead of the notebook.
So my question is.. it my GPU too small to be able to run these few layers of CNN?