Support CUDA stream with stream memory pool

[sonots]

Fix #225

Support CUDA stream. stream can be specified with with statement or use() method as:

import cupy as cp

x_gpu = cp.array([1, 2, 3])

with cp.cuda.stream.Stream():
   y_gpu = cp.linalg.norm(x_gpu)

stream = cp.cuda.stream.Stream()
stream.use()
y_gpu = cp.linalg.norm(x_gpu)

nvprof --print-gpu-trace python ~/test_cupy.py shows cupy kernels are executed in another stream.

To support CUDA stream, this PR changes memory pool, too.
A memory pool is created for each stream separately so that parallel computations among streams do not touch memory used in another stream.

Note that cudaMalloc is issued on default stream always.

Incompatibility changes:

• cupy.cuda.generator.RandomState: setStream() method was removed
• cupy.cuda.stream.Stream(null=True) is prohibited to assure cupy.cuda.stream.Stream.null object is always used to specify the default stream.

[sonots]

Added implementation to support stream on all CUDA libraries, cudnn, cublas, curand, cusparse, cusolver, and thrust.

[sonots]

Because it is difficult to test whether CUDA is really using stream, I added examples and verified by running them with nvprof --print-gpu-trace.

[sonots]

Wrote about incompatibility changes #306 (comment)

[sonots]

I am now thinking to add an interface to free_all_blocks of an arena for a given stream. <= done

[sonots]

Jenkins fails cyfunction is not a Python function with cd docs && make html as:

/work/cupy/docs/source/reference/generated/cupy.cuda.Event.rst:53: WARNING: error while formatting arguments for cupy.cuda.Event.record: <cyfunction Event.record at 0x2aea1bcb8d10> is not a Python function

I am struggling to resolve this, but I have no clue how to resolve this issue.
I found pandas has same issue pandas-dev/pandas#5218, but it seems they have also no idea.

[sonots]

Hmm, it seems make html succeeds with python 3.6.2, but fails with python 2.7.13.

[sonots]

It seems insepect.isfunction(cyfuction) returns True from python 3.4 https://groups.google.com/forum/#!topic/cython-users/FvcMPk9n2X8. Can we use python >= 3.4 to generate our sphinx docs in chainer-test?

[sonots]

#306 (comment)

I've debugged sphinx. sphinx has config.suprress_warnings http://www.sphinx-doc.org/ja/stable/config.html, but cyfunction is not a Python function was not supported.

So, another way (one way is to use python >=3.4 ) to avoid is not to treat warnings as error with a change:

Makefile

-  6 SPHINXBUILD   = sphinx-build -W
+   6 SPHINXBUILD   = sphinx-build

---

This is another problem, but I noticed that number of doctests were increased as:

sphinx-build -W

Doctest summary
===============
   60 tests
    0 failures in tests
    0 failures in setup code
    0 failures in cleanup code

sphinx-build

Doctest summary
===============
   95 tests
    0 failures in tests
    0 failures in setup code
    0 failures in cleanup code

[sonots]

Memory.__init__ also get raised TypeError, but it seems it is not included in the warnings. It seems that it was reason why doctest did not fail before... 🤔

EDIT: hmm, looks __init__ are ignored.

('util/inspect.py:115', <cyfunction Memory.__init__ at 0x7f744a3cab90>)
('util/inspect.py:115', <cyfunction Stream.__init__ at 0x7f744afb6ad0>)
('util/inspect.py:115', <cyfunction Event.__init__ at 0x7f744afb6710>)
('util/inspect.py:115', <cyfunction ReductionKernel.__init__ at 0x7f744a3f64d0>)
('util/inspect.py:115', <cyfunction Event.__init__ at 0x7f744afb6710>)
('util/inspect.py:115', <cyfunction Event.record at 0x7f744afb6950>)
('ext/autodoc.py:751', <type 'exceptions.TypeError'>, TypeError('<cyfunction Event.record at 0x7f744afb6950> is not a Python function',))
('util/inspect.py:115', <cyfunction Event.synchronize at 0x7f744afb6a10>)
('ext/autodoc.py:751', <type 'exceptions.TypeError'>, TypeError('<cyfunction Event.synchronize at 0x7f744afb6a10> is not a Python function',))
('util/inspect.py:115', <cyfunction Memory.__init__ at 0x7f744a3cab90>)
('util/inspect.py:115', <cyfunction Stream.__init__ at 0x7f744afb6ad0>)
('util/inspect.py:115', <cyfunction Stream.__enter__ at 0x7f744afb6c50>)
('ext/autodoc.py:751', <type 'exceptions.TypeError'>, TypeError('<cyfunction Stream.__enter__ at 0x7f744afb6c50> is not a Python function',))
('util/inspect.py:115', <cyfunction Stream.__exit__ at 0x7f744afb6d10>)
('ext/autodoc.py:751', <type 'exceptions.TypeError'>, TypeError('<cyfunction Stream.__exit__ at 0x7f744afb6d10> is not a Python function',))
('util/inspect.py:115', <cyfunction Stream.add_callback at 0x7f744a3ca050>)
('ext/autodoc.py:751', <type 'exceptions.TypeError'>, TypeError('<cyfunction Stream.add_callback at 0x7f744a3ca050> is not a Python function',))
('util/inspect.py:115', <cyfunction Stream.record at 0x7f744a3ca110>)
('ext/autodoc.py:751', <type 'exceptions.TypeError'>, TypeError('<cyfunction Stream.record at 0x7f744a3ca110> is not a Python function',))
('util/inspect.py:115', <cyfunction Stream.synchronize at 0x7f744afb6f50>)
('ext/autodoc.py:751', <type 'exceptions.TypeError'>, TypeError('<cyfunction Stream.synchronize at 0x7f744afb6f50> is not a Python function',))
('util/inspect.py:115', <cyfunction Stream.use at 0x7f744afb6dd0>)
('ext/autodoc.py:751', <type 'exceptions.TypeError'>, TypeError('<cyfunction Stream.use at 0x7f744afb6dd0> is not a Python function',))
('util/inspect.py:115', <cyfunction Stream.wait_event at 0x7f744a3ca1d0>)
('ext/autodoc.py:751', <type 'exceptions.TypeError'>, TypeError('<cyfunction Stream.wait_event at 0x7f744a3ca1d0> is not a Python function',))
('util/inspect.py:115', <cyfunction ufunc.__init__ at 0x7f744a3e1b90>)
('util/inspect.py:115', <cyfunction ReductionKernel.__init__ at 0x7f744a3f64d0>)
('util/inspect.py:115', <cyfunction ufunc.__init__ at 0x7f744a3e1b90>)

[sonots]

jenkins, test this please

[okuta]

LGTM!

[sunbearc22]

Can you provide a simple example of how to use Nvidia Visual profiler nvvp to profile a python-cupy script? Thanks.