All devices are equal!

[Liamdoult]

This PR aims to resolve 4 issues:

1. No tests for ANE
2. Device to device conversion
3. Failure to run GPU Tests
4. Failing GPU tests

Sorry, but this will be long.

1. No tests for ANE

Test calls have been added for the ANE device which will envoke ANE tensors when running tests. Same as GPU

@unittest.skipUnless(ANE, "Requires ANE")
class TestNNANE(TestNN):
  device=DeviceTypes.ANE

2. Device to device conversion

To be able to support ANE in tests, it is required that ANE support the same conversion as CPU->GPU and GPU->CPU.
I have created a universal any to any converter. This function accepts any of the accepted datatypes (GPUBuffer, ane.tensor and any np.ndarray compatible data) and returns the converted data.

@staticmethod
def data_to_device(data, device=DeviceTypes.CPU):

3. Failure to run GPU Tests

CI tests have not been run since 1a1c63a. This appears to have been caused by a missing six dependency that was thrown inside the try except for importing pyopencl, therefore defaulting GPU=False

4. Failing GPU tests

GPU tests are failing. Don't think I caused it as it is accuracy asserts being raised, but not 100%:
i.e.

>         np.testing.assert_allclose(t.grad, tt.grad.cpu().data, atol=grad_atol, rtol=grad_rtol)
E         AssertionError: 
E         Not equal to tolerance rtol=1e-06, atol=1e-06
E         
E         Mismatched elements: 2925 / 2925 (100%)

Potentially a change since OpenCL tests last passed.

[Liamdoult]

Before ready to review:

• Passing GPU tests
• Clean code
• GPU, CPU copy profiling still works
• ANE Profiling added
• Reduce lines

[geohot]

Very nice, much needed. Let's clean it up

[Liamdoult]

Important API change:

.cuda -> .gpu
.cuda_ -> .gpu_
.gpu -> .is_gpu

This now matches Device.GPU naming.

I do recommend that in the future this is changed to ocl and OCL for opencl as GPU can fallback to CPU using the opencl standard.

Tensor now also supports:

.cpu
.cpu_
.is_cpu
.ane
.ane_
.is_ane

[Liamdoult]

@geohot Can't test ANE personally.

[geohot]

Congrats, nice PR!