SystemError: Out of memory error when computing mmd.compute_statistic()

[amir-rahnama]

I am following http://shogun-toolbox.org/notebook/latest/mmd_two_sample_testing.html to run a Kernel MMD test. I am hitting a memory error when calling compute_statistic on my data, since it is a dense vector of (100, 19666):

I am using

# turn data into Shogun representation (columns vectors)
feat_p=sg.RealFeatures(X.reshape(1, 98330000))
feat_q=sg.RealFeatures(Y.reshape(1, 98330000))

error message:

 biased_statistic=mmd.compute_statistic()
SystemError: Out of memory error, tried to allocate 61880248960000 bytes using malloc

I assume that the problem is somewhere else since Shogun should work with high dimensional data, no?

[karlnapf]

It should bit you may have spotted a problem.
@lambday any ideas?
I’ll try to check this within the next days
Thanks for the report

[karlnapf]

Just looking at this again. The line sg.RealFeatures(X.reshape(1, 98330000)) creates 98330000 vectors of dimension 1. Building the kernel matrix for that many data is hopeless. Not sure why you added that, but it should be OK to compute the kernel matrix between 19666 vectors of dimension 100. Could you confirm?

[amir-rahnama]

Yes very true. I got misinformed because in one of your tutorials they were represented as column vectors, so I assumed that's the formation you need to fit them.

Thanks for the help

[karlnapf]

Yes, all shogun vectors are in column major format, so number of rows is the dimension and number of Cols is the number of vectors