How could I get the Buffer of NdArray returned by numjs?

[huan]

What I want to do, is to save the memory usage.

I have a project that will deal with large photos, like a 4000 x 4000 JPEG file. what I want to do is:

const image = nj.images.read('file-path.jpeg')`
image.flatten() // this will cause MEMORY OUT
const typedData = image.XXX? // HOW TO GET THIS?
const base64Text = new Buffer(typedData).toString('base64')

The current operation is use a image.flatten().tolist() and uses lots of memory, which caused a nodejs OUT OF HEAP MEMORY ERROR. (only add node --max-old-space-size=4096 can make it)

I saw a method of nj.getRawData(), may I use that? And another question: does it the same function like np.getbuffer()? If so, why not the same name.

Thanks!

---

UPDATE: I make a reproduce-able code to demo the problem:

Both flatten() and reshape() will not work with very large NdArray, like [8000, 10000].
But a for-loop with NdArray will be ok.

const a = nj.ones([8000, 10000])
console.log('DONE1', a)

/**
 * The following two functions will cause
 * FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed - JavaScript heap out of memory
 * flatten()
 * reshape()
 */
// a.flatten()
// a.reshape(1, -1)

/**
 * The following raw TypeArray will did the job without problem
 */
const [row, col] = a.shape
const typedData = new Float64Array(row * col)

let n = 0
for (let i = 0; i < row; i++) {
  for (let j = 0; j < col; j++) {
    typedData[n++] = a.get(i, j)
  }
}

[nicolaspanel]

Could you check that dtype is set to uint8 and that the underlying datastructure is an Uint8Array?

[nicolaspanel]

Can you share the 4000x4000 image?

[huan]

Yes I had confirmed, it's unit 8.

To reproduce you can just run my posted script, nj.ones can do the same.(memory out)

[nicolaspanel]

As you say, the problem comes from flatten + tolist that makes deep copies of the original array.

To access the raw data of the image, you can do this:

var nj = require('numjs')
var img = nj.images.data.lena;
img.shape // => [ 512, 512, 3 ]
img.dtype // => uint8
img.selection.data // Uint8Array 

see https://github.com/nicolaspanel/numjs/blob/master/src/images/read.js#L8-L26 and https://github.com/nicolaspanel/numjs/blob/master/src/images/read-dom.js#L20-L52 for more info

[huan]

Thank you very much!