Memory usage in numerical hessian

[davidsebfischer]

I have a peak memory load of 45GB on 500 cells, I believe this is during hessian estimation, on a test data set. That should not be the case I think, it s only 10 or so parameters? We should go over step by step whether really only the relevant parts of the overall hessian are computed.

[Hoeze]

A possible point to improve could be to use the new map-reduce operations:
#1

I'll look at that problem again as soon as my thesis is over / extended.

[davidsebfischer]

@Hoeze I am checking this now, I think there is a some other problem here, too.

[davidsebfischer]

The memory load goes up here in diffxpy/testing/base.py:_fit()
model = estim.finalize()

[Hoeze]

Yes, this is due to calculating the Hessian for sure.
I did not expect that this would be such an issue...

[LuckyMD]

Maybe this will improve if a sparse matrix representation is used?

[LuckyMD]

I have a peak memory load of 45GB on 500 cells, I believe this is during hessian estimation, on a test data set. That should not be the case I think, it s only 10 or so parameters? We should go over step by step whether really only the relevant parts of the overall hessian are computed.

Do you get this memory load also when running on a single core? I assumed it might have something to do with private copies of the data in each thread.

[davidsebfischer]

Yes, also on a single core. I also don't think that linear algebra multi-threading, ie. distributing matrix multiplications, should cause memory copying, but that is just a guess.

[davidsebfischer]

/Users/david.fischer/miniconda3/lib/python3.6/site-packages/tensorflow/python/ops/gradients_impl.py:108: UserWarning: Converting sparse IndexedSlices to a dense Tensor of unknown shape. This may consume a large amount of memory.
"Converting sparse IndexedSlices to a dense Tensor of unknown shape. "

is thrown during
hess = tf.hessians(- model.log_likelihood, params)
that might be related to this.

[davidsebfischer]

So I guess IndexedSlices are used in tf.map_fn to map the hessian function across the slices of X and param. I do not see why anything here would be sparse though if X is dense in the input. Where is sparsity of X handled in batchglm @Hoeze?

[Hoeze]

I think this is internally used by tf.hessians.
Tensorflow uses basically a tf.while_loop to differentiate each value of the gradient by a single parameter.
Probably these indexedslices will be used to generate the resulting hessian matrix.

[davidsebfischer]

Accidently posted the the resolving comment in a related issue:
#27 (comment)