move to predict_on_batch

[jeromelecoq]

The goal of this PR is to move to using predict_on_batch for inference since that should be both memory and computation efficient.

[codecov-commenter]

Codecov Report

Merging #78 (57ca75a) into master (8a7834c) will increase coverage by 0.12%.
The diff coverage is 100.00%.

@@            Coverage Diff             @@
##           master      #78      +/-   ##
==========================================
+ Coverage   51.49%   51.61%   +0.12%     
==========================================
  Files          11       11              
  Lines        1610     1614       +4     
==========================================
+ Hits          829      833       +4     
  Misses        781      781              

Impacted Files	Coverage Δ
deepinterpolation/__init__.py	100.00% <100.00%> (ø)
deepinterpolation/cli/schemas.py	93.91% <100.00%> (+0.16%)	⬆️
deepinterpolation/inferrence_collection.py	65.07% <100.00%> (ø)
deepinterpolation/trainor_collection.py	75.00% <0.00%> (ø)

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 8a7834c...57ca75a. Read the comment docs.

[jeromelecoq]

@danielsf @aamster any concerns with this simple PR?

[aamster]

@jeromelecoq I am not sure I understand why this fixes the issue. local_data[0] is a minibatch of data of type Sequence, correct? So predict will iterate through the dataset, but predict_on_batch will just predict on the entire minibatch without iterating. It seems like predict might be slower if the input is larger, but shouldn't use more memory.

The keras docs say to call the model directly for small inputs, ie self.model(local_data[0], training=False). Does that also fix the issue?

[jeromelecoq]

The issue is I think due to a memory leak. I found that looping with predict and GPUs will try to create huge arrays on the GPU. see here: #77

[jeromelecoq]

It also seemed like this was the expected use in our case, since I am actually feeding one batch of data as described here : https://stackoverflow.com/questions/44972565/what-is-the-difference-between-the-predict-and-predict-on-batch-methods-of-a-ker

[aamster]

@jeromelecoq I am still confused as to why predict is running into a memory issue if local_data[0] is only a single minibatch, since, as the docs say, the default batch_size on predict is 32, and I believe local_data[0] has a batch size of 5, correct? In which case it should just take the entire input and do a forward pass on it, which is the same as predict_on_batch.

Are you able to try using self.model(local_data[0], training=False) which the docs suggest for small inputs? If not then it is fine.

[jeromelecoq]

@jeromelecoq I am still confused as to why predict is running into a memory issue if local_data[0] is only a single minibatch, since, as the docs say, the default batch_size on predict is 32, and I believe local_data[0] has a batch size of 5, correct? In which case it should just take the entire input and do a forward pass on it, which is the same as predict_on_batch.

Yes batch size is usually set to 5 but in this case it is set by the generator so is user-defined. I am with you on not knowing why predict gives a GPU memory issue given the doc. But I can read from the doc that looping predict 1000 times might not be what they are testing for since predict is designed to break down batches internally. In addition, the array allocation request in the error is completely unexpected given the network architecture.

Are you able to try using self.model(local_data[0], training=False) which the docs suggest for small inputs? If not then it is fine.

I can try that, why do you think that would be superior to .predict_on_batch? Can you point me at the doc where this is described?
Thanks!

[aamster]

Here it says

For small amount of inputs that fit in one batch, directly using call() is recommended for faster execution, e.g., model(x), or model(x, training=False) if you have layers such as tf.keras.layers.BatchNormalization that behaves differently during inference.

I am just worried that they don't mention predict_on_batch in their docs here, though it is probably fine.

[jeromelecoq]

Ah I see, I am also confused now why do they have predict_on_batch and call options?