[Feature] Need an intelligent way of picking batch size.

[adamltyson]

The default batch size for classification (32) was chosen so it would run on pretty much any good GPU but as @larsrollik has found, this can be increased a lot, reducing interference time (128 works with the default resnet on a RTX 2080Ti).

Maybe use tf to get available GPU memory and pick batch size based on that? Allow users to specify max GPU memory to use?

[adamltyson]

@IgorTatarnikov in your move to torch, have you found anything that could help here? Do you have an idea about GPU memory usage as a function of batch size?

[IgorTatarnikov]

I haven't played with batch size during inference. In theory there should be a way to pre-calculate a maximal batch size to utilize as much VRAM as possible during inference. I'd want to do some profiling first though to see how much effect it has before going through the effort of implementing!