You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Following the code in README.md or example_inference.py to perform inference by calling model.preprocess_inputs(…) followed by model.generate(…) produces good results the first time the pair is called, but poor results for subsequent pairs of calls.
The reason is that model = Magma.from_checkpoint(…) loads the model with inconsistent training/eval settings. model.training is True but model.image_prefix.enc.training is False. The first call to model.preprocess_inputs(…) works correctly as the image encoder has training False and so its Batch Normalisation steps work correctly. The call to model.generate(…) records the training state on entry and restores it on exit, which because model.training is True puts the whole model into training state. Subsequent calls to model.preprocess_inputs(…) then don't perform Batch Normalisation steps correctly.
Following the code in README.md or example_inference.py to perform inference by calling
model.preprocess_inputs(…)
followed bymodel.generate(…)
produces good results the first time the pair is called, but poor results for subsequent pairs of calls.The reason is that
model = Magma.from_checkpoint(…)
loads the model with inconsistent training/eval settings.model.training
is True butmodel.image_prefix.enc.training
is False. The first call tomodel.preprocess_inputs(…)
works correctly as the image encoder hastraining
False and so its Batch Normalisation steps work correctly. The call tomodel.generate(…)
records the training state on entry and restores it on exit, which becausemodel.training
is True puts the whole model into training state. Subsequent calls tomodel.preprocess_inputs(…)
then don't perform Batch Normalisation steps correctly.The play space at https://huggingface.co/spaces/EleutherAI/magma has this problem too.
The fix is to add
model.eval()
aftermodel = Magma.from_checkpoint(…)
, setting the whole model to a consistent eval state.The text was updated successfully, but these errors were encountered: