You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Also we implemented custom abstract_init and load_on_cpu args that should be pushed to HuggingFace transformers for better support.
The text was updated successfully, but these errors were encountered:
borisdayma
changed the title
load directly state in sharded mode
cleanup loading method and set directly state in sharded mode
Feb 10, 2022
borisdayma
changed the title
cleanup loading method and set directly state in sharded mode
improve model/optimizer loading and set directly state in sharded mode
Feb 10, 2022
In the training script, we do:
params
on CPUopt_state
on CPU (if we restore them)state
on correct devicesIdeally we wouldn't need to go through CPU first.
Also we implemented custom
abstract_init
andload_on_cpu
args that should be pushed to HuggingFacetransformers
for better support.The text was updated successfully, but these errors were encountered: