Is there a way to prevent reloading parameters?

[Penguin-jpg]

Hello! I have been using CodeGen to generate lately. But I found that I spent most of the time on loading parameters.
I tried to separate out the create_model part to prevent it from reloading but cuda out of memory occurred.
So I want to know is there a way to prevent reloading parameters?
Thanks!

[enijkamp]

https://colab.research.google.com/drive/12LF5GlUmvchpx7niMiUD737q7xvoTzPr?usp=sharing

If you execute the last cell, it appears the amount of allocated cuda memory remains constant.

Does this help?

[Penguin-jpg]

https://colab.research.google.com/drive/12LF5GlUmvchpx7niMiUD737q7xvoTzPr?usp=sharing

If you execute the last cell, it appears the amount of allocated cuda memory remains constant.

Does this help?

Thanks for reply!! It works 0.0.
I think I probably missed some lines from source code. Thanks again!

[enijkamp]

Glad it helps. Perhaps, you could update your shared notebook, so that users can easily try multiple samples?

[Penguin-jpg]

Glad it helps. Perhaps, you could update your shared notebook, so that users can easily try multiple samples?

Updated!!

But I found that changing model will cause a TypeError.

I used 350M-mono to generate first and it works, then I changed to 2B-mono and reload the model. But when generating, it gave a TypeError.

[rooa]

Thanks for reporting this! I was able to reproduce the issue. This was because reloading of the tokenizer caused hiccups with max_model_input_sizes when tokenizing with truncation=True. I have modified the tokenizer call to explicitly use the specified max_length in this commit: 22b81d6.

Could you reset the runtime and try again?

[enijkamp]

There was a minor typo in the 350M-multi url. Here simplified code. Could you test and revise your shared notebook? Thank you.

https://colab.research.google.com/drive/12LF5GlUmvchpx7niMiUD737q7xvoTzPr?usp=sharing

[Penguin-jpg]

@enijkamp @rooa Problems solved. Thanks for your help!! The notebook is updated.
Here is the link: https://colab.research.google.com/drive/1fQI8OgzMAR0bquCrvhlAtXSw6iMFbVgI?usp=sharing

[enijkamp]

Great, thanks!