Extending context size

[almugabo]

this is more a question than an issue.
Can one extend the context size of the model ?
I am asking because I would like to test finetuning it to longer context to see how far one can get in terms of context size with constrained resources (RTX 4090)

[OmkarThawakar]

Hi @almugabo
Thanks for interest in our work.

Yes you can increase the context size of our model by increasing the hidden_size in config file.

Thanks,