Session crashed on colab

[bitsnaps]

Hi,

Have you guys managed to make it works on T4 colab?

P.S. It crashes multiple times even with offload_per_layer = 5 as mentioned in the comment.

[dvmazur]

Hey! Just tried running the notebook (in the offload_per_layer = 5 setting) and everything works for me. Have you tinkered with the original notebook in any way? If not, try restarting the session and running it again starting with the model initialization cell.

[lavawolfiee]

Hey!

Have you managed to solve this issue? If no, can you please provide some more information:

1. On which cell exactly did the notebook crush with offload_per_layer = 4?
2. How many of RAM and GPU VRAM did you have in Google Colab?

Note that our demo notebook should run normally in Google Colab with offload_per_layer = 4, but will crush sometimes with offload_per_layer = 5. The latter option is made for local run with low VRAM.

[RDvibe]

hqq_aten package not installed. HQQBackend.ATEN backend will not work unless you install the hqq_aten lib in hqq/kernels.

[dvmazur]

hqq_aten package not installed. HQQBackend.ATEN backend will not work unless you install the hqq_aten lib in hqq/kernels.

Hqq_aten is not required as we have custom triton kernels for GEMV.