torch.cuda.OutOfMemoryError: CUDA out of memory.

[8188]

I use 7b-model with 3090(24G), When loading, it cost 15G Memory, It's OK.
When referencing, indeed it is a nice model, but it eat too much Memory as metadata.sql increasing a bit.
Only a 4k metadata.sql probably causes CUDA out of memory. Any good idea to solve it?

[github-actions]

Stale issue message

[deqncho2]

I'm interested in the same question. How do we feed in large schemas (I'm talking > 10k) without running out of memory on inference?

[github-actions]

This issue is stale because it has been open for 30 days with no activity.

[fire717]

Same question here.

[dqatsh]

try

load_in_4bit=True
&
num_beams=3

it works for me with 4090(24G)

[github-actions]

This issue is stale because it has been open for 30 days with no activity.

[github-actions]

This issue was closed because it has been inactive for 14 days since being marked as stale.