Skip to content

Commit

Permalink
Temporary bump the memory buffer size - hopefully fix issues from 483…
Browse files Browse the repository at this point in the history 
…bab2
  • Loading branch information
@ggerganov
ggerganov committed Mar 24, 2023
1 parent f4f5362 commit 31572d9
Showing 1 changed file with 1 addition and 1 deletion.
2 changes: 1 addition & 1 deletion llama.cpp
View file
Original file line number Diff line number Diff line change
Expand Up @@ -632,7 +632,7 @@ static bool llama_eval_internal(
auto & mem_per_token = lctx.mem_per_token;

// TODO: fix this hardcoded size
static size_t buf_size = 512u*1024*1024;
static size_t buf_size = 2048u*1024*1024; // TMP !!!

This comment has been minimized.

Sign in to view

Copy link
@Green-Sky

Green-Sky Mar 24, 2023

Collaborator

i did this btw: https://github.com/Green-Sky/llama.cpp/blob/5dd94f70b2886839579b22e0603ed451f4babc8d/llama.cpp#L635
static void * buf = malloc(buf_size);

if (mem_per_token > 0 && mem_per_token*N > buf_size) {
Expand Down

2 comments on commit 31572d9

@rabidcopy
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I was just about to make an issue about this. It occurred to me no matter what ctx_size I set, I'd get ggml_new_tensor_impl: not enough space in the context's memory pool (needed 536877264, available 536870912) at a set amount of tokens. Didn't realize it was hardcoded and didn't use ctx_size currently.

@Green-Sky
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@rabidcopy yea, i implemented it properly growing here #438

Please sign in to comment.