【help】why function llama_build_graph is internal function llama_decode？

[xjj210130]

I read the llama.cpp source code。
I am confused as to why the function llama_build_graph needs to be called every time the function llama_decode is called.
The function llama_build_graph cannot be called during program initialization, which will reduce the inference time.

static int llama_decode_internal(
llama_context & lctx,
llama_batch batch) {
....
ggml_cgraph * gf = llama_build_graph(lctx, batch, false);
.....
}

Thanks

[ggerganov]

In general, the graph is not static - the shapes of the tensors change. There are tricks that can be done to avoid repetitive graph builds, but these are not general enough to put in the library and depend on the specific use cases

[github-actions]

This issue was closed because it has been inactive for 14 days since being marked as stale.