Code Questions About the LlamaSharp Project

[n-ate]

I've been digging into LlamaSharp Project and noticed that the Stateless executor declares and destroys an object level member during an asynchronous method call. I believe this has the potential to make it so while one call to InferAsync is still iterating awaiting tokens a second call to InferAsync could dispose the context behind the scenes.

Not understanding the llama.dll well, I don't even know if the llama.dll would throw an error because of some type of internal state problem during concurrent calls so maybe it's a moot point.

I'd suggest that if the context is not meant to exist outside of the infer processing that it be created within the InferAsync method and disposed before it completes. This way each call to InferAsync would create its own context and would not store the context in the object avoiding context change when concurrent calls are running.

[martindevans]

The reason this is exposed is purely for legacy reasons - the ILLamaExecutor interface exposes a Context parameter which is non-nullable, so something has to be put there. This is the same reasons the constructor creates and immediately destroys a context, it must be non-null when the constructor executes and that's the (extremely ugly) workaround.

In my opinion all of the high level executors need a total redesign, they've been around since very early in the start of the project and although a lot of foundational work has been done improving things they've barely changed.

I'd suggest that if the context is not meant to exist outside of the infer processing that it be created within the InferAsync method and disposed before it completes.

The problem here is some more advanced uses do need access to the context, so it can't be totally hidden away as a function local.

[martindevans]

I should mention my main contribution to redesigning the executors has been building the BatchedExecutor - that's a much lower level executor that just handles tokens in and logits out, it can support multiple independent Conversations executing in a batch (i.e. all using the same context) and has various safety checks built in to ensure you're not using it wrong. My goal has always been to build higher level executors wrapping a single Conversation, but I've not really had time to dig into that on top of simply maintaining the project (mostly updating the llama.cpp binaries).