AttributeError: 'NoneType' object has no attribute 'llama_batch_free'

[Fahmie23]

This is my code

This is the error that occur when i try to execute the file.

[hudson-ai]

I've noticed this as well. It seems to be an issue with how certain objects are garbage collected at system. I think it happens when llamacpp gets garbage collected before the actual model does -- does adding del model to the end of your file remove the error? (not a real fix, just a question whose answer may help debug this)

[paulbkoch]

I think the underlying issue is this line in llama_cpp:

https://github.com/abetlen/llama-cpp-python/blob/08e910f7a7e3657cf210ab8633a6994e1bde7164/llama_cpp/llama_cpp.py#L121

Executed in:

https://github.com/abetlen/llama-cpp-python/blob/08e910f7a7e3657cf210ab8633a6994e1bde7164/llama_cpp/llama_cpp.py#L1684-L1687

Python gives no guarantees on process shutdown regarding the order that objects are destructed. When it destructs objects, it changes any existing references to be 'None'. So, if the lib object above happens to be destructed first, then the call to getattr will pass in None for the lib parameter. The error message you get in a call to getattr(None, "llama_batch_free") matches the error observed in this issue: AttributeError: 'NoneType' object has no attribute 'llama_batch_free'.

I haven't verified this, but I will submit a PR to llama-cpp if it checks out.

[paulbkoch]

@Fahmie23, what version do you get when you do:

import llama_cpp
print(llama_cpp.__version__)

I know I've seen an exception similar to what you reported on an older version of llama_cpp, but I'm not getting it to repo now on newer versions. I would expect this error to occur fairly randomly though, so I can't be sure if that's because it's been fixed or if I'm just not happening to see it due to randomness.

[paulbkoch]

I believe PR #734 should resolve this issue, but I wasn't getting the error in the first place, so I can't confirm. Closing this issue for now, but if anyone continues to see it in the next release, please re-open this issue.