-
Notifications
You must be signed in to change notification settings - Fork 9.2k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
python bindings? #82
Comments
Python Bindings for llama.cpp: https://pypi.org/project/llamacpp/0.1.3/ (not mine, just found them) |
As a temporary work-around before an "official" binding is available, I've written a quick script to call the llama.cpp executable that supports streaming and interactive mode: https://github.com/shaunabanana/llama.py |
looks promising on description, will try try and feedback |
sweet will give it a shot
…On Wed, Mar 15, 2023 at 9:21 AM, aratic < ***@***.*** > wrote:
>
>
> Python Bindings for llama.cpp: https:/ / pypi. org/ project/ llamacpp/ 0. 1.
> 3/ ( https://pypi.org/project/llamacpp/0.1.3/ ) (not mine, just found them)
>
>
>
looks promising on description, will try try and feedback
—
Reply to this email directly, view it on GitHub (
#82 (comment) )
, or unsubscribe (
https://github.com/notifications/unsubscribe-auth/AXMEJMCMQRUJKKEAIAHOJXLW4HT77ANCNFSM6AAAAAAVYVLGQQ
).
You are receiving this because you authored the thread. Message ID: <ggerganov/llama
. cpp/issues/82/1470344972 @ github. com>
|
I hacked something together tonight on this. Its python-cpp bindings for the model directly (allowing you to call Merging it would however require splitting some parts of the code out of main.cpp which @ggerganov has argued against IIRc. |
@seemanne u did what I want, dude, easier to expose or integrate with web or chat, appreciate and will give it a try |
Ok I updated this and put it into a proper fork. You can now pass parameters in python. I will need to do some refactoring to pull upstream changes each time but it should work and i tested it on linux and mac. |
I wrote my own ctypes bindings and wrapped it an a KoboldAI compatible REST API. |
EDIT: I've adapted the single-file bindings into a pip-installable package (will build llama.cpp on install) called llama-cpp-python If anyone's just looking for python bindings I put together To use it you have to first build On Linux for example, to build the shared library, update the libllama.so: llama.o ggml.o
$(CXX) $(CXXFLAGS) -shared -fPIC -o libllama.so llama.o ggml.o $(LDFLAGS) Then run |
We are putting together a Huggingface-like library with python interface that auto-downloads pre-compressed models at https://github.com/NolanoOrg/cformers/#usage |
I also found these bindings https://github.com/PotatoSpudowski/fastLLaMa Some feature suggestions, mostly about low level capabilities:
|
Having issues with both variants on a M1 Mac: zsh: illegal hardware instruction The python bindings approach (after building the shared library) produces: |
We have added most of these suggestion in the latest fastLLaMa update 👀 |
As #1156 is closed as a duplicate of this issue, I am bringing the discussion here about the creation of an official python binding in the llama.cpp repository (which I now assume the the objective of this issue.) The current external python bindings seem to be:
But none really stand out as a candidate to be merged into llama.cpp. My proposal is to model the llama.cpp bindings after rwkv.cpp by @saharNooby (bert.cpp also follows a similar path).
We could keep the following in mind for the basic binding:
Any suggestion on which of the current external python bindings could be considered a good start for eventual merge into llama.cpp? |
Hey @dmahurin w.r.t your proposal I should point out that what you describe is the current state of llama-cpp-python
That being said I don't have anything against moving these bindings to llama.cpp if that's something the maintainers think is worthwhile / the right approach. I would also be happy to transfer over the PyPI package as long as we don't break downstream users (text-generation-webui, langchain, babyagi, etc). |
@dmahurin I don't see how merging python bindings into this repo is needed when solutions like the repo of @abetlen exist already. |
Hi @seemanne, the purpose is not to replace bash. The purpose is to widen the development community. Like it or not, Python is a very common language in AI development. I do not think having supported python code would put any burden on cpp developers. Again, reference rwkv.cpp and bert.cpp. The python support in rwkv.cpp for example comes in the form of two python files. As mentioned, there are 5 independent python bindings for llama.cpp. Unifying at least the base python binding would help to focus related python llama.cpp development. |
@abetlen, perhaps you saw that I created pull request #1660 to add low level python bindings from llama-cpp-python. The PR puts llama_cpp.py and low level examples in the examples/ folder. There was a bit of filtering and some squashing to get clean history for the low level commits. For now I excluded the multi char change, mainly because it created a dependency on another file, util.py. (and the change looks more complex than I would expect). Any comments on the approach of the PR? |
No description provided.
The text was updated successfully, but these errors were encountered: