alpaca-server

Modified alpaca.cpp to run as http server.

Run alpaca LLM on your local machine as HTTP server which you can connect with your web application (or use provided minimal HTTP UI).

Compiled and tested on MacBook Pro with M1 Pro CPU (10 core = 8 performance + 2 efficiency) and 16GB RAM. When using 8 threads performance is ~40ms per token. Should compile on Linux and Windows too.

Prerequisites (Mac)

You need C and C++ compiler, for example clang and clang++ which is shipped with XCode and installed with:

xcode-select --install

CMake:

brew install cmake

Prerequisites (Linux and Windows)

You need C and C++ compiler and CMake. If someone compiled on these platforms, please make a pull request with this section of README updated. Thanks :)

Download model

1. Make models/ subdirectory here
2. Download ggml-alpaca-7b-q4.bin model and put it into models/ directory

Compile

1. Create build/ directory, cd into it and run cmake .. then make
2. Executable will be written into bin/ directory. cd into it and run server with ./alpaca-server --port 8080 --threads 8 --temp 0.1 (use number of threads optimal for your CPU. In my case it is 8 which is number of M1 Pro performance cores. It works slower with 10 threads because then it uses 2 slower efficiency cores)
3. Navigate your browser to http://localhost:8080 and enjoy.

Using the server

Server has two endpoints:

GET at / where it serves html/index.html file which has simple input for prompt, makes POST request on click and displays response. You can ignore this file and use following POST request to use server with your application:

POST at /completions where it expects json payload in form:

{ "prompt": "Write a short poem about the moon" }

answer is in format similar to ChatGPT:

{
  "id": "391UXaLHbavJvFZGfO47XWS2",
  "object": "text_completion",
  "created": 1683496283,
  "choices": [
    {
      "text": "The moon shines bright in night sky, \nIlluminating darkness with its light;  \nIt's been there for all time passes by —   \nA constant companion through lonely skies.",
      "index": 0
    }
  ]
}

Or in case of error:

{
    "error": {
        "message": "We could not parse the JSON body of your request.",
        "type": "invalid_request_error",
        "param": null,
        "code": null
    }
}

Credit

This is based on Kevin Kwok's alpaca.cpp

Disclaimer

Note that the model weights are only to be used for research purposes, as they are derivative of LLaMA, and uses the published instruction data from the Stanford Alpaca project which is generated by OpenAI, which itself disallows the usage of its outputs to train competing models.

License

MIT