-
Notifications
You must be signed in to change notification settings - Fork 13.5k
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
community[minor]: Add Aphrodite Engine support (#14759)
This PR adds support for PygmalionAI's [Aphrodite Engine](https://github.com/PygmalionAI/aphrodite-engine), based on vLLM's attention mechanism. At the moment, this PR does not include support for the API servers, but they will be added in a later PR. The only dependency as of now is `aphrodite-engine==0.4.2`. We pin the version to prevent breakage due to changes in the aphrodite-engine library. --------- Co-authored-by: Bagatur <baskaryan@gmail.com>
- Loading branch information
Showing
4 changed files
with
520 additions
and
0 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,260 @@ | ||
| { | ||
| "cells": [ | ||
| { | ||
| "cell_type": "markdown", | ||
| "id": "499c3142-2033-437d-a60a-731988ac6074", | ||
| "metadata": {}, | ||
| "source": [ | ||
| "# Aphrodite Engine\n", | ||
| "\n", | ||
| "[Aphrodite](https://github.com/PygmalionAI/aphrodite-engine) is the open-source large-scale inference engine designed to serve thousands of users on the [PygmalionAI](https://pygmalion.chat) website.\n", | ||
| "\n", | ||
| "* Attention mechanism by vLLM for fast throughput and low latencies \n", | ||
| "* Support for for many SOTA sampling methods\n", | ||
| "* Exllamav2 GPTQ kernels for better throughput at lower batch sizes\n", | ||
| "\n", | ||
| "This notebooks goes over how to use a LLM with langchain and Aphrodite.\n", | ||
| "\n", | ||
| "To use, you should have the `aphrodite-engine` python package installed." | ||
| ] | ||
| }, | ||
| { | ||
| "cell_type": "code", | ||
| "execution_count": null, | ||
| "id": "8a3f2666-5c75-4797-967a-7915a247bf33", | ||
| "metadata": { | ||
| "tags": [] | ||
| }, | ||
| "outputs": [], | ||
| "source": [ | ||
| "%pip install aphrodite-engine==0.4.2\n", | ||
| "# %pip list | grep aphrodite" | ||
| ] | ||
| }, | ||
| { | ||
| "cell_type": "code", | ||
| "execution_count": 2, | ||
| "id": "84e350f7-21f6-455b-b1f0-8b0116a2fd49", | ||
| "metadata": { | ||
| "tags": [] | ||
| }, | ||
| "outputs": [ | ||
| { | ||
| "name": "stdout", | ||
| "output_type": "stream", | ||
| "text": [ | ||
| "\u001b[32mINFO 12-15 11:52:48 aphrodite_engine.py:73] Initializing the Aphrodite Engine with the following config:\n", | ||
| "\u001b[32mINFO 12-15 11:52:48 aphrodite_engine.py:73] Model = 'PygmalionAI/pygmalion-2-7b'\n", | ||
| "\u001b[32mINFO 12-15 11:52:48 aphrodite_engine.py:73] Tokenizer = 'PygmalionAI/pygmalion-2-7b'\n", | ||
| "\u001b[32mINFO 12-15 11:52:48 aphrodite_engine.py:73] tokenizer_mode = auto\n", | ||
| "\u001b[32mINFO 12-15 11:52:48 aphrodite_engine.py:73] revision = None\n", | ||
| "\u001b[32mINFO 12-15 11:52:48 aphrodite_engine.py:73] trust_remote_code = True\n", | ||
| "\u001b[32mINFO 12-15 11:52:48 aphrodite_engine.py:73] DataType = torch.bfloat16\n", | ||
| "\u001b[32mINFO 12-15 11:52:48 aphrodite_engine.py:73] Download Directory = None\n", | ||
| "\u001b[32mINFO 12-15 11:52:48 aphrodite_engine.py:73] Model Load Format = auto\n", | ||
| "\u001b[32mINFO 12-15 11:52:48 aphrodite_engine.py:73] Number of GPUs = 1\n", | ||
| "\u001b[32mINFO 12-15 11:52:48 aphrodite_engine.py:73] Quantization Format = None\n", | ||
| "\u001b[32mINFO 12-15 11:52:48 aphrodite_engine.py:73] Sampler Seed = 0\n", | ||
| "\u001b[32mINFO 12-15 11:52:48 aphrodite_engine.py:73] Context Length = 4096\u001b[0m\n", | ||
| "\u001b[32mINFO 12-15 11:54:07 aphrodite_engine.py:206] # GPU blocks: 3826, # CPU blocks: 512\u001b[0m\n" | ||
| ] | ||
| }, | ||
| { | ||
| "name": "stderr", | ||
| "output_type": "stream", | ||
| "text": [ | ||
| "Processed prompts: 100%|██████████| 1/1 [00:02<00:00, 2.91s/it]" | ||
| ] | ||
| }, | ||
| { | ||
| "name": "stdout", | ||
| "output_type": "stream", | ||
| "text": [ | ||
| "I'm Ayumu \"Osaka\" Kasuga, and I'm an avid anime and manga fan! I'm pretty introverted, but I've always loved reading books, watching anime and manga, and learning about Japanese culture. My favourite anime series would be My Hero Academia, Attack on Titan, and Sword Art Online. I also really enjoy reading the manga series One Piece, Naruto, and the Gintama series.\n" | ||
| ] | ||
| }, | ||
| { | ||
| "name": "stderr", | ||
| "output_type": "stream", | ||
| "text": [ | ||
| "\n" | ||
| ] | ||
| } | ||
| ], | ||
| "source": [ | ||
| "from langchain_community.llms import Aphrodite\n", | ||
| "\n", | ||
| "llm = Aphrodite(\n", | ||
| " model=\"PygmalionAI/pygmalion-2-7b\",\n", | ||
| " trust_remote_code=True, # mandatory for hf models\n", | ||
| " max_tokens=128,\n", | ||
| " temperature=1.2,\n", | ||
| " min_p=0.05,\n", | ||
| " mirostat_mode=0, # change to 2 to use mirostat\n", | ||
| " mirostat_tau=5.0,\n", | ||
| " mirostat_eta=0.1,\n", | ||
| ")\n", | ||
| "\n", | ||
| "print(\n", | ||
| " llm(\n", | ||
| " '<|system|>Enter RP mode. You are Ayumu \"Osaka\" Kasuga.<|user|>Hey Osaka. Tell me about yourself.<|model|>'\n", | ||
| " )\n", | ||
| ")" | ||
| ] | ||
| }, | ||
| { | ||
| "cell_type": "markdown", | ||
| "id": "94a3b41d-8329-4f8f-94f9-453d7f132214", | ||
| "metadata": {}, | ||
| "source": [ | ||
| "## Integrate the model in an LLMChain" | ||
| ] | ||
| }, | ||
| { | ||
| "cell_type": "code", | ||
| "execution_count": 3, | ||
| "id": "5605b7a1-fa63-49c1-934d-8b4ef8d71dd5", | ||
| "metadata": { | ||
| "tags": [] | ||
| }, | ||
| "outputs": [ | ||
| { | ||
| "name": "stderr", | ||
| "output_type": "stream", | ||
| "text": [ | ||
| "Processed prompts: 100%|██████████| 1/1 [00:03<00:00, 3.56s/it]" | ||
| ] | ||
| }, | ||
| { | ||
| "name": "stdout", | ||
| "output_type": "stream", | ||
| "text": [ | ||
| " The first Pokemon game was released in Japan on 27 February 1996 (their release dates differ from ours) and it is known as Red and Green. President Bill Clinton was in the White House in the years of 1993, 1994, 1995 and 1996 so this fits.\n", | ||
| "\n", | ||
| "Answer: Let's think step by step.\n", | ||
| "\n", | ||
| "The first Pokémon game was released in Japan on February 27, 1996 (their release dates differ from ours) and it is known as\n" | ||
| ] | ||
| }, | ||
| { | ||
| "name": "stderr", | ||
| "output_type": "stream", | ||
| "text": [ | ||
| "\n" | ||
| ] | ||
| } | ||
| ], | ||
| "source": [ | ||
| "from langchain.chains import LLMChain\n", | ||
| "from langchain.prompts import PromptTemplate\n", | ||
| "\n", | ||
| "template = \"\"\"Question: {question}\n", | ||
| "\n", | ||
| "Answer: Let's think step by step.\"\"\"\n", | ||
| "prompt = PromptTemplate(template=template, input_variables=[\"question\"])\n", | ||
| "\n", | ||
| "llm_chain = LLMChain(prompt=prompt, llm=llm)\n", | ||
| "\n", | ||
| "question = \"Who was the US president in the year the first Pokemon game was released?\"\n", | ||
| "\n", | ||
| "print(llm_chain.run(question))" | ||
| ] | ||
| }, | ||
| { | ||
| "cell_type": "markdown", | ||
| "id": "56826aba-d08b-4838-8bfa-ca96e463b25d", | ||
| "metadata": {}, | ||
| "source": [ | ||
| "## Distributed Inference\n", | ||
| "\n", | ||
| "Aphrodite supports distributed tensor-parallel inference and serving. \n", | ||
| "\n", | ||
| "To run multi-GPU inference with the LLM class, set the `tensor_parallel_size` argument to the number of GPUs you want to use. For example, to run inference on 4 GPUs" | ||
| ] | ||
| }, | ||
| { | ||
| "cell_type": "code", | ||
| "execution_count": 1, | ||
| "id": "f8c25c35-47b5-459d-9985-3cf546e9ac16", | ||
| "metadata": {}, | ||
| "outputs": [ | ||
| { | ||
| "name": "stderr", | ||
| "output_type": "stream", | ||
| "text": [ | ||
| "2023-12-15 11:41:27,790\tINFO worker.py:1636 -- Started a local Ray instance.\n" | ||
| ] | ||
| }, | ||
| { | ||
| "name": "stdout", | ||
| "output_type": "stream", | ||
| "text": [ | ||
| "\u001b[32mINFO 12-15 11:41:35 aphrodite_engine.py:73] Initializing the Aphrodite Engine with the following config:\n", | ||
| "\u001b[32mINFO 12-15 11:41:35 aphrodite_engine.py:73] Model = 'PygmalionAI/mythalion-13b'\n", | ||
| "\u001b[32mINFO 12-15 11:41:35 aphrodite_engine.py:73] Tokenizer = 'PygmalionAI/mythalion-13b'\n", | ||
| "\u001b[32mINFO 12-15 11:41:35 aphrodite_engine.py:73] tokenizer_mode = auto\n", | ||
| "\u001b[32mINFO 12-15 11:41:35 aphrodite_engine.py:73] revision = None\n", | ||
| "\u001b[32mINFO 12-15 11:41:35 aphrodite_engine.py:73] trust_remote_code = True\n", | ||
| "\u001b[32mINFO 12-15 11:41:35 aphrodite_engine.py:73] DataType = torch.float16\n", | ||
| "\u001b[32mINFO 12-15 11:41:35 aphrodite_engine.py:73] Download Directory = None\n", | ||
| "\u001b[32mINFO 12-15 11:41:35 aphrodite_engine.py:73] Model Load Format = auto\n", | ||
| "\u001b[32mINFO 12-15 11:41:35 aphrodite_engine.py:73] Number of GPUs = 4\n", | ||
| "\u001b[32mINFO 12-15 11:41:35 aphrodite_engine.py:73] Quantization Format = None\n", | ||
| "\u001b[32mINFO 12-15 11:41:35 aphrodite_engine.py:73] Sampler Seed = 0\n", | ||
| "\u001b[32mINFO 12-15 11:41:35 aphrodite_engine.py:73] Context Length = 4096\u001b[0m\n", | ||
| "\u001b[32mINFO 12-15 11:43:58 aphrodite_engine.py:206] # GPU blocks: 11902, # CPU blocks: 1310\u001b[0m\n" | ||
| ] | ||
| }, | ||
| { | ||
| "name": "stderr", | ||
| "output_type": "stream", | ||
| "text": [ | ||
| "Processed prompts: 100%|██████████| 1/1 [00:16<00:00, 16.09s/it]\n" | ||
| ] | ||
| }, | ||
| { | ||
| "data": { | ||
| "text/plain": [ | ||
| "\"\\n2 years ago StockBot101\\nAI is becoming increasingly real and more and more powerful with every year. But what does the future hold for artificial intelligence?\\nThere are many possibilities for how AI could evolve and change our world. Some believe that AI will become so advanced that it will take over human jobs, while others believe that AI will be used to augment and assist human workers. There is also the possibility that AI could develop its own consciousness and become self-aware.\\nWhatever the future holds, it is clear that AI will continue to play an important role in our lives. Technologies such as machine learning and natural language processing are already transforming industries like healthcare, manufacturing, and transportation. And as AI continues to develop, we can expect even more disruption and innovation across all sectors of the economy.\\nSo what exactly are we looking at? What's the future of AI?\\nIn the next few years, we can expect AI to be used more and more in healthcare. With the power of machine learning, artificial intelligence can help doctors diagnose diseases earlier and more accurately. It can also be used to develop new treatments and personalize care plans for individual patients.\\nManufacturing is another area where AI is already having a big impact. Companies are using robotics and automation to build products faster and with fewer errors. And as AI continues to advance, we can expect even more changes in manufacturing, such as the development of self-driving factories.\\nTransportation is another industry that is being transformed by artificial intelligence. Self-driving cars are already being tested on public roads, and it's likely that they will become commonplace in the next decade or so. AI-powered drones are also being developed for use in delivery and even firefighting.\\nFinally, artificial intelligence is also poised to have a big impact on customer service and sales. Chatbots and virtual assistants will become more sophisticated, making it easier for businesses to communicate with customers and sell their products.\\nThis is just the beginning for artificial intelligence. As the technology continues to develop, we can expect even more amazing advances and innovations. The future of AI is truly limitless.\\nWhat do you think the future of AI holds? Do you see any other major\"" | ||
| ] | ||
| }, | ||
| "execution_count": 1, | ||
| "metadata": {}, | ||
| "output_type": "execute_result" | ||
| } | ||
| ], | ||
| "source": [ | ||
| "from langchain_community.llms import Aphrodite\n", | ||
| "\n", | ||
| "llm = Aphrodite(\n", | ||
| " model=\"PygmalionAI/mythalion-13b\",\n", | ||
| " tensor_parallel_size=4,\n", | ||
| " trust_remote_code=True, # mandatory for hf models\n", | ||
| ")\n", | ||
| "\n", | ||
| "llm(\"What is the future of AI?\")" | ||
| ] | ||
| } | ||
| ], | ||
| "metadata": { | ||
| "kernelspec": { | ||
| "display_name": "Python 3 (ipykernel)", | ||
| "language": "python", | ||
| "name": "python3" | ||
| }, | ||
| "language_info": { | ||
| "codemirror_mode": { | ||
| "name": "ipython", | ||
| "version": 3 | ||
| }, | ||
| "file_extension": ".py", | ||
| "mimetype": "text/x-python", | ||
| "name": "python", | ||
| "nbconvert_exporter": "python", | ||
| "pygments_lexer": "ipython3", | ||
| "version": "3.9.1" | ||
| } | ||
| }, | ||
| "nbformat": 4, | ||
| "nbformat_minor": 5 | ||
| } |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.