llm-mpt30b

Plugin for LLM adding support for the MPT-30B language model.

This plugin uses TheBloke/mpt-30B-GGML. The code was inspired by abacaj/mpt-30B-inference.

Installation

Install this plugin in the same environment as LLM.

llm install llm-mpt30b

After installing the plugin you will need to download the ~19GB model file. You can do this by running:

llm mpt30b download

Usage

This plugin adds a model called mpt30b. You can execute it like this:

llm -m mpt30b "Three great names for a pet goat"

The alias -m mpt works as well.

You can pass the option -o verbose 1 to see more verbose output - currently a progress bar showing any additional downloads that are made during execution.

Development

To set up this plugin locally, first checkout the code. Then create a new virtual environment:

cd llm-mpt30b
python3 -m venv venv
source venv/bin/activate

Now install the dependencies and test dependencies:

pip install -e '.[test]'

To run the tests:

pytest