(Long)LLMLingua: Enhancing Large Language Model Inference via Prompt Compression

| Project Page | LLMLingua Paper | LongLLMLingua Paper | HF Space Demo |

Install & Run

# install
pip install .
# copy config and edit it.
cp config_template.yaml config.yaml
# run
python main.py

API

• path：api/v1/compress_prompt
• method：GET
• params example:

{
    "ratio": 0.5,
    "context": [""],
    "question": ""
}

• response example:

{
    "data": {
        "compressed_prompt": "",
        "origin_tokens": 3701,
        "compressed_tokens": 2778,
        "ratio": "1.3x",
        "saving": ", Saving $0.1 in GPT-4."
    }
}

Example

• example_zh

Reference

https://llmlingua.com/

https://github.com/microsoft/LLMLingua

https://zhuanlan.zhihu.com/p/660805821