implement ooba-booga's text gen API

[ramzeez88]

if someone could implement this. I tried but it is too hard for me.

[kthomasms]

Yeah I want to run this with the WizardCoder Model I think it will CHANGE THE GAME!!!!

[noxiouscardiumdimidium]

YES GOD PLEASE!!!!
I Spent $9 in openai api calls 2 days ago...

[rombodawg]

Im already working on it, so far it partially works, but it still has bugs, like empty generations, and no files being created.

You can follow the project here
https://github.com/rombodawg/gpt-engineer-Text-generation-web-ui-version/releases/tag/ai

[rombodawg]

If anyone is willing with coding skills is willing to help me finish the project i could really use it. Hit me up on discord, i have the same discord username

[upbit]

You can try the latest version, because ai.py has switched to the OpenAI SDK, you can switch to oobabooga/text-generation-webui by defining the following environment variables:

OPENAI_API_KEY=sk-111111111111111111111111111111111111111111111111
OPENAI_API_BASE=http://127.0.0.1:5001/v1

See An OpenedAI API - Client Application Setup for more detail.

ps: I have not tried on WizardCoder yet.

[kthomasms]

OpenLLM might be the better approach, I don't have the skills to help but can research real well 😁. Here is a video that put it on my radar: https://youtu.be/8nZZ2oQhx4E

…

On Thu, Jun 22, 2023, 4:46 PM rombodawg ***@***.***> wrote: If anyone is willing with coding skills is willing to help me finish the project i could really use it. Hit me up on discord, i have the same discord username — Reply to this email directly, view it on GitHub <#122 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AL4TTKJDDS5B5VNAWKTHPULXMTKMVANCNFSM6AAAAAAZKL2RCI> . You are receiving this because you commented.Message ID: ***@***.***>

[rombodawg]

I saw OpenLLM the only problem is that is barely supports any models, and i dont believe it supports quantized models. So theres no chance of anyone running starcoder unless you have a rtx 8000 with 48gb of vram

[upbit]

Looks like it will work with WebUI/OpenAI extensions + WizardCoder-15B-GPTQ

On Windows:

# Install gpt-engineer or clone https://github.com/AntonOsika/gpt-engineer
set OPENAI_API_KEY=sk-111111111111111111111111111111111111111111111111
set OPENAI_API_BASE=http://127.0.0.1:5001/v1

cd  projects
gpt-engineer example

[ramzeez88]

For those who are struggling to run this you need to create a new conda environment. Once done , activate it and git clone gpt-engineer. Navigate in cmd to gpt-engineer folder (not gpt_engineer) and type pip install -e . (with the dot!)
Then go to ooba's folder -> extensions ->openAI and type pip install -r requirements.txt . Let it install everything.
Then copy and paste line by line :
set OPENAI_API_KEY=sk-111111111111111111111111111111111111111111111111
set OPENAI_API_BASE=http://127.0.0.1:5001/v1
Next, open up ooba's text gen webui and move to interface mode tab on the top and select api and openai and hit 'apply and restart the interface'.
And then from the cmd you can type gpt-engineer projects/example(or your folder name).
This should work on Windows.Hope this helps !

[rombodawg]

Can you please make a youtube video on how to set this up, ive been trying but for the life of me i cant get it to work

[rombodawg]

I keep getting this error on text gen web ui's server:

code 404, message Not Found
"GET /v1/models/gpt-4 HTTP/1.1" 404 -

[upbit]

This feature is a plugin of WebUI. Based on the error message, you need to enable the openai extension of WebUI (add the startup parameter --extensions openai)

See Setup & installation for more details

[linussmashtips]

I tried this and im getting:
AuthenticationError: No API key provided. You can set your API key in code using 'openai.api_key = <API-KEY>', or you can set the environment variable OPENAI_API_KEY=<API-KEY>). If your API key is stored in a file, you can point the openai module at it with 'openai.api_key_path = <PATH>'. You can generate API keys in the OpenAI web interface. See https://platform.openai.com/account/api-keys for details.

did i miss anything?

[Chrusciki]

linux user here:

to use with oobabooga
make a .env at the base of the workspace with

OPENAI_API_KEY=sk-111111111111111111111111111111111111111111111111
OPENAI_API_BASE=http://0.0.0.0:5001/v1

edit ai.py

add the following prior to import openai

from dotenv import load_dotenv

load_dotenv()

run text webui with --extention openai --listen --api

i've loaded wizardvicuna 30b and it would generate some files and attempts to run them but doesn't generate working code, or complete files. i havent played with it much

[noxiouscardiumdimidium]

if someone could implement this. I tried but it is too hard for me.

STEPS:

0. (optional) if you haven't tried it, download PYGMALION-AI Metharme 1.3B , a 2.9GB high-context AUTOTASKING ENGINE, add the metharme template to the bottom of your prompt <|system|><|user|>Input<|model|>

1. create prelaunchers

2. launch textgen, confirm api address

3. launch gpt-engineer

4. you're done, have fun

prelauncher script i use for textgen (modify args and gpu settings to meet your individual needs and hardware)

create a fake/dummy/spoofed key, doesn't matter what it is, as long as it matches what you put in gpt-engineer

launch(textgen).sh

#!/bin/bash
export HOST_PORT=7861
export CUDA_VISIBLE_DEVICES=0
export TORCH_USE_CUDA_DSA=1
export TORCH_CUDA_ARCH_LIST=8.6+PTX
export PYTORCH_CUDA_ALLOC_CONF=garbage_collection_threshold:0.25,max_split_size_mb:128M
export OPENAI_API_KEY="dummy" # export ANY string in this field, including the default textgen key "dummy" it works!

python3 ./server.py --model TheBloke_orca_mini_7B-GPTQ --loader exllama --listen --listen-port 7861 --api --gpu-split 8_20 --trust-remote-code --extensions openai

prelauncher i use for gpt-engineer

launch(gpte).sh

#!/bin/bash
export OPENAI_API_KEY="dummy" # make sure it matches what you export/set in textgen
export OPENAI_API_BASE=http://0.0.0.0:5001/v1 #depending on settings, and ports in use when you launched textgen, this may change slightly if 5001 is in use - confirm base with textgen launch-post, prior to launching gpt-engineer

python3 gpt_engineer/main.py projects/MasterCodeBase/ --temperature 1 --steps tdd+

[upbit]

This one?

https://github.com/oobabooga/text-generation-webui/blob/main/extensions/openai/script.py#L39

[noxiouscardiumdimidium]

no there not, you set the new token in textgen, and if you're not connecting to openai, there tokens caps don't matter.....

2048 : 1024 prompt truncate ; 1024 max new ,thats about as high as you can before it breaks the format of templating and starts to spit out gibberish

[rombodawg]

Yea see that why this method isnt viable. Most coding projects take well over 1,000 tokens, and if you wanna use the new 8k models you are better off using my prompt engineering template to accomplish the same thing that gpt engineer does. But with it you can use all 8k tokens and use local ai, and have entire code projects made just like the gpt-engineer program, you just have to enter the prompts manually

[rombodawg]

You can use my prompt engineering template here
https://github.com/rombodawg/gpt-engineer-Text-generation-web-ui-version/blob/main/Gpt-engineer%20prompt%20template.rtf

Im still testing what models are the best 8k ones, but here are the to contenders. I think guanaco might be faster but i still have to test it
https://huggingface.co/Panchovix/Guanaco-33B-SuperHOT-8k/tree/main
and
https://huggingface.co/TheBloke/airoboros-33B-gpt4-1-4-SuperHOT-8K-fp16/tree/main

[rombodawg]

I did test that guanaco model, and it should be faster but it was having weird issues generating in web ui. Not sure if you'll have the same issues but the airoboros model is a good alternative. I believe it should close to the as good as the quantized version of guanaco 65b