Replace openai to llama2? #1

hackhy · 2023-08-07T03:26:43Z

How can I replace the OpenAI API with Llama2?

joonspk-research · 2023-08-09T23:33:14Z

There is a no built-in function for that yet, but it should be relatively simple to make it happen. The main place you will need to edit would be:
generative_agents/reverie/backend_server/persona/prompt_template/gpt_structure.py

This is where all the actual API calls to OpenAI are made. You will want to edit the functions there to instead call Llama2. (You may have to edit utils.py + a few prompts if they break when using it with Llama)

huvers · 2023-08-10T00:09:14Z

llama.cpp has a neat api_like_OAI.py drop in Flask server to mimic all the OpenAI API calls. You should be able to run that, then set ``openai.api_base = `"http://127.0.0.1:8081"```, and llama2 should work. I'll hopefully have time to try this weekend.

wonhyeongseo · 2023-08-10T00:35:40Z

May I please try writing up a PR using @huvers 's method? This thread seems to be a part of #3
Kudos for opensourcing this wonderful experiment!

AWAS666 · 2023-08-10T11:50:37Z

llama.cpp has a neat api_like_OAI.py drop in Flask server to mimic all the OpenAI API calls. You should be able to run that, then set ``openai.api_base = `"http://127.0.0.1:8081"```, and llama2 should work. I'll hopefully have time to try this weekend.

Yeah that works but you can also use textgenwebui with the openai extension, which is likely what many playing with llama based llm have installed already anyway.

SaturnCassini · 2023-08-10T13:46:38Z

This version runs on GPT4All model which is free and doesn't require a GPU

https://github.com/SaturnCassini/gpt4all_generative_agents

pjq · 2023-08-10T13:52:07Z

Probably you need this, and replace the base URL with your local URL.

https://github.com/abetlen/llama-cpp-python

pip install llama-cpp-python[server]
python3 -m llama_cpp.server --model models/7B/ggml-model.bin

The update to the baseUrl

http://localhost:8000/

ishaan-jaff · 2023-08-15T16:26:10Z

@hackhy I'm the maintainer of liteLLM https://github.com/BerriAI/litellm/ - we make it easy to switch between models.

Here's how you can call llama2 using liteLLM

from litellm import completion

## set ENV variables
os.environ["OPENAI_API_KEY"] = "openai key"

messages = [{ "content": "Hello, how are you?","role": "user"}]

# openai call
response = completion(model="gpt-3.5-turbo", messages=messages)

# llama2 call
response = completion(model="meta-llama/Llama-2-7b-hf", messages=messages)

PiPi-happy · 2023-08-18T14:22:38Z

"Excuse me, if I want to replace it with Azure OpenAI, what should I do?"

comaniac · 2023-08-22T23:14:22Z

@joonspk-research I have the same requirement and have changed the code in my fork. Would you want me to file a PR or it's already an ongoing feature on your side? I could send out a PR for it early next week if needed. Thanks.

certainforest · 2023-08-23T18:38:56Z

@comaniac how are you handling inconsistent completion structure? I'm running into issues with implementing llama because prompt responses don't appear to be coming back in json format and/or are inappropriate. 😓

comaniac · 2023-08-23T20:55:08Z

I actually commented this to another issue. I did encounter this problem and my feeling is this framework is tightly-coupled with certain OpenAI models in terms of prompts and response formats. We may need some efforts (e.g., prompt engineering) to make other models work seamlessly. Meanwhile, the first step toward to this goal is to refactor the framework so that we can configure the model. Then we could start tweaking.

ElliottDyson · 2023-09-06T15:05:05Z

I actually commented this to another issue. I did encounter this problem and my feeling is this framework is tightly-coupled with certain OpenAI models in terms of prompts and response formats. We may need some efforts (e.g., prompt engineering) to make other models work seamlessly. Meanwhile, the first step toward to this goal is to refactor the framework so that we can configure the model. Then we could start tweaking.

@metrics-dawg @comaniac
We're working on this over at OSGA (open source generative agents: https://github.com/ElliottDyson/OSGA), you're welcome to join us, there's a bit more activity on the Discord, but I've made good progress to getting working outputs out of llama 2.

babytdream · 2023-09-22T03:25:36Z

I use my own api_base.
Whenever run xxx ends, this error will appear. Has anyone encountered this problem?

-==- -==- -==- 
Traceback (most recent call last):
  File "/data/generative_agents/reverie/backend_server/reverie.py", line 471, in open_server
    rs.start_server(int_count)
  File "/data/generative_agents/reverie/backend_server/reverie.py", line 379, in start_server
    next_tile, pronunciatio, description = persona.move(
  File "/data/generative_agents/reverie/backend_server/persona/persona.py", line 222, in move
    plan = self.plan(maze, personas, new_day, retrieved)
  File "/data/generative_agents/reverie/backend_server/persona/persona.py", line 148, in plan
    return plan(self, maze, personas, new_day, retrieved)
  File "/data/generative_agents/reverie/backend_server/persona/cognitive_modules/plan.py", line 959, in plan
    _determine_action(persona, maze)
  File "/data/generative_agents/reverie/backend_server/persona/cognitive_modules/plan.py", line 573, in _determine_action
    generate_task_decomp(persona, act_desp, act_dura))
  File "/data/generative_agents/reverie/backend_server/persona/cognitive_modules/plan.py", line 164, in generate_task_decomp
    return run_gpt_prompt_task_decomp(persona, task, duration)[0]
  File "/data/generative_agents/reverie/backend_server/persona/prompt_template/run_gpt_prompt.py", line 439, in run_gpt_prompt_task_decomp
    output = safe_generate_response(prompt, gpt_param, 5, get_fail_safe(),
  File "/data/generative_agents/reverie/backend_server/persona/prompt_template/gpt_structure.py", line 262, in safe_generate_response
    return func_clean_up(curr_gpt_response, prompt=prompt)
  File "/data/generative_agents/reverie/backend_server/persona/prompt_template/run_gpt_prompt.py", line 378, in __func_clean_up
    duration = int(k[1].split(",")[0].strip())
IndexError: list index out of range
Error.

ElliottDyson · 2023-09-22T06:11:17Z

I use my own api_base.
Whenever run xxx ends, this error will appear. Has anyone encountered this problem?

-==- -==- -==- 
Traceback (most recent call last):
  File "/data/generative_agents/reverie/backend_server/reverie.py", line 471, in open_server
    rs.start_server(int_count)
  File "/data/generative_agents/reverie/backend_server/reverie.py", line 379, in start_server
    next_tile, pronunciatio, description = persona.move(
  File "/data/generative_agents/reverie/backend_server/persona/persona.py", line 222, in move
    plan = self.plan(maze, personas, new_day, retrieved)
  File "/data/generative_agents/reverie/backend_server/persona/persona.py", line 148, in plan
    return plan(self, maze, personas, new_day, retrieved)
  File "/data/generative_agents/reverie/backend_server/persona/cognitive_modules/plan.py", line 959, in plan
    _determine_action(persona, maze)
  File "/data/generative_agents/reverie/backend_server/persona/cognitive_modules/plan.py", line 573, in _determine_action
    generate_task_decomp(persona, act_desp, act_dura))
  File "/data/generative_agents/reverie/backend_server/persona/cognitive_modules/plan.py", line 164, in generate_task_decomp
    return run_gpt_prompt_task_decomp(persona, task, duration)[0]
  File "/data/generative_agents/reverie/backend_server/persona/prompt_template/run_gpt_prompt.py", line 439, in run_gpt_prompt_task_decomp
    output = safe_generate_response(prompt, gpt_param, 5, get_fail_safe(),
  File "/data/generative_agents/reverie/backend_server/persona/prompt_template/gpt_structure.py", line 262, in safe_generate_response
    return func_clean_up(curr_gpt_response, prompt=prompt)
  File "/data/generative_agents/reverie/backend_server/persona/prompt_template/run_gpt_prompt.py", line 378, in __func_clean_up
    duration = int(k[1].split(",")[0].strip())
IndexError: list index out of range
Error.

Yep, I encountered it too. Unfortunately can't remember what the fix was, but you're welcome to head over to the OSGA GitHub and use the code from there (which includes the fix, as well as other fixes)

Murat2283plus · 2024-01-03T06:13:15Z

"Excuse me, if I want to replace it with Azure OpenAI, what should I do?"“请问，如果我想更换为 Azure OpenAI，该怎么办？”

did you success?

ketsapiwiq · 2024-05-20T14:51:31Z

Here's a draft that kinda works: https://github.com/joonspk-research/generative_agents/pull/155/files

This was referenced Aug 16, 2023

Becareful price #71

Closed

Add alternative interface for self-hosted LLM #57

Open

Add alternative interface for RWKV-Runner #68

Open

Azure Openai #61

Open

ketsapiwiq mentioned this issue May 20, 2024

Work with ollama #155

Draft

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Replace openai to llama2? #1

Replace openai to llama2? #1

hackhy commented Aug 7, 2023

joonspk-research commented Aug 9, 2023

huvers commented Aug 10, 2023

wonhyeongseo commented Aug 10, 2023

AWAS666 commented Aug 10, 2023

SaturnCassini commented Aug 10, 2023 •

edited

Loading

pjq commented Aug 10, 2023 •

edited

Loading

ishaan-jaff commented Aug 15, 2023 •

edited

Loading

PiPi-happy commented Aug 18, 2023

comaniac commented Aug 22, 2023 •

edited

Loading

certainforest commented Aug 23, 2023

comaniac commented Aug 23, 2023

ElliottDyson commented Sep 6, 2023 •

edited

Loading

babytdream commented Sep 22, 2023

ElliottDyson commented Sep 22, 2023 •

edited

Loading

Murat2283plus commented Jan 3, 2024

ketsapiwiq commented May 20, 2024

Replace openai to llama2? #1

Replace openai to llama2? #1

Comments

hackhy commented Aug 7, 2023

joonspk-research commented Aug 9, 2023

huvers commented Aug 10, 2023

wonhyeongseo commented Aug 10, 2023

AWAS666 commented Aug 10, 2023

SaturnCassini commented Aug 10, 2023 • edited Loading

pjq commented Aug 10, 2023 • edited Loading

ishaan-jaff commented Aug 15, 2023 • edited Loading

PiPi-happy commented Aug 18, 2023

comaniac commented Aug 22, 2023 • edited Loading

certainforest commented Aug 23, 2023

comaniac commented Aug 23, 2023

ElliottDyson commented Sep 6, 2023 • edited Loading

babytdream commented Sep 22, 2023

ElliottDyson commented Sep 22, 2023 • edited Loading

Murat2283plus commented Jan 3, 2024

ketsapiwiq commented May 20, 2024

SaturnCassini commented Aug 10, 2023 •

edited

Loading

pjq commented Aug 10, 2023 •

edited

Loading

ishaan-jaff commented Aug 15, 2023 •

edited

Loading

comaniac commented Aug 22, 2023 •

edited

Loading

ElliottDyson commented Sep 6, 2023 •

edited

Loading

ElliottDyson commented Sep 22, 2023 •

edited

Loading