🪐Feat: Support of local LLMs which are Free to run (DONE)✅

[SaturnCassini]

Hello, would it be possible to integrate something like GPT4All which runs locally and doesn't cost unlike OpenAI?

[drax-xard]

I second this motion, there are several local LLM software solutions that provide API access and it'd open this project to the majority of people.

[joonspk-research]

+1 That would be great. I will try my best to find some time, but for now, here is a response I gave in another thread:
It should be relatively simple to make it happen. The main place you will need to edit would be:
generative_agents/reverie/backend_server/persona/prompt_template/gpt_structure.py

This is where all the actual API calls to OpenAI are made. You will want to edit the functions there to instead call Llama2. (You may have to edit utils.py + a few prompts if they break when using it with Llama)

[InconsolableCellist]

You can run https://github.com/oobabooga/text-generation-webui with this extension: https://github.com/oobabooga/text-generation-webui/tree/main/extensions/openai

You can also replace the embeddings with Sentence Transformers by replacing get_embeddings(...):

from sentence_transformers import SentenceTransformer
def get_embedding(text, model_name="paraphrase-distilroberta-base-v1"):
    # Initialize the SentenceTransformer model
    model = SentenceTransformer(model_name)

    text = text.replace("\n", " ")
    if not text:
        text = "this is blank"

    # Get the embedding
    embedding = model.encode(text)

    return embedding

And then override api_base in utils.py to point to your proxy.

I have a hacky fork doing this at the moment: main...InconsolableCellist:generative_agents:main

However I'm not seeing any observed actions in my simulation. I suspect my model isn't generating content that's parsed into useful actions, though I haven't gone that far debugging it yet.

[370025263]

we have a similiar work like this standford town, but work in progress. our team put some open source llm in the senario and we tested that THEY JUST DO NOT WORK without finetuing. the open source llms like llama and chatglm can not generate reasonable and stable content.

[SaturnCassini]

Implemented GPT4All here :) https://github.com/SaturnCassini/gpt4all_generative_agents
Stars appreciated⭐️

[SaturnCassini]

It was super easy so thanks for modularizing all the gpt logic in one place @joonspk-research

[pjq]

Implemented GPT4All here :) https://github.com/SaturnCassini/gpt4all_generative_agents Stars appreciatedstar

Just curious does the simulation work with gpt4all?

[SaturnCassini]

@pjq It does work, it tends to loop but i think that can be solved with increased temperature and changing the params of the model, like penalty for repetition.

Also I didnt try models with more params, only orca mini which is known to not give the best results

[musellama]

You can run https://github.com/oobabooga/text-generation-webui with this extension: https://github.com/oobabooga/text-generation-webui/tree/main/extensions/openai

You can also replace the embeddings with Sentence Transformers by replacing get_embeddings(...):

from sentence_transformers import SentenceTransformer
def get_embedding(text, model_name="paraphrase-distilroberta-base-v1"):
    # Initialize the SentenceTransformer model
    model = SentenceTransformer(model_name)

    text = text.replace("\n", " ")
    if not text:
        text = "this is blank"

    # Get the embedding
    embedding = model.encode(text)

    return embedding

And then override api_base in utils.py to point to your proxy.

I have a hacky fork doing this at the moment: main...InconsolableCellist:generative_agents:main

However I'm not seeing any observed actions in my simulation. I suspect my model isn't generating content that's parsed into useful actions, though I haven't gone that far debugging it yet.

Does this mean replacing the entire utils.py with this code? But it seems that the error is reported openai_api_key does not exist, how to call this code instead?

[InconsolableCellist]

Does this mean replacing the entire utils.py with this code? But it seems that the error is reported openai_api_key does not exist, how to call this code instead?

If you want to give it a try you can fork my repo. I had to add api_base in utils.py and then change the GPT calls to include api_base as a param (not sure if I could've done an environment variable or global variable instead). I replaced get_embedding where it appeared in gpt_structure.py as well.

Follow the instructions as normal to create your utils.py and add api_base = "http://yourserver:5001/v1"

[musellama]

这是否意味着用此代码替换整个 utils.py？但是似乎报告了错误openai_api_key不存在，如何调用此代码？

如果你想试一试，你可以分叉我的回购。我不得不在 utils.py 中添加api_base，然后更改 GPT 调用以包含api_base作为参数（不确定我是否可以代替环境变量或全局变量）。我也替换了get_embedding它出现在gpt_structure.py的地方。

照常按照说明创建 utils.py 并添加api_base = "http://yourserver:5001/v1"

I found an API interface that mimics OpenAPI under Alpaca2 in Stanford, and now it's deployed, and I only need URL http://localhost:19327/v1/completions to access the local port, so how can I now dock the port. Do I need to make changes to the gpt_structure.py script or utils.py?

[changeling]

@SaturnCassini Could you open up 'Issues" on your repo?

[chuyqa]

#3 (comment)
The extension + setting export OPENAI_API_BASE="http://mylanahost:5001/v1" did it

[SaturnCassini]

Here ! @changeling https://github.com/SaturnCassini/gpt4all_generative_agents/issues

[babytdream]

Whenever run xxx ends, this error will appear. Has anyone encountered this problem?

-==- -==- -==- 
Traceback (most recent call last):
  File "/data/generative_agents/reverie/backend_server/reverie.py", line 471, in open_server
    rs.start_server(int_count)
  File "/data/generative_agents/reverie/backend_server/reverie.py", line 379, in start_server
    next_tile, pronunciatio, description = persona.move(
  File "/data/generative_agents/reverie/backend_server/persona/persona.py", line 222, in move
    plan = self.plan(maze, personas, new_day, retrieved)
  File "/data/generative_agents/reverie/backend_server/persona/persona.py", line 148, in plan
    return plan(self, maze, personas, new_day, retrieved)
  File "/data/generative_agents/reverie/backend_server/persona/cognitive_modules/plan.py", line 959, in plan
    _determine_action(persona, maze)
  File "/data/generative_agents/reverie/backend_server/persona/cognitive_modules/plan.py", line 573, in _determine_action
    generate_task_decomp(persona, act_desp, act_dura))
  File "/data/generative_agents/reverie/backend_server/persona/cognitive_modules/plan.py", line 164, in generate_task_decomp
    return run_gpt_prompt_task_decomp(persona, task, duration)[0]
  File "/data/generative_agents/reverie/backend_server/persona/prompt_template/run_gpt_prompt.py", line 439, in run_gpt_prompt_task_decomp
    output = safe_generate_response(prompt, gpt_param, 5, get_fail_safe(),
  File "/data/generative_agents/reverie/backend_server/persona/prompt_template/gpt_structure.py", line 262, in safe_generate_response
    return func_clean_up(curr_gpt_response, prompt=prompt)
  File "/data/generative_agents/reverie/backend_server/persona/prompt_template/run_gpt_prompt.py", line 378, in __func_clean_up
    duration = int(k[1].split(",")[0].strip())
IndexError: list index out of range
Error.