[Added Feature] Interactive Mode #43

Kasimir123 · 2023-04-07T17:55:48Z

I added an additional Python callback that allows main.cpp to grab user input from the Python script. This allows users to run interactive mode as if they were running the standard llama.cpp file. For testing I used the bob prompt from llama's prompts, this can be run with the following Python file:

from pyllamacpp.model import Model

def new_text_callback(text: str):
    print(text, end="", flush=True)

def grab_text_callback() -> str:
    return input() + "\n"

model = Model(ggml_model='..\\gpt4allconverted.bin', n_ctx=1024)

prompt = """
Transcript of a dialog, where the User interacts with an Assistant named Bob. Bob is helpful, kind, honest, good at writing, and never fails to answer the User's requests immediately and with precision.

User: Hello, Bob.
Bob: Hello. How may I help you today?
User: Please tell me the largest city in Europe.
Bob: Sure. The largest city in Europe is Moscow, the capital of Russia.
User:"""

model.generate(prompt, n_predict=256, new_text_callback=new_text_callback, grab_text_callback=grab_text_callback, n_threads=8, interactive=True, repeat_penalty=1.0, antiprompt=["User:"])```

abdeladim-s · 2023-04-07T22:49:41Z

Thanks @Kasimir123 for the PR. Great idea!
But I think stdin should work without the need to any callback!
Have you tried it before and didn't work for you ?

Kasimir123 · 2023-04-08T14:29:43Z

Thanks @Kasimir123 for the PR. Great idea! But I think stdin should work without the need to any callback! Have you tried it before and didn't work for you ?

stdin does work if you are just trying to do a basic chatbot in a terminal. The callback allows people to use the python wrapper to build out applications using the llma model. For example now you could have a webui that can interact with the bot, or you can automatically feed it certain data as well.

nuance1979 · 2023-04-08T15:59:31Z

src/main.cpp

-                        // input stream is bad or EOF received
-                        return 0;
-                    }
+                    line = grab_text_callback().cast<std::string>();


Is there a way to let the callback signal EOF then check it here so that the function can terminate gracefully?

@Kasimir123 Yeah You are right.
I had a similar idea but was working with a different approach. I want to have more control over the inference so I decided to re-implement the logic in a similar way.
What @nuance1979 is asking for (and many other people) is difficult to do from the callback.

@Kasimir123 let me know if you have any idea on how to signal EOF, otherwise I will merge the PR ?

I can think of two ways to signal EOF:

Use a special string (e.g., "[EOF]") or character (e.g., "\x03");

Let the callback return a Tuple[string, bool] where the boolean means EOF.

How do you think?

Of course, we could merge this PR first and continue this discussion in another PR. @Kasimir123 @abdeladim-s

@Kasimir123 is not replying, so I don't know if he is still working on it.

@nuance1979, I already implemented a different logic for interactive mode. So, this approach won't be necessary any more.

Oh great! Are you going to make a PR soon? Look forward to it.

Hey! Was a busy weekend. I will look into this tomorrow and see what I can come up with. Sounds like @abdeladim-s already has a different approach but I want to see if its possible just out of curiosity, will share what I find.

@nuance1979 @abdeladim-s I managed to add EOF file handling by adding a check to see if the callback result was of type string. This means that if you want end of file you can return None and it will correctly exit the program. To test this change change your callback to:

def grab_text_callback() -> str: inpt = input() if inpt == "END": return None return inpt + "\n"

nuance1979 · 2023-04-13T16:22:24Z

src/main.cpp

-                        // input stream is bad or EOF received
+                    py::handle x = grab_text_callback();
+
+                    if (!py::isinstance<py::str>(x))


Great! Does it mean that I can return anything other than a Python str to signal EOF?
If that's the idea, I would suggest the following:

first check if x is None as the EOF signal; (ref: https://pybind11.readthedocs.io/en/stable/reference.html?highlight=is%20None#_CPPv4NK10object_api7is_noneEv)

then check if isinstance(x, str) and give out a warning if not;

Do we want it to warn and exit if the type is not string or simply warn and try again for input? I personally prefer the warn and exit method as the other way could lead to some weird loops.

Also, what string do we want the warning to be? Something like: "Input was not of type py:str"?

Do we want it to warn and exit if the type is not string or simply warn and try again for input? I personally prefer the warn and exit method as the other way could lead to some weird loops.

Well if it exits, it's not a warning anymore; it's a failure. I think failure here is a bit heavy-handed; warn and ignore is enough.

Updated with the above. To test you can use:

def grab_text_callback() -> str: inpt = input() if inpt == "END": return None if inpt == "1": return 1 if inpt == "dict": return {1: 'help'} return inpt + "\n"

… warn user

nuance1979 · 2023-04-13T20:33:03Z

src/main.cpp

-                        line.pop_back(); // Remove the continue character
+                    else if (!py::isinstance<py::str>(x))
+                    {
+                        new_text_callback("Input was not of type py:str");


No, I don't think it's a good idea to use new_text_callback to print out warning. I'm suggesting fprintf(stderr,...) to give out the warning in a "side channel", without affecting the "main channel" at all.

Do we want to keep that warning message or have a different message? just before I do another commit changing the way we print it out.

Following the convention in this file, I would suggest:

fprintf(stderr, "%s : input was not of type py::str. will ignore.\n", __func__);

Good idea, just pushed the latest change.

Great! Can you update the README file to let users know that they should return None from grab_text_callback to signal EOF? Also I trust you've already tested it.

Updated the README with an example of how to run interactive mode as well since I saw people asking about that in the issues and a quick start will help. And yes, I have been running this on my end and recompiling and testing with the code snippet above each time.

Thanks @Kasimir123 and @nuance1979 for the contribution & sorry for this late reply.
I will merge it for now.

nuance1979 · 2023-04-13T21:49:24Z

README.md

+    # To signal EOF, return None
+    if inpt == "END":
+        return None
+    return inpt + "\n"


Are you sure about return inpt + "\n" instead of return inpt? Because when I test it, only return inpt works correctly. It makes sense because the original std::getline() call also returns a line without the trailing \n.

Thanks @nuance1979
I think you are right, I will fix it.

nuance1979 · 2023-04-13T21:51:15Z

README.md

+def new_text_callback(text: str):
+    print(text, end="", flush=True)
+
+def grab_text_callback() -> str:


def grab_text_callback() -> Optional[str]:

added interactive mode to python keybindings

57315b3

nuance1979 reviewed Apr 8, 2023

View reviewed changes

nuance1979 mentioned this pull request Apr 9, 2023

Refactor with pyllamacpp nuance1979/llama-server#3

Merged

Added EOF safe handling

7295b17

nuance1979 reviewed Apr 13, 2023

View reviewed changes

Added a check to see if None type, if not, check if str, if not, then…

405586f

… warn user

nuance1979 reviewed Apr 13, 2023

View reviewed changes

Kasimir123 added 2 commits April 13, 2023 16:47

changed warning from using the callback to using printf to stderr

f5bd061

added interactive mode to the readme

f384b73

nuance1979 reviewed Apr 13, 2023

View reviewed changes

abdeladim-s merged commit 8f8666d into nomic-ai:main Apr 23, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Added Feature] Interactive Mode #43

[Added Feature] Interactive Mode #43

Kasimir123 commented Apr 7, 2023

abdeladim-s commented Apr 7, 2023

Kasimir123 commented Apr 8, 2023

nuance1979 Apr 8, 2023

abdeladim-s Apr 9, 2023

nuance1979 Apr 9, 2023

nuance1979 Apr 10, 2023 •

edited

abdeladim-s Apr 10, 2023

nuance1979 Apr 11, 2023

Kasimir123 Apr 11, 2023

Kasimir123 Apr 12, 2023

nuance1979 Apr 13, 2023

Kasimir123 Apr 13, 2023

Kasimir123 Apr 13, 2023

nuance1979 Apr 13, 2023

Kasimir123 Apr 13, 2023

nuance1979 Apr 13, 2023

Kasimir123 Apr 13, 2023

nuance1979 Apr 13, 2023 •

edited

Kasimir123 Apr 13, 2023

nuance1979 Apr 13, 2023

Kasimir123 Apr 13, 2023

abdeladim-s Apr 23, 2023

nuance1979 Apr 13, 2023 •

edited

abdeladim-s Apr 23, 2023

nuance1979 Apr 13, 2023

[Added Feature] Interactive Mode #43

[Added Feature] Interactive Mode #43

Conversation

Kasimir123 commented Apr 7, 2023

abdeladim-s commented Apr 7, 2023

Kasimir123 commented Apr 8, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

nuance1979 Apr 10, 2023 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

nuance1979 Apr 13, 2023 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

nuance1979 Apr 13, 2023 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

nuance1979 Apr 10, 2023 •

edited

nuance1979 Apr 13, 2023 •

edited

nuance1979 Apr 13, 2023 •

edited