Add RetrieveChat #1158
Add RetrieveChat #1158
Conversation
flaml/autogen/retrieval_utils.py
Outdated
There was a problem hiding this comment.
nice. could be useful for other applications too.
@QingYun
| if "exitcode: 0 (execution succeeded)" in message.get("content", ""): | ||
| return "TERMINATE" |
There was a problem hiding this comment.
Sometimes even when the execution succeeds, the task is still not finished.
The result might indicate logic error or requires further steps.
There was a problem hiding this comment.
Although sometimes even when the code execution succeeds, the task is not solved, but it's hard to tell. If the human_input_mode of RetrieveUserProxyAgent is "TERMINATE" or "ALWAYS", user can still continue the conversation.
| # return exitcode, log, None | ||
|
|
||
| result = self._ipython.run_cell(code) | ||
| log = str(result.result) |
There was a problem hiding this comment.
The problem with this is that the result.result is often None because the stdout + stderr are not part of the result.result. This is true for the example you had in the notebook.
If capture doesn't work, then we need to find some other way to get the stdout + stderr.
There was a problem hiding this comment.
I'm not sure whether we should capture all of the stdout and stderr outputs and include them in the message. In the case of FLAML, the training logs can often be too long to fit into the LLM tokens. The result.result can only capture concise error information, which may not be sufficient for debugging problems. Therefore, I believe that for complex problems, it's essential to include human input as well.
There was a problem hiding this comment.
We can work on this in future PR.
| self.register_auto_reply(Agent, RetrieveUserProxyAgent._generate_retrieve_user_reply) | ||
|
|
||
| @staticmethod | ||
| def get_max_tokens(model="gpt-3.5-turbo"): |
There was a problem hiding this comment.
This method shouldn't be in this class. Move it to oai.openai_utils? Can be done in next PR.
| self._client = self._retrieve_config.get("client", chromadb.Client()) | ||
| self._docs_path = self._retrieve_config.get("docs_path", "./docs") | ||
| self._collection_name = self._retrieve_config.get("collection_name", "flaml-docs") | ||
| self._model = self._retrieve_config.get("model", "gpt-4") |
There was a problem hiding this comment.
This info is supposed to be in the sender's llm_config, not here. Refactoring will be needed later.
| self._docs_path = self._retrieve_config.get("docs_path", "./docs") | ||
| self._collection_name = self._retrieve_config.get("collection_name", "flaml-docs") | ||
| self._model = self._retrieve_config.get("model", "gpt-4") | ||
| self._max_tokens = self.get_max_tokens(self._model) |
There was a problem hiding this comment.
As discussed in #1160 , where to get the token limit info may change in future.
| self._doc_idx = -1 # the index of the current used doc | ||
| self._results = {} # the results of the current query |
There was a problem hiding this comment.
This assumes a single agent to work with. Different agents will need different context. Need to change later.
| self._ipython = get_ipython() | ||
| self._doc_idx = -1 # the index of the current used doc | ||
| self._results = {} # the results of the current query | ||
| self.register_auto_reply(Agent, RetrieveUserProxyAgent._generate_retrieve_user_reply) |
There was a problem hiding this comment.
A more recommended way is to register the context needed by this function through context instead of class variables. Can refactor later.
| # return exitcode, log, None | ||
|
|
||
| result = self._ipython.run_cell(code) | ||
| log = str(result.result) |
There was a problem hiding this comment.
We can work on this in future PR.
| TEXT_FORMATS = ["txt", "json", "csv", "tsv", "md", "html", "htm", "rtf", "rst", "jsonl", "log", "xml", "yaml", "yml"] | ||
|
|
||
|
|
||
| def num_tokens_from_text( |
There was a problem hiding this comment.
oai.openai_utils is a better fit? Can change later.
|
For future PRs that requires OpenAI test, please use a branch in microsoft/FLAML |
|
Since the notebook is very long, can you add a table of contents for the examples at the beginning? That is to show what the example demonstrates and unique features of retrieve chat, and the user can go to the corresponding examples. For example, Example 5: we demonstrate the unique feature "UPDATE CONTEXT"... |
Thank you @kevin666aa for the suggestion. I've added the ToC, it works in VSCode and local jupyter server, but doesn't work as expected in GitHub preview. Looks like github has limited preview support on the links. |
|
We can just put the contents without the links. I think people can easily go to the example by searching the keywords. I have no more questions. |
Why are these changes needed?
RetrieveChat is a convesational framework for retrieve augmented code generation and question answering. Essentially,
RetrieveAssistantAgentandRetrieveUserProxyAgentimplements a different auto reply mechanism corresponding to the RetrieveChat prompts.Related issue number
Closes #1107
Checks