Integrate Open-source Models #245

HalberdOfPineapple · 2023-08-11T06:36:53Z

Description

Tried to introduce open-source models into CAMEL framework. Have successfully used LLaMA2 (7b, chat) and Vicuna (7b, v1.5) for assistant role, with corresponding ModelType elements defined.
The basic workflow is based on an external server running LLM inference. This is because:
1. such a server can keep the LLM in memory eliminating the need to repeatedly loading it from file for every request;
2. such type of servers typically implemented OpenAI compatible interfaces, which makes the interactions between them quite convenient. Current implementation assumes the user is running a server by himself and should provide CAMEL with the URL to this server, e.g. http://localhost:8000/v1
3. This can provided some degree of freedom to choose their preferred server and decouples the inference logic of LLMs from CAMEL. My current choice is FastChat, which supports OpenAI compatible interfaces and many mainstream open-source models.
Necessary arguments and how they are specified and passed:
- path to model tokenizers: added an extra argument model_path for BaseModelBackend class, though this should only be applicable for open-source models. It can either be a model id in HuggingFace such as lmsys/vicuna-13b-v1.5-16k or a path to the local folder containing the tokenizer (necessary) and models.
- URL to the server running the LLM: It should be specified in the environmental variable OPENAI_API_BASE, which will be set to alter openai.api_base.
Refactored the token counting functionality as different types of open-source models have different tokenization regulations. This PR packed the related functionalities into utils/token_counting.py and equips each model object with a specific token counter.
Removed BaseMessage.token_len as this function seems not useful. Counting number of tokens cannot be decoupled from the specific model and especially the open-source models need to load specific tokenizers. Each time of counting tokens here needs to load the tokenizer, which is quite costly. We can just count the tokens after models are defined with their token counters.

The functionality structure may need to be further optimized but Guohao told me to create this PR for more convenient reviewing.

Motivation and Context

Why is this change required? What problem does it solve?
close #225

I have raised an issue to propose this change (required for new features and bug fixes)

Types of changes

What types of changes does your code introduce? Put an x in all the boxes that apply:

Bug fix (non-breaking change which fixes an issue)
New feature (non-breaking change which adds core functionality)
Breaking change (fix or feature that would cause existing functionality to change)
Documentation (update in the documentation)
Example (update in the folder of example)

Implemented Tasks

Subtask 1
Subtask 2
Subtask 3

Checklist

Go over all the following points, and put an x in all the boxes that apply.
If you are unsure about any of these, don't hesitate to ask. We are here to help!

I have read the CONTRIBUTION guide. (required)
My change requires a change to the documentation.
I have updated the tests accordingly. (required for a bug fix or a new feature)
I have updated the documentation accordingly.

dandansamax

This is an awesome change and we can invoke any model using openAI API after merging it!

There are some suggestions about code structure.

However, I think the biggest problem is that I should set up a entirely new environment to launch a LLM server in my computer to use open source models, which is diffcult and time wasting. If I am a user who want to use a opensource model in camel, it is possibly because I don't have an openai key but I want to try camel for free. These complicated set up processes may prevent me from trying camel. We should either provide detailed instructions of how to set up a llm server or using a wrapper to do that in the code.

Anyway, this PR is very valuable for invoking other models, thank you so much!

camel/models/open_source_model.py

camel/typing.py

camel/utils/functions.py

camel/utils/token_counting.py

examples/test/test_ai_society_example.py

dandansamax · 2023-08-13T11:06:23Z

camel/models/model_factory.py

+    def create(
+        model_type: ModelType,
+        model_config_dict: Dict,
+        model_path: Optional[str] = None,


Is it possible to not contain model_path parameter in this? I think it is only used to create the tokenizer. So can we assign it inside the factory by model_type instead of passing it in?

The thing is the current model_type has only general information of the model but without the specific model name/path (e.g. lm-sys/vicuna-7b-v1.5).
I might get what you mean by modifying ModelType to be a data class? Should we implement another class called like ModelCard, which has a field being ModelType in limited range and can have other information like model path?

HalberdOfPineapple · 2023-08-14T02:51:49Z

This is an awesome change and we can invoke any model using openAI API after merging it!

There are some suggestions about code structure.

However, I think the biggest problem is that I should set up a entirely new environment to launch a LLM server in my computer to use open source models, which is diffcult and time wasting. If I am a user who want to use a opensource model in camel, it is possibly because I don't have an openai key but I want to try camel for free. These complicated set up processes may prevent me from trying camel. We should either provide detailed instructions of how to set up a llm server or using a wrapper to do that in the code.

Anyway, this PR is very valuable for invoking other models, thank you so much!

Thanks for these valuable suggestions! Yeah I totally agree with the problem for the inconvenience launching the server, which I have discussed with Guohao. I am now trying to find a way to automatically launching a FastChat server given a model path. But after all, such an option will make users' options of the server limited to our choice.

Obs01ete

Great feature you are adding! Thanks! Let's polish the code.

camel/agents/chat_agent.py

camel/agents/critic_agent.py

camel/agents/task_agent.py

camel/models/open_source_model.py

examples/test/test_ai_society_example.py

examples/open_source_models/role_playing_with_llama2.py

Obs01ete · 2023-08-14T13:18:00Z

camel/models/open_source_model.py

+class OpenSourceModel(BaseModelBackend):
+    r"""OpenAI API in a unified BaseModelBackend interface."""
+
+    def __init__(self, model_type: ModelType, model_config_dict: Dict[str,


We need a test for this newly introduced class.

camel/models/open_source_model.py

Obs01ete · 2023-08-14T13:21:13Z

camel/utils/token_counting.py

+        return num_tokens
+
+
+class TokenCounterFactory:


We need a test for TokenCounterFactory with all possible models types.

HalberdOfPineapple · 2023-08-14T16:14:42Z

Have just modified the current implementation but has not completed the documentation and tests. Will do in tomorrow morning as it is quite late (UTC+8)

lightaime

Awesome! Left some comments.

camel/agents/chat_agent.py

camel/models/base_model.py

lightaime · 2023-08-15T01:37:32Z

camel/models/model_factory.py

+    def create(
+        model_type: ModelType,
+        model_config_dict: Dict,
+        model_path: Optional[str] = None,


camel/models/model_factory.py

camel/utils/token_counting.py

camel/models/open_source_model.py

HalberdOfPineapple · 2023-08-15T08:43:09Z

I cannot agree with this idea. A enumeration containing too much information is not good.

I might get what you mean and write the code below. Could you have a look at the implementation below? This code isolate the data and behaviour of each model type into another dataclass.

e.g.

@dataclass(frozen=True)
class ModelConfig:
    name: str = ...
    model_path: str = ...
    is_open_source: bool = ...
    ...

class ModelType(Enum):
    GPT_3_5_TURBO = "gpt-3.5-turbo"
    GPT_3_5_TURBO_16K = "gpt-3.5-turbo-16k"
    GPT_4 = "gpt-4"
    GPT_4_32k = "gpt-4-32k"
    STUB = "stub"

    LLAMA_2 = "llama-2"
    VICUNA = "vicuna"
    VICUNA_16K = "vicuna-16k"

class ModelConfigFactory:
    def create(model_type: ModelType, kwargs: Dict[str, Any]):
        if model_type in {
                ModelType.GPT_4,
                ModelType.GPT_3_5_TURBO,
                ...
        }:
            return ModelConfig(
                name = kwargs.get("model_name", model_type.value),
                model_path = None,
                is_open_source = False,
                ...
            )
       elif model_type in {
                ModelType.LLAMA_2,
                ...,
        }:
            if "model_path" not in kwargs:
                raise ValueError(...)

            return ModelConfig(
                name = kwargs.get("model_name", model_type.value),
                model_path = kwargs["model_path"]
                is_open_source = True,
                ...
            )
        else:
                raise ValueError("Invalid model type")

Hi @Obs01ete @lightaime I write the sample code following Tianqi's suggestion and am wondering whether such an implementation of ModelType is viable?
And some of the comments given by Guohao are mismatched with the commit done on yesterday (but I uploaded a commit after Guohao's comments without reading them today)

Obs01ete · 2023-08-15T22:27:10Z

@MorphlingEd Why do we need a factory for configs? Can't we make model backends in the factory and that's it?

HalberdOfPineapple · 2023-08-16T04:40:19Z

@MorphlingEd Why do we need a factory for configs? Can't we make model backends in the factory and that's it?

Yeah that's a illed design shown here using a redundant factory. Later I tried to introduce a unified ModelBackendConfig which can be created with the model backend objects together in ModelFactory, containing all the information that is possessed by ModelType now. But I did not see the obvious convenience by completely removing properties from the ModelType enumeration. I need to write ModelBackendConfig for all of the already defined constants (e.g. GPT_4) one-by-one as they do not share properties. For example:

...
if model_type is ModelType.GPT_4:
    model_config = ModelConfig(
                ...
                token_limit = 8192,
                ...
            )
    return OpenAIModel(model_config, ...)
elif model_type is ModelType.GPT_3_5_TURBO:
    model_config = ModelConfig(
                ...
                token_limit = 4096
                ...
            )
    return OpenAIModel(model_config, ...)
...

Though I could use a dictionary like token_limit_map for mapping the model types to different token limits to reduce the repeated code, I think this is not simpler than using a "naive" enum class.

In addition, by introducing the new OpenSourceConfig in config.py, both the new arguments, api_base (server_url) and model_path, needed by open-source models can be conveniently passed.
So I guess we can just keep the original implementation of enum?

HalberdOfPineapple · 2023-08-16T09:45:55Z

I just found there are some authentication problems due to the HuggingFace repo for LLaMA2 is gated. If we do not authenticate by the following commands in the check environment like below:

export HUGGINGFACE_TOKEN="<huggingface_token>"
huggingface-cli login --token $HUGGINGFACE_TOKEN

when using HuggingFace's AutoTokenizer, it will report errors like:
Cannot access gated repo for url https://huggingface.co/meta-llama/Llama-2-7b-chat-hf/resolve/main/tokenizer_config.json. Repo model meta-llama/Llama-2-7b-chat-hf is gated. You must be authenticated to access it.

HalberdOfPineapple · 2023-08-16T09:46:56Z

Sorry just closed the branch by mistake

Obs01ete

Great tests in test_open_source_model.py. Left some comments.

camel/models/base_model.py

camel/configs.py

camel/models/model_factory.py

camel/models/base_model.py

camel/models/open_source_model.py

camel/models/openai_model.py

camel/models/stub_model.py

camel/societies/role_playing.py

camel/typing.py

Obs01ete

Good progress. Left comments.

camel/models/stub_model.py

camel/models/openai_model.py

.github/workflows/documentation.yaml

HalberdOfPineapple · 2023-08-24T05:35:59Z

@Obs01ete I have reverted the python version back to 3.8. But I failed trying to implement an equivalent functionality to kw_only even with a decorator. Finally I changed the OpenSourceConfig to be composed of an instance of ChatGPTConfig. This might be meaningful as configurations for open-source models should be different from ChatGPTConfig and the parameter values in ChatGPTConfig should be compacted into "parameters for OpenAI API". I am not sure whether this makes sense to you.

Obs01ete

I see, now all comments are resolved, and the code looks nice. Approved.

dandansamax · 2023-08-24T10:57:00Z

LGTM. Only the config part is a little redundant now. We can consider refactoring it later. Anyway, good job! Feel free to merge it.

Obs01ete · 2023-08-24T12:08:48Z

I changed the OpenSourceConfig to be composed of an instance of ChatGPTConfig

Yeah, this does not look natural... Okay, let's leave it like this and rework later. Go ahead with merging this PR.

lightaime · 2023-08-24T12:13:49Z

Could you wait for my final check a bit?

…

On Thu 24. Aug 2023 at 14:08, Dmitrii Khizbullin ***@***.***> wrote: I changed the OpenSourceConfig to be composed of an instance of ChatGPTConfig Yeah, this does not look natural... Okay, let's leave it like this and rework later. Go ahead with merging this PR. — Reply to this email directly, view it on GitHub <#245 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AFUJTYHKLVWV6CYEHSIGHXDXW475VANCNFSM6AAAAAA3MPOFIA> . You are receiving this because you were mentioned.Message ID: ***@***.***>

HalberdOfPineapple · 2023-08-24T12:17:22Z

Could you wait for my final check a bit?

Yes of course :D

HalberdOfPineapple · 2023-08-30T03:30:50Z

@lightaime Hi Guohao, could I ask whether there is still something to be improved further? :)

lightaime

Look great!! Thanks @MorphlingEd. Left some small comments. Please free feel to merge it.

README.md

camel/models/open_source_model.py

README.md

camel/models/open_source_model.py

ShukriChiu · 2023-09-02T00:55:01Z

@CodiumAI-Agent /review

CodiumAI-Agent · 2023-09-02T00:56:02Z

PR Analysis

🎯 Main theme: Integration of open-source models into the CAMEL framework
📝 PR summary: This PR introduces open-source models into the CAMEL framework, specifically LLaMA2 and Vicuna. The implementation assumes the user is running a server by himself and should provide CAMEL with the URL to this server. The PR also includes refactoring of the token counting functionality and removal of BaseMessage.token_len function.
📌 Type of PR: Enhancement
🧪 Relevant tests added: Yes
🔒 Security concerns: No security concerns found

PR Feedback

💡 General suggestions: The PR is well-structured and includes a good amount of detail in the description. The integration of open-source models into the CAMEL framework is a significant enhancement. The refactoring of the token counting functionality and the removal of the BaseMessage.token_len function seem to be well-justified. However, it would be beneficial to include more comments in the code to explain the logic and the functionality of the methods, especially for the token counting functionality.
🤖 Code feedback:
- relevant file: camel/utils/token_counting.py
  suggestion: Consider adding more comments to explain the logic and functionality of the methods, especially for the token counting functionality. This will make the code more maintainable and understandable for other developers. [medium]
  relevant line: class BaseTokenCounter(ABC):
- relevant file: camel/utils/token_counting.py
  suggestion: It would be beneficial to handle exceptions more gracefully in the get_model_encoding function. Instead of just printing "Model not found. Using cl100k_base encoding.", consider logging this information and also adding more context to the error message. [medium]
  relevant line: def get_model_encoding(value_for_tiktoken: str):
- relevant file: camel/utils/functions.py
  suggestion: It seems like the openai_api_key_required function could be simplified. The function currently checks if the model is a stub or an open-source model, and if the OpenAI API key is in the environment variables. Consider simplifying this function to only check if the OpenAI API key is in the environment variables, as this is the only requirement for the function to work correctly. [medium]
  relevant line: def openai_api_key_required(func: F) -> F:
- relevant file: camel/configs.py
  suggestion: The OpenSourceConfig class could benefit from more detailed docstrings. Specifically, it would be helpful to explain what the model_path and server_url parameters are used for, and what their expected values are. [medium]
  relevant line: class OpenSourceConfig(BaseConfig):

How to use

Tag me in a comment '@CodiumAI-Agent' and add one of the following commands:
/review [-i]: Request a review of your Pull Request. For an incremental review, which only considers changes since the last review, include the '-i' option.
/describe: Modify the PR title and description based on the contents of the PR.
/improve [--extended]: Suggest improvements to the code in the PR. Extended mode employs several calls, and provides a more thorough feedback.
/ask <QUESTION>: Pose a question about the PR.
/update_changelog: Update the changelog based on the PR's contents.

To edit any configuration parameter from configuration.toml, add --config_path=new_value
For example: /review --pr_reviewer.extra_instructions="focus on the file: ..."
To list the possible configuration parameters, use the /config command.

CodiumAI-Agent · 2023-09-02T15:16:50Z

PR Analysis

🎯 Main theme: Integration of Open-source Models into CAMEL Framework
📝 PR summary: This PR introduces open-source models into the CAMEL framework. It includes the implementation of token counting for different types of open-source models and the necessary arguments for their specification. The PR also includes tests for the new functionality and refactors some existing code.
📌 Type of PR: Enhancement
🧪 Relevant tests added: Yes
🔒 Security concerns: No security concerns found

PR Feedback

💡 General suggestions: The PR is well-structured and includes a significant amount of work. It is good to see that tests have been added for the new functionality. However, it would be beneficial to add more comments in the code to explain the logic and the purpose of the functions, especially for complex ones. This will make the code easier to understand and maintain in the future.
🤖 Code feedback:
- relevant file: camel/utils/token_counting.py
  suggestion: Consider handling exceptions more gracefully in the OpenSourceTokenCounter class. Instead of raising a ValueError when the tokenizer fails to load, you could log an error message and return a default value or handle the exception in a way that doesn't interrupt the execution of the program. [important]
  relevant line: '+ raise ValueError(
- relevant file: camel/utils/token_counting.py
  suggestion: The messages_to_prompt function seems to have a lot of repeated code for different model types. Consider refactoring this function to reduce code duplication. You could create a dictionary to map model types to their corresponding separators and role maps, and then use this dictionary in the function. [medium]
  relevant line: '+def messages_to_prompt(messages: List[OpenAIMessage], model: ModelType) -> str:'
- relevant file: camel/utils/functions.py
  suggestion: The openai_api_key_required decorator could be improved by using a more descriptive error message. Instead of just saying "OpenAI API key not found", you could provide more information about how to set the API key. [medium]
  relevant line: '+ raise ValueError("OpenAI API key not found")'
- relevant file: camel/configs.py
  suggestion: In the OpenSourceConfig class, it would be helpful to add type hints for the model_path and server_url parameters. This will make the code more readable and will allow static type checkers and IDE features to work correctly. [medium]
  relevant line: '+ model_path: str'

How to use

Tag me in a comment '@CodiumAI-Agent' and add one of the following commands:
/review [-i]: Request a review of your Pull Request. For an incremental review, which only considers changes since the last review, include the '-i' option.
/describe: Modify the PR title and description based on the contents of the PR.
/improve [--extended]: Suggest improvements to the code in the PR. Extended mode employs several calls, and provides a more thorough feedback.
/ask <QUESTION>: Pose a question about the PR.
/update_changelog: Update the changelog based on the PR's contents.

To edit any configuration parameter from configuration.toml, add --config_path=new_value
For example: /review --pr_reviewer.extra_instructions="focus on the file: ..."
To list the possible configuration parameters, use the /config command.

HalberdOfPineapple and others added 5 commits August 9, 2023 11:58

Initially refactored token counting functionality

bba76c5

Basically implemented token counting for Open-source (to test)

60d3c96

Basically implemented mechanisms for open-source models

1908ebc

Merge branch 'master' into llama_wenxuan

f248246

Merge master

f658115

HalberdOfPineapple requested review from lightaime and Obs01ete August 11, 2023 06:37

Fixed check errors

1aed788

dandansamax requested changes Aug 13, 2023

View reviewed changes

Obs01ete requested changes Aug 14, 2023

View reviewed changes

Wenxuan Li and others added 2 commits August 15, 2023 06:26

fix some inconsistencies. Tests need to be added

ba930c6

Merge branch 'master' into llama_wenxuan

b64afb4

lightaime reviewed Aug 15, 2023

View reviewed changes

Fix some documentation and import problems.

09b35af

Wenxuan Li and others added 3 commits August 16, 2023 10:30

Modified OpenS Config, model class and added test for them

a24f918

Fix check errors

8a1b317

Removed LLaMA2 from test to avoid authentcation problem

aeab52b

HalberdOfPineapple closed this Aug 16, 2023

HalberdOfPineapple reopened this Aug 16, 2023

Wenxuan Li and others added 3 commits August 16, 2023 12:59

Update dependency to include protobuf

090c1d5

Modified dependency

45c1907

Fix typo

a887cb7

Obs01ete requested changes Aug 16, 2023

View reviewed changes

Wenxuan Li and others added 3 commits August 21, 2023 18:15

Hide implementation of token counter from BaseModel

c4e3901

Merge branch 'master' into llama_wenxuan

facdca6

Add a draft version of documentation about using this functionality

f0987a7

Obs01ete requested changes Aug 22, 2023

View reviewed changes

camel/models/stub_model.py Outdated Show resolved Hide resolved

camel/models/openai_model.py Show resolved Hide resolved

.github/workflows/documentation.yaml Outdated Show resolved Hide resolved

Wenxuan Li and others added 4 commits August 22, 2023 16:52

Added STUB Token Counter

c9f1413

Removed redundant part of testing exceeding tokens

03b39fa

Merge branch 'master' into llama_wenxuan

f0c73ee

Revert vertion change and make opensource config composite

d4f32b8

Obs01ete approved these changes Aug 24, 2023

View reviewed changes

HalberdOfPineapple requested a review from dandansamax August 24, 2023 09:47

dandansamax approved these changes Aug 24, 2023

View reviewed changes

Merge branch 'master' into llama_wenxuan

fa8d3eb

lightaime approved these changes Aug 30, 2023

View reviewed changes

Modified README, open-source model and corresponding test

93a28c2

HalberdOfPineapple mentioned this pull request Aug 31, 2023

[BUG] HuggingFace Tool Connection Problem #268

Closed

3 tasks

lightaime reviewed Aug 31, 2023

View reviewed changes

camel/models/open_source_model.py Outdated Show resolved Hide resolved

dandansamax and others added 3 commits August 31, 2023 13:18

Ignore hugging face test error

75c561b

Merged Tianqi's Fix

4085c5a

Merge branch 'master' into llama_wenxuan

57def33

HalberdOfPineapple merged commit 7f0315f into master Sep 1, 2023
10 checks passed

HalberdOfPineapple deleted the llama_wenxuan branch September 1, 2023 03:28

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Integrate Open-source Models #245

Integrate Open-source Models #245

HalberdOfPineapple commented Aug 11, 2023

dandansamax left a comment •

edited

Loading

dandansamax Aug 13, 2023

HalberdOfPineapple Aug 14, 2023

lightaime Aug 15, 2023

HalberdOfPineapple commented Aug 14, 2023

Obs01ete left a comment

Obs01ete Aug 14, 2023

Obs01ete Aug 14, 2023

HalberdOfPineapple commented Aug 14, 2023

lightaime left a comment

lightaime Aug 15, 2023

HalberdOfPineapple commented Aug 15, 2023

Obs01ete commented Aug 15, 2023

HalberdOfPineapple commented Aug 16, 2023 •

edited

Loading

HalberdOfPineapple commented Aug 16, 2023 •

edited

Loading

HalberdOfPineapple commented Aug 16, 2023

Obs01ete left a comment

Obs01ete left a comment

HalberdOfPineapple commented Aug 24, 2023

Obs01ete left a comment

dandansamax commented Aug 24, 2023

Obs01ete commented Aug 24, 2023 •

edited

Loading

lightaime commented Aug 24, 2023 via email

HalberdOfPineapple commented Aug 24, 2023

HalberdOfPineapple commented Aug 30, 2023

lightaime left a comment

ShukriChiu commented Sep 2, 2023

CodiumAI-Agent commented Sep 2, 2023

CodiumAI-Agent commented Sep 2, 2023

Integrate Open-source Models #245

Integrate Open-source Models #245

Conversation

HalberdOfPineapple commented Aug 11, 2023

Description

Motivation and Context

Types of changes

Implemented Tasks

Checklist

dandansamax left a comment • edited Loading

Choose a reason for hiding this comment

dandansamax Aug 13, 2023

Choose a reason for hiding this comment

HalberdOfPineapple Aug 14, 2023

Choose a reason for hiding this comment

lightaime Aug 15, 2023

Choose a reason for hiding this comment

HalberdOfPineapple commented Aug 14, 2023

Obs01ete left a comment

Choose a reason for hiding this comment

Obs01ete Aug 14, 2023

Choose a reason for hiding this comment

Obs01ete Aug 14, 2023

Choose a reason for hiding this comment

HalberdOfPineapple commented Aug 14, 2023

lightaime left a comment

Choose a reason for hiding this comment

lightaime Aug 15, 2023

Choose a reason for hiding this comment

HalberdOfPineapple commented Aug 15, 2023

Obs01ete commented Aug 15, 2023

HalberdOfPineapple commented Aug 16, 2023 • edited Loading

HalberdOfPineapple commented Aug 16, 2023 • edited Loading

HalberdOfPineapple commented Aug 16, 2023

Obs01ete left a comment

Choose a reason for hiding this comment

Obs01ete left a comment

Choose a reason for hiding this comment

HalberdOfPineapple commented Aug 24, 2023

Obs01ete left a comment

Choose a reason for hiding this comment

dandansamax commented Aug 24, 2023

Obs01ete commented Aug 24, 2023 • edited Loading

lightaime commented Aug 24, 2023 via email

HalberdOfPineapple commented Aug 24, 2023

HalberdOfPineapple commented Aug 30, 2023

lightaime left a comment

Choose a reason for hiding this comment

ShukriChiu commented Sep 2, 2023

CodiumAI-Agent commented Sep 2, 2023

PR Analysis

PR Feedback

How to use

CodiumAI-Agent commented Sep 2, 2023

PR Analysis

PR Feedback

How to use

dandansamax left a comment •

edited

Loading

HalberdOfPineapple commented Aug 16, 2023 •

edited

Loading

HalberdOfPineapple commented Aug 16, 2023 •

edited

Loading

Obs01ete commented Aug 24, 2023 •

edited

Loading