Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Use a FakeLLM to be able to test certain features without an API KEY #20

Closed
este6an13 opened this issue Mar 9, 2024 · 4 comments
Closed

Use a FakeLLM to be able to test certain features without an API KEY #20

este6an13 opened this issue Mar 9, 2024 · 4 comments
Labels
feature New feature

Comments

@este6an13
Copy link
Contributor

There should be a way for contributors to test most of the features without using the actual OpenAI model. This would allow them implementing features and making contributions without being charged.
@este6an13 este6an13 added the feature New feature label Mar 9, 2024
@este6an13 este6an13 changed the title Use a FakeLLM to be able to test certain features with an API KEY Use a FakeLLM to be able to test certain features without an API KEY Mar 9, 2024
este6an13 added a commit that referenced this issue Mar 10, 2024
@este6an13
reprebot/refac: #20 update setup model method to receive model type
d425796 
- add `model_type` argument which can be `gpt`, `hugging-face` or `fake`
- note: currently `hugging-face` model cannot be use
  due to [issue](langchain-ai/langchain#18639)
- note: `FakeLLM` is not supported on Windows yet
- replace client test with `TestFakeLLM` which should work in the
  pipeline without needing any API KEY or TOKEN, however won't work on Windows
- add `clean-env.bat` script to re-install direct dependencies and
  remove non-used ones
este6an13 added a commit that referenced this issue Mar 10, 2024
@este6an13
reprebot/refac: #20 update setup model method to receive model type
706b16b 
- add `model_type` argument which can be `gpt`, `hugging-face` or `fake`
- note: currently `hugging-face` model cannot be use
  due to [issue](langchain-ai/langchain#18639)
- note: `FakeLLM` is not supported on Windows yet
- replace client test with `TestFakeLLM` which should work in the
  pipeline without needing any API KEY or TOKEN, however won't work on Windows
- add `clean-env.bat` script to re-install direct dependencies and
  remove non-used ones
@este6an13
Copy link
Contributor Author

Currently FakeLLM is not supported for Windows, so if you use Windows for running the unit test TestFakeLLM, it will fail. However, in the pipeline, it will work, which means contributors won't see the build fail in their forks. I will try to incorporate a HuggingFace model which would require a token but users wouldn't be charged, I think. At this moment, it seems to be an error with LangChain and HuggingFace according to this issue but once it's solved, we will try to implement it.

este6an13 added a commit that referenced this issue Mar 12, 2024
@este6an13
reprebot/refac: #20 remove fake-llm
b729372 
- remove `FakeLLM` implementation because `langchain-contrib`
  doesn't support latest version of `langchain`
- remove `langchain-contrib` dependency
- update `langchain` to latest version: `0.1.11` #23
- remove `TestFakeLLM` unit test
- restore `TestLLMClient` unit test
- add versions to `direct-requirements.txt`
este6an13 added a commit that referenced this issue Mar 12, 2024
@este6an13
reprebot/refac: #20 remove fake-llm
0d5d676 
- remove `FakeLLM` implementation because `langchain-contrib`
  doesn't support latest version of `langchain`
- remove `langchain-contrib` dependency
- update `langchain` to latest version: `0.1.11` #23
- remove `TestFakeLLM` unit test
- restore `TestLLMClient` unit test
- add versions to `direct-requirements.txt`
@este6an13
Copy link
Contributor Author

I decided to remove FakeLLM class because it requires langchain-contrib dependency which doesn't support latest version of langchain. This resulted in conflicts when trying to implement the RAG chain in LLMClient. We will try to use a HuggingFace model for testing in the future.

@este6an13 este6an13 reopened this Mar 15, 2024
este6an13 added a commit that referenced this issue Mar 15, 2024
@este6an13
reprebot/feat: #20 use fake-list-chat-model
9d0afc0 
- use `FakeListChatModel` to instantiate fake models for testing
- remove `gpt` model unit test
- add `fake` model unit test
- note: with this, we won't incur in costs when running unit tests
este6an13 added a commit that referenced this issue Mar 15, 2024
@este6an13
reprebot/feat: #20 use fake-list-chat-model
8429150 
- use `FakeListChatModel` to instantiate fake models for testing
- remove `gpt` model unit test
- add `fake` model unit test
- note: with this, we won't incur in costs when running unit tests
@este6an13
Copy link
Contributor Author

We will use FakeListChatModel from langchain for testing. This way we won't be need an OPENAI API KEY for unit testing anymore.

este6an13 added a commit that referenced this issue Mar 16, 2024
@este6an13
reprebot/feat: #3 add hugging face model to llm client
eeee89d 
- use `ChatHuggingFace` to instantiate a `Hugging Face` models
- use `HuggingFaceEndpoint` to connect to the `Hugging Face` model
- add `hugging-face` to `LLMClient`
- add `TestHuggingFaceLLMClient` unit test #20
- update `langchain` to `0.1.12` version #23
- install `transformers` dependency
- install `Jinja2` dependency
este6an13 added a commit that referenced this issue Mar 16, 2024
@este6an13
reprebot/feat: #3 add hugging face model to llm client
e54af39 
- use `ChatHuggingFace` to instantiate a `Hugging Face` models
- use `HuggingFaceEndpoint` to connect to the `Hugging Face` model
- add `hugging-face` to `LLMClient`
- add `TestHuggingFaceLLMClient` unit test #20
- update `langchain` to `0.1.12` version #23
- install `transformers` dependency
- install `Jinja2` dependency
este6an13 added a commit that referenced this issue Mar 16, 2024
@este6an13
reprebot/feat: #3 add hugging face model to llm client
b6e18cf 
- use `ChatHuggingFace` to instantiate a `Hugging Face` models
- use `HuggingFaceEndpoint` to connect to the `Hugging Face` model
- add `hugging-face` to `LLMClient`
- add `TestHuggingFaceLLMClient` unit test #20
- update `langchain` to `0.1.12` version #23
- install `transformers` dependency
- install `Jinja2` dependency
- add `HUGGINGFACEHUB_API_TOKEN` to `pipeline.yml`
este6an13 added a commit that referenced this issue Mar 16, 2024
@este6an13
reprebot/feat: #3 add hugging face model to llm client
d0d684b 
- use `ChatHuggingFace` to instantiate a `Hugging Face` models
- use `HuggingFaceEndpoint` to connect to the `Hugging Face` model
- add `hugging-face` to `LLMClient`
- add `TestHuggingFaceLLMClient` unit test #20
- update `langchain` to `0.1.12` version #23
- install `transformers` dependency
- install `Jinja2` dependency
- add `HUGGINGFACEHUB_API_TOKEN` to `pipeline.yml`
@este6an13 este6an13 mentioned this issue Mar 16, 2024
Merged
este6an13 added a commit that referenced this issue Mar 16, 2024
@este6an13
reprebot/feat: #3 add hugging face model to llm client
a0ef774 
- use `ChatHuggingFace` to instantiate a `Hugging Face` models
- use `HuggingFaceEndpoint` to connect to the `Hugging Face` model
- add `hugging-face` to `LLMClient`
- add `TestHuggingFaceLLMClient` unit test #20
- update `langchain` to `0.1.12` version #23
- install `transformers` dependency
- install `Jinja2` dependency
- add `HUGGINGFACEHUB_API_TOKEN` to `pipeline.yml`
@este6an13
Copy link
Contributor Author

Issue seemed to have been solved in langchain==0.1.12

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
feature New feature
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@este6an13