Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Good First Issue]: Verify red-pajama-3b-chat with GenAI text_generation #263

Closed
p-wysocki opened this issue Mar 1, 2024 · 13 comments · Fixed by #320
Closed

[Good First Issue]: Verify red-pajama-3b-chat with GenAI text_generation #263

p-wysocki opened this issue Mar 1, 2024 · 13 comments · Fixed by #320
Assignees
Labels
good first issue Good for newcomers

Comments

@p-wysocki
Copy link
Collaborator

Context

This task regards enabling tests for red-pajama-3b-chat. You can find more details under openvino_notebooks LLM chatbot README.md.

Please ask general questions in the main issue at #259

What needs to be done?

Described in the main Discussion issue at: #259

Example Pull Requests

Described in the main Discussion issue at: #259

Resources

Contact points

Described in the main Discussion issue at: #259

Ticket

@tranchung163
Copy link
Contributor

.take

Copy link

github-actions bot commented Mar 5, 2024

Thank you for looking into this issue! Please let us know if you have any questions or require any help.

@p-wysocki
Copy link
Collaborator Author

Hello @tranchung163, are you still working on this? Is there anything we could help you with?

@tranchung163
Copy link
Contributor

tranchung163 commented Mar 12, 2024

Hi @p-wysocki, yes i am still working on this.

I have tried Convert a model to OpenVINO IR and benchmark for red-pajama model. This is the output I got, it seems fine to me.

Screenshot 2024-03-12 133352

But I ran into some issues with cmake files [Text_generation]
(https://github.com/openvinotoolkit/openvino.genai/tree/master/text_generation/causal_lm/cpp)

I build greedy_causal_lm <MODEL_DIR> ""

build greedy error

I tried cmake -S .\ -B .\build\ && cmake --build .\build\ --config Release -j
and got this

nor found tcmalloc

no openvino-config file in cmake list

I am trying to install all packages that are required and fix the issues. Sorry for the late response, I will try figure it out. Thanks

@p-wysocki
Copy link
Collaborator Author

@pavel-esir, @Wovchena could you please take a look?

@pavel-esir
Copy link
Contributor

@tranchung163 thanks a lot for your analysis! According to logs looks like activate setupvars.sh was not executed or it was done with errors.

Could you please try to make cd to OPENVINO_INSTALL_DIR path and call source setupvars.sh form there?

cd <OPENVINO_INSTALL_DIR>
source setupvars.sh
cd - (back to genai repo)
cmake -DCMAKE_BUILD_TYPE=Release -S ./ -B ./build/ && cmake --build ./build/ -j

For some reason setupvars.sh works correctly only if you call it from the folder it's located
image

@tranchung163
Copy link
Contributor

tranchung163 commented Mar 14, 2024

Hi @pavel-esir, thank you for your help. I set up source setupvars.sh and after building, installing tokenizers from openvino_tokenizers I was able to test beam search and greedy search. I wonder if the output should look like this ??

./build/beam_search_causal_lm
beam_search

./build/greedy_causal_lm
build greedy causal lm

@pavel-esir
Copy link
Contributor

@tranchung163 thanks for the update. Outputs do not look meaningful. Could you please play around with different questions and what are the results?

@tranchung163
Copy link
Contributor

tranchung163 commented Mar 16, 2024

Hi @pavel-esir, there is an issue while running
python3 -m pip install --upgrade-strategy eager "transformers<4.38" -r ../../../llm_bench/python/requirements.txt ../../../thirdparty/openvino_tokenizers/"[transformers]" --extra-index-url https://download.pytorch.org/whl/cpu
error with file requirement
It seems like file ./../../llm_bench/python/requirements.txt can not install auto_gptq version 0.5.1. When I changed one line in requirements.txt file with auto_gptq==0.2.0 instead of auto_gptq>=0.5.1. It works and I was able to installed all packages including auto_gpt 0.2.0. (I wonder is it because I am using MacOS M1, so I run into this error ??)

Then I run these commands without having any issue

python3 ../../../llm_bench/python/convert.py --model_id TinyLlama/TinyLlama-1.1B-Chat-v1.0 --output_dir ./TinyLlama-1.1B-Chat-v1.0/ --precision FP16
convert_tokenizer ./TinyLlama-1.1B-Chat-v1.0/pytorch/dldt/FP16/ --output ./TinyLlama-1.1B-Chat-v1.0/pytorch/dldt/FP16/ --with-detokenizer --trust-remote-code

But I build my model again with ./build/beam_search_causal_lm /opt/intel/openvino_2024.0.0/openvino.genai/text_generation/causal_lm/cpp/RedPajama-INCITE-Chat-3B-v1/pytorch/dldt/FP32/ "What is openvino?"

..And I got the same result

Screenshot 2024-03-15 at 7 21 06 PM

@pavel-esir
Copy link
Contributor

thanks for reporting the issue with installing requirements. It can be because of the M1. I'll check on x86 the versions of auto_gpt as well a bit later.

Hmm, outputs again look strange. But i noticed that you are using togethercomputer/RedPajama-INCITE-Chat-3B-v1 but issue is for another model https://huggingface.co/ikala/redpajama-3b-chat. I quickly looked and results look correct. Please check on that model and if you also get correct results add tests to .github/workflows/causal_lm_cpp.yml as well

image

@tranchung163
Copy link
Contributor

tranchung163 commented Mar 22, 2024

Hi @pavel-esir, Thank you for your feedback. I think Auto-gptq new versions do not support M1 MacOS, so the output did not make sense. After I changed my laptop, I was able to run the test and got expected result. I also switched to model ikala/redpajama-3b-chat

Screenshot 2024-03-22 at 11 40 30 AM

I am going to add tests to .github/workflows/causal_lm_cpp.yml.

@DaBaoDIY
Copy link

#WLB#+ .take

Copy link

Thanks for being interested in this issue. It looks like this ticket is already assigned to a contributor. Please communicate with the assigned contributor to confirm the status of the issue.

@p-wysocki p-wysocki linked a pull request Apr 3, 2024 that will close this issue
Wovchena added a commit that referenced this issue Apr 23, 2024
This pull request is focused on expanding the test coverage for the
[redpajama-3b-chat
](https://huggingface.co/ikala/redpajama-3b-chat)Large Language Model
(LLM) within OpenVINO GenAI. The redpajama-3b model is a significant
addition to the supported models list, and testing its functionality.
(Issue #263)

---------

Co-authored-by: Ilya Lavrenov <ilya.lavrenov@intel.com>
Co-authored-by: Zlobin Vladimir <vladimir.zlobin@intel.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
good first issue Good for newcomers
Projects
Archived in project
Development

Successfully merging a pull request may close this issue.

4 participants