Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Addition of Mistral in HF Transformers with Quality comparison README and Logging Improvements #167

Merged
merged 13 commits into from
Apr 15, 2024

Conversation

Anindyadeep
Copy link
Member

@Anindyadeep Anindyadeep commented Apr 13, 2024

This PR Adds some improvements to

  1. Base class, by fixing some bugs, and addition of utilities to make chat templates
  2. Adds Mistral support in benchmarks
  3. Quality comparison readme in engine specific readme
  4. Finally template for mistral containing Mistral specific Performance logs

Please note (Proposal):

We should Merge this PR to a different staging branch. Reason being, README will be broken after this which can also reflect in main branch. And since we changed the model from Llama 2 to LLama 2 Chat, we get bit different results. So proposal is, up next all the PRs, gets merged under staging branch. Once most of all the benchmarks are added, we can merge staging to main.


Update:

There are some improvements introduced in this PR for Logging (under utils.py) Where the main focus was the following:

  1. Small bug fixes where quality.json was taking redundant values
  2. Generates a quality.md README from quality.json. The reason being, it is super hard to copy paste the output in readme manually, so better to copy paste the generated README.
  3. Integrating those in PyTorch Benchmark

This PR also solves issues #162 #157

@Anindyadeep Anindyadeep requested a review from nsosio April 13, 2024 07:42
@Anindyadeep Anindyadeep self-assigned this Apr 13, 2024
@Anindyadeep Anindyadeep marked this pull request as draft April 13, 2024 07:44
@Anindyadeep Anindyadeep changed the title Addition of Mistral in HF Transformers with Quality comparison README Addition of Mistral in HF Transformers with Quality comparison README and Logging Improvements Apr 13, 2024
@Anindyadeep Anindyadeep marked this pull request as ready for review April 13, 2024 17:51
@nsosio
Copy link
Collaborator

nsosio commented Apr 15, 2024

@Anindyadeep I've created a dev branch where you can create all the PRs and then once everything is done we can merge dev into main. Can you update the PRs against dev?

@Anindyadeep Anindyadeep changed the base branch from main to dev April 15, 2024 07:15
@Anindyadeep
Copy link
Member Author

@Anindyadeep I've created a dev branch where you can create all the PRs and then once everything is done we can merge dev into main. Can you update the PRs against dev?

Awesome, I changed this PR to dev branch, lemme know if we can merge this one and all looks good

bench_pytorch/README.md Show resolved Hide resolved
bench_pytorch/bench.sh Show resolved Hide resolved
bench_pytorch/requirements.txt Outdated Show resolved Hide resolved
@Anindyadeep Anindyadeep requested a review from nsosio April 15, 2024 14:49
@nsosio nsosio merged commit 20ac994 into dev Apr 15, 2024
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants