Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

DistilBert gs demo benchmarking #4482

Closed
Sudharsan-V opened this issue Dec 22, 2023 · 2 comments
Closed

DistilBert gs demo benchmarking #4482

Sudharsan-V opened this issue Dec 22, 2023 · 2 comments
Assignees
@Sudharsan-V @saichandax @kkeerthana0573
Labels
models Models that run in tt-metal

Comments

@Sudharsan-V
Copy link
Contributor

GS e2e Demo

test uses batched real inputs and produces batched outputs

this test is looped (takes several batches of inputs, produces several batches of outputs)

test evaluates results of TT model against validation data in selected data set

test is documented

how to run test (commands to execute)
what does the model do (high-level explanation in a couple of sentences)
description of what is input and what is output
this demo should load inputs and expected outputs from a data set on Weka (TBD which one)
@Sudharsan-V Sudharsan-V added the models Models that run in tt-metal label Dec 22, 2023
kkeerthana0573 added a commit that referenced this issue Dec 26, 2023
@kkeerthana0573
#4482: DistilBert gs demo benchmarking
6521b33
kkeerthana0573 added a commit that referenced this issue Dec 26, 2023
@kkeerthana0573
#4482: DistilBert gs demo benchmarking
0487dd1
kkeerthana0573 added a commit that referenced this issue Dec 27, 2023
@kkeerthana0573
#4482: DistilBert gs demo benchmarking
1ebd981
@saichandax saichandax self-assigned this Dec 27, 2023
@kkeerthana0573
Copy link
Contributor

kkeerthana0573 commented Dec 27, 2023 •
edited
Loading

This commit contains benchmarking of Distilbert for Question and Answering using SQUAD dataset.

  • Exact Match : 62.0
  • F1 Score : 74.24285714285713

Corresponding commit - commit
Corresponding PR - #4510

kkeerthana0573 added a commit that referenced this issue Jan 4, 2024
@kkeerthana0573
#4482: DistilBert gs demo benchmarking
7237665
@Sudharsan-V Sudharsan-V mentioned this issue Jan 4, 2024
Open
17 tasks
kkeerthana0573 added a commit that referenced this issue Jan 5, 2024
@kkeerthana0573
#4482: DistilBert gs demo benchmarking
79ea8cf
kkeerthana0573 added a commit that referenced this issue Jan 8, 2024
@kkeerthana0573
#4482: DistilBert gs demo benchmarking
d041471
kkeerthana0573 added a commit that referenced this issue Jan 8, 2024
@kkeerthana0573
#4482: DistilBert gs demo benchmarking
ba49ed5
@Sudharsan-V
Copy link
Contributor Author

Distilbert GS demo Benchmarking is completed and it is merged with mail. All the CI tests passed.

muthutt pushed a commit that referenced this issue Jan 8, 2024
@kkeerthana0573 @muthutt
#4482: DistilBert gs demo benchmarking
4e6d610
muthutt pushed a commit that referenced this issue Jan 8, 2024
@kkeerthana0573 @muthutt
#4482: DistilBert gs demo benchmarking
93cdabe
muthutt pushed a commit that referenced this issue Jan 8, 2024
@kkeerthana0573 @muthutt
#4482: DistilBert gs demo benchmarking
ea11305
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@Sudharsan-V Sudharsan-V

@saichandax saichandax

@kkeerthana0573 kkeerthana0573

Labels
models Models that run in tt-metal
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@Sudharsan-V @saichandax @kkeerthana0573