Add HuggingFace Llama3.2 1B to benchmark

[guangy10]

Add llama3.2 1b from Hugging Face to benchmark w/ the following configs:

• SpinQuant
• QAT+Lora
• Original BF16

Switched to use the memory intensive runners in the benchmark workflow to reduce operation cost.

[pytorch-bot]

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/5368

• 📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

❌ 1 New Failure, 1 Cancelled Job

As of commit e2779ee with merge base 8460d42 ():

NEW FAILURE - The following job has failed:

• pull / unittest / macos / macos-job (gh)
  backends/xnnpack/test/ops/test_conv1d.py::TestConv1d::test_qs8_conv1d_batchnorm_seq

CANCELLED JOB - The following job was cancelled. Please retry:

• Check Labels (gh)

This comment was automatically generated by Dr. CI and updates every 15 minutes.

[facebook-github-bot]

@guangy10 has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

[facebook-github-bot]

@guangy10 has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

[guangy10]

Upload model artifacts to GitHub is skipped https://github.com/pytorch/executorch/actions/runs/10858058150/job/30136354800. Don't see the reason behind from the log. The model artifacts are placed under artifacts-to-be-uploaded/google/gemma-2b_xnnpack/model.zip @huydhn Any clue why isn't uploaded? IIUC you mentioned any subdirectry under artifacts-to-be-uploaded/ will be uploaded right?

[huydhn]

Upload model artifacts to GitHub is skipped https://github.com/pytorch/executorch/actions/runs/10858058150/job/30136354800. Don't see the reason behind from the log. The model artifacts are placed under artifacts-to-be-uploaded/google/gemma-2b_xnnpack/model.zip @huydhn Any clue why isn't uploaded? IIUC you mentioned any subdirectry under artifacts-to-be-uploaded/ will be uploaded right?

Oops, the size of the export model is 11+ GB I think. I think uploading such large file to GH is taking too long and the job timed out.

2024-09-14T02:21:26.5056615Z + ls -All ./gemma-2b_xnnpack_fp32.pte
2024-09-14T02:21:26.5057369Z -rw-r--r--. 1 ci-user ci-user 12122356576 Sep 14 01:43 ./gemma-2b_xnnpack_fp32.pte

I think I need to rework the upload part here as GH doesn't scale, so we need to go straight to S3.

[huydhn]

@guangy10 You could try to rebase the PR now and re-run the test now that #5375 has been merged

[facebook-github-bot]

@guangy10 has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

[facebook-github-bot]

@guangy10 has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

[guangy10]

Tried running the gemma-2b on Google Pixel 8 Pro (w/ 12GB RAM). The failure is same. Some I/O failures when connecting the device in the pool: https://github.com/pytorch/executorch/actions/runs/10908663134/job/30277474048. In the stacktrace I see there is a call extra_data_arn = upload_file( trying to upload the extra_data_archive, I'm wondering if it's exceed the set limit because the file is +10GB. cc: @huydhn

[huydhn]

I'm checking AWS doc on this https://docs.aws.amazon.com/devicefarm/latest/developerguide/limits.html and it mentions a 4GB limit, but that's for the size of the app, not the extra data archive. Let me run this manually using AWS UI and see if it accepts the model.

The archive size is 5.4 GB https://github.com/pytorch/executorch/actions/runs/10908663134/job/30278173066#step:11:38. IIRC, llam2 7b works but it's only ~3GB

[guangy10]

SpinQuant and QLORA are passing.

• job link: https://github.com/pytorch/executorch/actions/runs/12379908626/job/34556180815
• view on dashboard

[guangy10]

Orignal BF16 is passing:

• job link: https://github.com/pytorch/executorch/actions/runs/12380832321/job/34559109226
• dashboard

[facebook-github-bot]

@guangy10 has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

[guangy10]

Decided to leaving the logics of running the 1b model in scheduled jobs in a separate PR to simplify the review as it will require significant refactoring in the workflow.

[huydhn]

Just a data point that I see your test run shows up on the dashboard at https://hud.pytorch.org/benchmark/llms?startTime=Wed%2C%2011%20Dec%202024%2004%3A07%3A58%20GMT&stopTime=Wed%2C%2018%20Dec%202024%2004%3A07%3A58%20GMT&granularity=hour&lBranch=add_hf_model_to_benchinfra&lCommit=e2779ee5cbe666072a2d0f7a6821d640a11d1ad9&rBranch=add_hf_model_to_benchinfra&rCommit=e2779ee5cbe666072a2d0f7a6821d640a11d1ad9&repoName=pytorch%2Fexecutorch&modelName=All%20Models&backendName=All%20Backends&dtypeName=All%20DType&deviceName=All%20Devices and the extraction logic looks wrong in which llama model has the backend and benchmark configs swapped. I guess this is what you means by introducing the new benchmark_config. So, the https://github.com/pytorch/executorch/blob/main/.github/scripts/extract_benchmark_results.py script will need to be updated accordingly I guess.