Skip to content

Conversation

@kirklandsign
Copy link
Contributor

@kirklandsign kirklandsign commented Sep 13, 2024

ns * 1e-6 = ms

Example output:
(9.36ms inference mv2_xnnpack)

[{"actualValue":9.3611927,"benchmarkModel":{"backend":"","name":"mv2_xnnpack","quantization":""},"deviceInfo":{"arch":"SM-S926U1","availMem":0,"device":"samsung","os":"Android 14","totalMem":0},"metric":"avg_inference_latency(ms)","targetValue":0.0},{"actualValue":27.418698,"benchmarkModel":{"backend":"","name":"mv2_xnnpack","quantization":""},"deviceInfo":{"arch":"SM-S926U1","availMem":0,"device":"samsung","os":"Android 14","totalMem":0},"metric":"model_load_time(ms)","targetValue":0.0},{"actualValue":0.0,"benchmarkModel":{"backend":"","name":"mv2_xnnpack","quantization":""},"deviceInfo":{"arch":"SM-S926U1","availMem":0,"device":"samsung","os":"Android 14","totalMem":0},"metric":"load_status","targetValue":0.0}]

and for LLM
(12.34 tps mocked number)

[{"actualValue":0.0,"benchmarkModel":{"backend":"","name":"et_exported_llama","quantization":""},"deviceInfo":{"arch":"SM-S926U1","availMem":0,"device":"samsung","os":"Android 14","totalMem":0},"metric":"load_status","targetValue":0.0},{"actualValue":3375.450312,"benchmarkModel":{"backend":"","name":"et_exported_llama","quantization":""},"deviceInfo":{"arch":"SM-S926U1","availMem":0,"device":"samsung","os":"Android 14","totalMem":0},"metric":"model_load_time(ms)","targetValue":0.0},{"actualValue":8220.467966,"benchmarkModel":{"backend":"","name":"et_exported_llama","quantization":""},"deviceInfo":{"arch":"SM-S926U1","availMem":0,"device":"samsung","os":"Android 14","totalMem":0},"metric":"generate_time(ms)","targetValue":0.0},{"actualValue":12.3456,"benchmarkModel":{"backend":"","name":"et_exported_llama","quantization":""},"deviceInfo":{"arch":"SM-S926U1","availMem":0,"device":"samsung","os":"Android 14","totalMem":0},"metric":"token_per_sec","targetValue":0.0}]

@pytorch-bot
Copy link

pytorch-bot bot commented Sep 13, 2024

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/5362

Note: Links to docs will display an error until the docs builds have been completed.

✅ No Failures

As of commit 11d53dd with merge base 034e098 (image):
💚 Looks good so far! There are no failures yet. 💚

This comment was automatically generated by Dr. CI and updates every 15 minutes.

@facebook-github-bot facebook-github-bot added the CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. label Sep 13, 2024
@facebook-github-bot
Copy link
Contributor

@kirklandsign has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

results.add(
new BenchmarkMetric(
benchmarkModel,
"model_load_time(ns)",
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

still ns?

results.add(
new BenchmarkMetric(
benchmarkModel,
"model_load_time(ns)",
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit: The metric value is now in (ms) instead of (ns), so its name need to be updated accordingly

results.add(
new BenchmarkMetric(
benchmarkModel,
"generate_time(ns)",
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same comment about the metric name

Copy link
Contributor

@huydhn huydhn left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Stamped!

results.add(
new BenchmarkMetric(
benchmarkModel,
"generate_time(ns)",
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ditto

@facebook-github-bot
Copy link
Contributor

@kirklandsign has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

@facebook-github-bot
Copy link
Contributor

@kirklandsign merged this pull request in 62024d8.

@kirklandsign kirklandsign deleted the android-number-ms branch September 13, 2024 22:57
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. Merged

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants