Create benchmarking tools for saving run/measurement data (with Falcon7b example) and model-demo utilities for verifying tokens/perf

[skhorasganiTT]

Ticket

• Create benchmarking tools for tracking run and measurement data in tt-metal model tests #9648
• Falcon7B single/multi-chip bringup and optimization #5383

Problem description

• Currently, model tests (notably the demos, which are our key deliverables) do not have a standardized way of tracking run and measurement data. There are parallel efforts from the dev-ops and data science teams to bring up tools for monitoring such data, however a set of tools for logging and saving the data in the appropriate formats is required for model developers.

What's changed

• Created a file models/perf/benchmarking_utils.py containing tools for profiling data and saving CSVs in the appropriate formats (the requirements for these CSVs were specified by the data science team). These tools can be used for any type of test, not only demos. Two CSVs are saved for any run, "run_<start_ts>.csv" and "measurement_<start_ts>.csv".
• Created a file models/demos/utils/llm_demo_utils.py containing a function which adds demo measurements using the benchmarking tools mentioned above. This is only applicable to model demos and defines certain requirements for data produced by the demos.
• In the same llm_demo_utils.py file mentioned above, created functions for doing output token verification and output perf verification.
• Modified the Falcon7b demo to save run/measurement CSVs using the tools mentioned above.
• Added 128/2k seq len options for the Falcon7b demo (perf-mode)

Checklist

• Post commit CI passes
• Model regression CI testing passes (if applicable)
• New/Existing tests provide coverage for changes

cc @uaydonat

[skhorasganiTT]

T3k demo tests - https://github.com/tenstorrent/tt-metal/actions/runs/9668590898
All post-commit tests - https://github.com/tenstorrent/tt-metal/actions/runs/9650060689
Docs tests - https://github.com/tenstorrent/tt-metal/actions/runs/9685787015

[tt-rkim]

Two things before I approve:

• resolve all convos I started
• could you please show that these are generated on the demo CI, and upload them as an artifact? generated/benchmarks should be a good path to upload as an artifact

[skhorasganiTT]

Have you considered using https://pypi.org/project/pytest-benchmark/ and build extra code on top of it ?

Thanks for sharing that Bill, I hadn't considered it. It seems like a good tool for benchmarking functions/tests, but I don't think it will be easy to integrate with our requirements for the CSVs since we require a very specific format for timestamps (which is why I added the BenchmarkProfiler) and we will need to benchmark blocks of code, not only functions.

[eyonland]

I don't see any tests with the following decorator/annotation tags. We are trying to make sure that all demos collect metrics.
Look for existing tests with these markings.
@pytest.mark.models_device_performance_bare_metal
@pytest.mark.models_performance_bare_metal

Note that we document this here: https://tenstorrent.github.io/tt-metal/latest/ttnn/ttnn/demos.html
See for example:
tests/ttnn/integration_tests/resnet/test_performance.py

[skhorasganiTT]

Hi @eyonland, those decorators are used to specify which tests should be included in perf pipelines. The demos belong to a separate pipeline and should not use them. Also, the decorators of existing demo tests is orthogonal to this PR and not affected. The purpose of this PR is to create new benchmarking tools for measuring metrics that will be initially adopted by the demos and subsequently by other tests.

[TT-billteng]

Have you considered using https://pypi.org/project/pytest-benchmark/ and build extra code on top of it ?

Thanks for sharing that Bill, I hadn't considered it. It seems like a good tool for benchmarking functions/tests, but I don't think it will be easy to integrate with our requirements for the CSVs since we require a very specific format for timestamps (which is why I added the BenchmarkProfiler) and we will need to benchmark blocks of code, not only functions.

you could move those blocks of code into functions :)

[skhorasganiTT]

you could move those blocks of code into functions :)

We may want to log many steps (such as in the demo) so making everything a function might be overkill (plus the point about the timestamps).

[eyonland]

Hi @eyonland, those decorators are used to specify which tests should be included in perf pipelines. The demos belong to a separate pipeline and should not use them. Also, the decorators of existing demo tests is orthogonal to this PR and not affected. The purpose of this PR is to create new benchmarking tools for measuring metrics that will be initially adopted by the demos and subsequently by other tests.

Could we have a meeting on this? It sounds like we want to deviate from the expectations of demos as described here: https://tenstorrent.github.io/tt-metal/latest/ttnn/ttnn/demos.html
@uaydonat is this a shift in expectations on demos and how we track their e2e and device performance going forward?

[eyonland]

@skhorasganiTT / @uaydonat , based on our conversation, feel free to add the update to this PR for how this new pipeline fits into the docs here: https://tenstorrent.github.io/tt-metal/latest/ttnn/ttnn/demos.html
Thanks for the clarity on this.