External Source Code: Integrate Coremark as a module

[carlescufi]

Origin

Coremark, by the Embedded Microprocessor Benchmark Consortium:
https://github.com/eembc/coremark

Purpose

Coremark is an industry standard for CPU benchmarking.

Mode of integration

Would be integrated as a module, upstream already contains the zephyr/ folder so I just suggest forking the coremark repo into the Zephyr GH org.

Maintainership

@volodymyr-bondarchuk
@carlescufi

Pull Request

#55461

Description

The Coremark library is very useful to those wanted to evaluate the performance and energy efficiency of a particular SoC. It allows users to compare different silicon vendors using the same standardized benchmark.

Goals: Benchmarking of MCUs without the hassle of porting the benchmarking software, using Zephyr's portability.

Coremark is the industry standard, it's maintained by a non-profit consortium and is small in code size and complexity.

Dependencies

None

Revision

d5fad6bd094899101a4e5fd53af7298160ced6ab

License

There is an Acceptable User Agreement, that covers the Coremark trademark, but it clearly states that "Licensor offers benchmarking software (“Software”) pursuant to an open source license".
All source code is provided under the Apache v2 license.

Apache-2.0

[volodymyr-bondarchuk]

PR #55461 with Coremark bring up created.

[stephanosio]

Re: Standalone application vs Zephyr module

#55461 (review)

As previously mentioned in the Discord thread, this is arguably better off as a standalone application (like the example-application) since coremark is not really a component that can be reused by applications (i.e. not a library).

But, I can also see the value of having this as an upstream test/sample in order to take advantage of the upstream CI and testing process to ensure that coremark can (ideally) run on all upstream Zephyr boards, since coremark is a quite popular benchmark suite that can be used for various purposes (e.g. Zephyr performance sanity check, performance comparisons among supported Zephyr boards and configurations, etc.).

Similar attempt, but as a standalone application: https://github.com/claziss/coremark-pro-zephyr/tree/main

[ruuddw]

Similar attempt, but as a standalone application: https://github.com/claziss/coremark-pro-zephyr/tree/main

Note that that's for a subset of CoreMark-PRO, that's not the same as the base CoreMark.

[galak]

I don't think this belongs in Zephyr. Coremark glue code for Zephyr we should try and upstream to Coremark. As Coremark is primarily benchmarking the CPU hardware and not RTOS I don't think Zephyr should be including.

[abrodkin]

I don't think this belongs in Zephyr. Coremark glue code for Zephyr we should try and upstream to Coremark. As Coremark is primarily benchmarking the CPU hardware and not RTOS I don't think Zephyr should be including.

That's for sure, all needed changes need to go to the upstream Coremark* projects.
Just in case, since we've spent quite some time looking at various Coremark peculiarities, I'd say CoreMark Pro (https://github.com/eembc/coremark-pro) is a bit more appropriate choice for Zephyr because it naturally supports multithreaded execution via a simple pthread API, while still having exact same code as an olde good original CoreMark in one of its sub-tests https://github.com/eembc/coremark-pro/tree/main/benchmarks/core.

So maybe consider CoreMark Pro instead as it includes the original CoreMark and adds more benchmarks in addition.

[carlescufi]

Removed the TSC label until we decide how to proceed with this internally at Nordic.

[carlescufi]

closing this, not pursuing it for now.