Skip to content

Latest commit

 

History

History
151 lines (91 loc) · 6.7 KB

2021-04-19-minutes.md

File metadata and controls

151 lines (91 loc) · 6.7 KB
Raw

Embench Meeting Minutes

Monday 19 April 2021

  • 08:00-09:00 Pacific Time
  • 11:00-12:00 East Coast Time
  • 15:00-16:00 UTC
  • 16:00-17:00 UK Time
  • 17:00-18:00 Central European Time
  • 23:00-00:00 Beijing Time

Summary of actions

Avishai Tvila and Ofer Shinaar to revise their speed/rate metric proposal in the light on comments received.

Nidal Faour to review comments and bring back a proposal for how to incorporate the Western Digital code density benchmarks within the Embench family

Alexander Tessarolo to work on proposal on how to turn the TI benchmark of an AC motor model into a real-time benchmark for general use.

Agenda topics

Welcome

Avishay Tvila from Western Digital was welcomed to the meeting.

Actions from previous meeting

Ofer Shinaar to create specification of the embench rate statistic

Alexander Tessarolo to work on proposal on how to turn the TI benchmark of an AC motor model into a real-time benchmark for general use.

Speed/rate metric proposal

Following on from our last meeting, a proposal has been drafted by Avishai Tvila and Ofer Shinaar. A discussion has been started on the mailing list, see New metrics for Embench results - Base/Peak and Speed/Rate.

Avishai and Ofer presented their proposal, which was welcomed and discussed.

There was concern over accounting for warm up time. How do you account for this without masking the latency effects?

Action Avishai Tvila and Ofer Shinaar to revise their speed/rate metric proposal in the light on comments received.

GitHub pull requests

From embench-iot:

#120 Fixes issue #119

  • Commit moved to PR #131
  • closed.

#121 Update doc/README.md: add details about the section options of the size benchmark

  • Commit moved to PR #131
  • closed.

#127 Benchmarking of Macs

  • Commit moved to PR #132
  • closed.

#131 PR for issue #119

  • merged.

#132 Benchmarking of Macs (2)

  • thanks to Roger Shepherd for the big contribution.
  • merged.

New GitHub new open issues

From embench-iot:

  • #116 malloc_beebs does not ensure correct alignment of allocated objects
  • #117 [CLOSED] How to handle mach-o executables as well as elf executables
  • #118 [CLOSED] Why do we benchmark size using .text and not .text and .rodata?
  • #119 [CLOSED] bench_speed.py - text, data, rodata, bss and metric parameter handling
  • #122 [CLOSED] Linking the GCC and maths libraries needs a differnt order of arguments to GCC then generated by the buld_all.py script
  • #123 [CLOSED] Design rationale for handling of --text (etc) options in benchmark_size.py
  • #124 [CLOSED] Copyright notices
  • #125 [CLOSED] Use of lief to handle macho
  • #126 [CLOSED] Dummy library uses "unsigned int" rather than "size_t"
  • #128 [CLOSED] Problems with documentation makefile/scripts
  • #129 doc/README.md doesn't generate html header defining the character set
  • #134 Handle sections by category, not by name
  • #135 Proposal: Mechanism for per-benchmark iteration control and to clarify the purpose of cpu-mhz

There was some discussion on #135, and the need to better scale benchmarks.

Thank you to Roger Shepherd, Tariq Kurd and Edward Jones for raising these issues.

Discussion of generalizing the framework to allow it to be used with other sets of tests. Need to document sufficiently that these would not be official Embench scores.

New benchmarks for Embench IoT 2.0

Nidal Faour presented experience of developing benchmarks for exercising compilers for code size optimiztion at Western Digital. He has number of test cases, which are tiny, so might need to be combined into one single test. The benchmarks are available on GitHub as the riscv32-Code-density-test-bench.

The importance of being self testing was discussed.

Concerns were raised that it is a) based on what RISC-V does badly and b) it might not be useful for performance benchmarking.

Discussion of importance of linking and effect it has. For example on long branches. May have an impact, even on small machines, where memory is partitioned.

Comment that this feels different to other benchmarks, since it doesn't feel to have the same quality as the other benchmarks in Embench. It would be better perhaps as a specialist Embench benchmark in its own repository aimed at compiler writers.

Action Nidal Faour to review comments and bring back a proposal for how to incorporate the Western Digital code density benchmarks within the Embench family

We have proposals to add the following benchmarks for Embench IoT 2.0:

  • prime numbers from Bruce Hoult (see issue #30)
  • MD5sum and tar comparison benchmarks from Julian Kunkel (see PR #104)

Update on Embench DSP

Ray Simar reported making progress, but currently at quiet phase in semester

Update on Embench RT

Ofer Shinaar report lots of progress on operating systems context switch. Hope to have PR in two weeks, and discuss next month.

Publicity

All are invited to report back, including on opportunities for meetings where Embench should be presented.

Next meeting

We currently meet regularly on the third Monday of each month 8-9am California time.

The following dates are proposed for 2021

  • Mon 17 May 2021
  • Mon 21 Jun 2021
  • Mon 19 Jul 2021
  • Mon 16 Aug 2021
  • Mon 21 Sep 2021
  • Mon 18 Oct 2021
  • Mon 15 Nov 2021
  • Mon 13 Dec 2021 (week 2, to keep clear of Christmas)

I've created a calendar entry for our regular attendees, to provide automated reminders. Please ask if you wish to be added to this.

AOB

None.

Jeremy Bennett
Vice chair