Benchmark blog post #460

hassila · 2023-12-11T18:51:26Z

Benchmark blog post draft as previously agreed, I put in December 21 as publication date as a placeholder, looking forward to your feedback.

ktoso

LGTM, this is a great post (and project)!

Thank you for the post, I think it’s great — just a few minor “yay!” and rephrasing ideas inline.

_posts/2023-12-15-benchmarks.md

MahdiBM

Nice one :)

Just pointing out that the png should probably have a white/transparent background, but maintainers can let you know better if the black background is actually an issue or not 🙂

_posts/2023-12-15-benchmarks.md

Co-authored-by: Mahdi Bahrami <github@mahdibm.com>

…bsite into benchmark-blog-post

_posts/2023-12-21-benchmarks.md

finagolfin · 2024-01-06T12:18:21Z

Good post, but it would benefit from some motivation. Obviously this filled a need in your own work, maybe you could tell the story of how this package got started or some important incident where it caught some performance regression?

Such stories help motivate a post with real-world examples, would be great if you can add one or two.

hassila · 2024-01-08T09:56:46Z

Thanks @finagolfin - addressed the straightforward fixes in bd96a6e - will think if there's any good additional motivation/background we can add which doesn't just focus on our niche (tried to motivate it with the analogy to unit testing which seems fairly straightforward as an argument for the need for non-functional requirement validation).

finagolfin · 2024-01-08T10:20:09Z

Your niche is fine too, as that is a big market.

hassila · 2024-01-09T09:54:38Z

Your niche is fine too, as that is a big market.

Ok, did a stab at 43290b5 with some background.

hassila · 2024-02-02T15:44:24Z

So I guess this is stuck at review with the blog post owners, or is there anything else to do?

alexandersandberg · 2024-02-02T15:45:52Z

cc: @cthielen

cthielen · 2024-03-12T17:59:21Z

_posts/2023-12-21-benchmarks.md

+author: [hassila]
+---
+
+Have you ever encountered a _performance problem_ that slipped through to end users which resulted in a bug report? Do you systematically measure and validate performance metrics when making changes to your Swift package?


I think you could make the intro stronger by bringing up your point about the importance of unit and integration testing. The old adage "make it work, make it right, make it fast" is essentially what you're saying here. Putting that framing closer to the top will provide a better hook to encourage developers to read the rest of the post.

cthielen · 2024-03-12T18:00:44Z

_posts/2023-12-21-benchmarks.md

+layout: post
+published: true
+date: 2023-12-21 10:00:00
+title: Benchmarking in Swift


Any title alternatives that expand a bit more? e.g. "Introducing X …" or "A Quick Look …" etc.

cthielen · 2024-03-13T05:00:30Z

_posts/2023-12-21-benchmarks.md

+
+Have you ever encountered a _performance problem_ that slipped through to end users which resulted in a bug report? Do you systematically measure and validate performance metrics when making changes to your Swift package?
+
+In the realm of professional trading software, the role of a comprehensive benchmarking framework integrated with Continuous Integration (CI) parallels the importance of unit and integration testing. Just as unit and integration tests are essential for ensuring the functional correctness of software, benchmarking within a CI pipeline is crucial for continuously validating the non-functional aspects, such as high throughput, low latency, predictable performance and consistent resource usage. This is vital for maintaining the competitive edge in a fast-paced financial environment where the extreme market data rates and performance requirements means that even small variations in response time - on the scale of microseconds - can significantly impact trade outcomes.


This is a good spot to take your "Have you ever encountered a performance problem" point, i.e. it's apparent in trading systems why performance is important, but it's important to the average developer too, who doesn't want poor or regressed performance.

cthielen · 2024-03-13T05:01:21Z

_posts/2023-12-21-benchmarks.md

+
+In the realm of professional trading software, the role of a comprehensive benchmarking framework integrated with Continuous Integration (CI) parallels the importance of unit and integration testing. Just as unit and integration tests are essential for ensuring the functional correctness of software, benchmarking within a CI pipeline is crucial for continuously validating the non-functional aspects, such as high throughput, low latency, predictable performance and consistent resource usage. This is vital for maintaining the competitive edge in a fast-paced financial environment where the extreme market data rates and performance requirements means that even small variations in response time - on the scale of microseconds - can significantly impact trade outcomes.
+
+After examining the existing infrastructure within the Swift ecosystem, we concluded that there were no existing solutions meeting our needs for multi-platform and rich metrics support, CI integration, and developer-friendliness. Therefore, we decided to develop a  [Benchmark package](https://github.com/ordo-one/package-benchmark) and open-source it, believing it could help advance performance for the Swift community and benefit all of us.


Minor nit but "open source", not "open-source"

cthielen · 2024-03-13T05:02:54Z

_posts/2023-12-21-benchmarks.md

+
+### The role of benchmarks 
+
+Constructing a set of benchmarks and consistently running them provides an indication when something is not performing as expected. Typically, other more specialized tools are employed for root-cause analysis to analyze and fix the problem (e.g., Instruments, DTrace, Heaptrack, Leaks, Sample, …).


It's not clear what "other more specialized tools" is talking about. I believe the point is that software analysis requires specialized tools, e.g. memory leak detectors for memory leaks, etc., and benchmarking fits in here for performance regressions - it's another specialized tool. Better to state that plainly.

cthielen · 2024-03-13T05:07:00Z

_posts/2023-12-21-benchmarks.md

+}
+```
+
+### Performance metrics


This is a good section but it feels like a condensed version of it belongs in your introduction, not after you start diving into the package.

cthielen · 2024-03-13T05:07:41Z

_posts/2023-12-21-benchmarks.md

+  alt="Sample text output for benchmarks"
+  src="/assets/images/benchmark-blog/Benchmark.png" />
+
+### Key Benchmark workflows are supported


For better or worse, Swift.org is All Caps All The Time, e.g. "Key Benchmark Workflows Are Supported"

cthielen · 2024-03-13T05:07:56Z

_posts/2023-12-21-benchmarks.md

+
+### Key Benchmark workflows are supported
+
+* **[Automated Pull Request performance regression checks](https://swiftpackageindex.com/ordo-one/package-benchmark/documentation/benchmark/comparingbenchmarksci)** by comparing the performance metrics of a pull request with the main branch and having the PR workflow check fail if there is a regression according to absolute or relative thresholds specified per benchmark


Why are these bolded?

cthielen · 2024-03-13T05:08:42Z

_posts/2023-12-21-benchmarks.md

+* **[Export of benchmark results in several formats](https://swiftpackageindex.com/ordo-one/package-benchmark/documentation/benchmark/exportingbenchmarks)** for analysis or visualization
+* Running the Instruments profiler [on the benchmark suite executable directly from Xcode](https://github.com/ordo-one/package-benchmark/releases/tag/1.11.0)
+
+### Community Adoption


Good info, but this doesn't need its own section. Recommend folding it into your conclusion.

cthielen · 2024-03-13T05:09:35Z

_posts/2023-12-21-benchmarks.md

+
+### Summary
+
+Performance is a key non-functional requirement for many software packages. Early and continuous benchmarking will help you to ship software with consistent performance and controlled resource usage that delights your users.


I'd recommend a shorter wrap-up here with a call-to-action, e.g. "Give it a try". You don't want to be listing virtues here.

hassila · 2024-03-13T15:19:15Z

Thanks for the extensive feedback @cthielen, I have tried to address it in 4744b04

cthielen · 2024-03-19T05:25:36Z

@hassila Thank you! Are you okay if I publish this (i.e. I'll adjust the date and merge) on Wednesday, 3/20 around 10am Pacific?

hassila · 2024-03-19T05:29:39Z

Thanks @cthielen that'd be perfectly fine, thanks.

cthielen · 2024-03-19T05:53:34Z

@swift-ci please test

cthielen · 2024-03-20T17:06:07Z

@swift-ci please test

shahmishal · 2024-03-20T17:06:26Z

@swift-ci please test

* Adding placeholder for blog post + updating author file * First iteration * Link to latest documentation * Add link to Swift Certificates benchmark workflow * Add link to Instruments integration * Add link to community section at Swift forums * Clarify language * Fix SPI links to be version-independent * Addrees PR feedback * Update _posts/2023-12-15-benchmarks.md Co-authored-by: Mahdi Bahrami <github@mahdibm.com> * Address PR feedback, fix image background * Fix wording * Update _posts/2023-12-15-benchmarks.md Co-authored-by: Alexander Sandberg <hi@alexandersandberg.com> * Apply suggestions from code review Co-authored-by: Alexander Sandberg <hi@alexandersandberg.com> * Use comma consistently after e.g. * Compress png image * Feature -> Requirement * Put placeholder date into the future again * Add newline in the intro for nicer display * Put placeholder date into the future again * Address PR feedback * Address PR feedback * Add some background information on package origin as request * Add Swift Package Manager as another package that have adopted Benchmark * Fix authors for rebase * Try to address PR feedback from cthielen * Update publish date/time --------- Co-authored-by: Mahdi Bahrami <github@mahdibm.com> Co-authored-by: Alexander Sandberg <hi@alexandersandberg.com> Co-authored-by: Christopher Thielen <77445+cthielen@users.noreply.github.com>

hassila added 7 commits December 9, 2023 14:58

Adding placeholder for blog post + updating author file

67d5250

First iteration

1685f67

Link to latest documentation

fdee173

Add link to Swift Certificates benchmark workflow

c872419

Add link to Instruments integration

67bf447

Add link to community section at Swift forums

bbd20d6

Merge branch 'apple:main' into benchmark-blog-post

5392dc8

hassila requested review from 0xTim, TimTr, alexandersandberg, cthielen, daveverwer, dempseyatgithub, kaishin, mschinis, parispittman, shahmishal, tkremenek and tomerd as code owners December 11, 2023 18:51

hassila marked this pull request as draft December 11, 2023 18:53

hassila added 2 commits December 12, 2023 13:23

Merge branch 'apple:main' into benchmark-blog-post

d57e979

Clarify language

e8e09c3

hassila marked this pull request as ready for review December 12, 2023 13:34

Fix SPI links to be version-independent

484f233

ktoso approved these changes Dec 14, 2023

View reviewed changes

_posts/2023-12-15-benchmarks.md Outdated Show resolved Hide resolved

_posts/2023-12-15-benchmarks.md Outdated Show resolved Hide resolved

_posts/2023-12-15-benchmarks.md Outdated Show resolved Hide resolved

_posts/2023-12-15-benchmarks.md Outdated Show resolved Hide resolved

Addrees PR feedback

d2e7903

MahdiBM reviewed Dec 14, 2023

View reviewed changes

_posts/2023-12-15-benchmarks.md Outdated Show resolved Hide resolved

hassila and others added 3 commits December 14, 2023 12:26

Update _posts/2023-12-15-benchmarks.md

7ad534d

Co-authored-by: Mahdi Bahrami <github@mahdibm.com>

Address PR feedback, fix image background

b948b6c

Merge branch 'benchmark-blog-post' of github.com:hassila/swift-org-we…

b58de2a

…bsite into benchmark-blog-post

finagolfin reviewed Jan 6, 2024

View reviewed changes

_posts/2023-12-21-benchmarks.md Outdated Show resolved Hide resolved

finagolfin reviewed Jan 6, 2024

View reviewed changes

_posts/2023-12-21-benchmarks.md Outdated Show resolved Hide resolved

finagolfin reviewed Jan 6, 2024

View reviewed changes

_posts/2023-12-21-benchmarks.md Outdated Show resolved Hide resolved

finagolfin reviewed Jan 6, 2024

View reviewed changes

_posts/2023-12-21-benchmarks.md Outdated Show resolved Hide resolved

Address PR feedback

bd96a6e

Add some background information on package origin as request

43290b5

ktoso self-requested a review February 11, 2024 07:14

hassila added 2 commits February 26, 2024 18:05

Add Swift Package Manager as another package that have adopted Benchmark

5f21273

Merge branch 'main' into benchmark-blog-post

b5440c6

cthielen suggested changes Mar 13, 2024

View reviewed changes

hassila added 3 commits March 13, 2024 12:30

Fix authors for rebase

b15a792

Merge branch 'main' into benchmark-blog-post

beb1682

Try to address PR feedback from cthielen

4744b04

Update publish date/time

b507100

hassila requested a review from federicobucchi as a code owner March 19, 2024 05:53

cthielen approved these changes Mar 20, 2024

View reviewed changes

cthielen merged commit fcdf332 into swiftlang:main Mar 20, 2024


		Have you ever encountered a _performance problem_ that slipped through to end users which resulted in a bug report? Do you systematically measure and validate performance metrics when making changes to your Swift package?

		In the realm of professional trading software, the role of a comprehensive benchmarking framework integrated with Continuous Integration (CI) parallels the importance of unit and integration testing. Just as unit and integration tests are essential for ensuring the functional correctness of software, benchmarking within a CI pipeline is crucial for continuously validating the non-functional aspects, such as high throughput, low latency, predictable performance and consistent resource usage. This is vital for maintaining the competitive edge in a fast-paced financial environment where the extreme market data rates and performance requirements means that even small variations in response time - on the scale of microseconds - can significantly impact trade outcomes.


		In the realm of professional trading software, the role of a comprehensive benchmarking framework integrated with Continuous Integration (CI) parallels the importance of unit and integration testing. Just as unit and integration tests are essential for ensuring the functional correctness of software, benchmarking within a CI pipeline is crucial for continuously validating the non-functional aspects, such as high throughput, low latency, predictable performance and consistent resource usage. This is vital for maintaining the competitive edge in a fast-paced financial environment where the extreme market data rates and performance requirements means that even small variations in response time - on the scale of microseconds - can significantly impact trade outcomes.

		After examining the existing infrastructure within the Swift ecosystem, we concluded that there were no existing solutions meeting our needs for multi-platform and rich metrics support, CI integration, and developer-friendliness. Therefore, we decided to develop a [Benchmark package](https://github.com/ordo-one/package-benchmark) and open-source it, believing it could help advance performance for the Swift community and benefit all of us.


		### The role of benchmarks

		Constructing a set of benchmarks and consistently running them provides an indication when something is not performing as expected. Typically, other more specialized tools are employed for root-cause analysis to analyze and fix the problem (e.g., Instruments, DTrace, Heaptrack, Leaks, Sample, …).


		### Key Benchmark workflows are supported

		* [Automated Pull Request performance regression checks](https://swiftpackageindex.com/ordo-one/package-benchmark/documentation/benchmark/comparingbenchmarksci) by comparing the performance metrics of a pull request with the main branch and having the PR workflow check fail if there is a regression according to absolute or relative thresholds specified per benchmark


		### Summary

		Performance is a key non-functional requirement for many software packages. Early and continuous benchmarking will help you to ship software with consistent performance and controlled resource usage that delights your users.

Benchmark blog post #460

Benchmark blog post #460

Uh oh!

Conversation

hassila commented Dec 11, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ktoso left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

MahdiBM left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

finagolfin commented Jan 6, 2024

Uh oh!

hassila commented Jan 8, 2024

Uh oh!

finagolfin commented Jan 8, 2024

Uh oh!

hassila commented Jan 9, 2024

Uh oh!

hassila commented Feb 2, 2024

Uh oh!

alexandersandberg commented Feb 2, 2024

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

hassila commented Mar 13, 2024

Uh oh!

cthielen commented Mar 19, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

hassila commented Mar 19, 2024

Uh oh!

cthielen commented Mar 19, 2024

Uh oh!

cthielen commented Mar 20, 2024

Uh oh!

shahmishal commented Mar 20, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

8 participants

hassila commented Dec 11, 2023 •

edited

Loading

cthielen commented Mar 19, 2024 •

edited

Loading