[SR-9713] Adding a new benchmark should be easier and less error-prone

[swift-ci]

Previous ID	SR-9713
Radar	None
Original Reporter	BalestraPatrick (JIRA User)
Type	Improvement

Additional Detail from JIRA

Votes	0
Component/s	Standard Library
Labels	Improvement
Assignee	None
Priority	Medium

md5: 4c7513e1924c08dd259349acbc66153e

Issue Description:

After working on SR-8908 as part of SR-8905, I think the process of adding a new benchmark could be improved and automated. Right now, the [documentation](https://github.com/apple/swift/tree/master/benchmark#adding-new-benchmarks) mentions 3 steps in order to create and register a new single-source benchmark with the driver:

1. Add a new Swift file (YourTestNameHere.swift), built according to the template below, to the {{single-source}}directory.

2. Add the filename of the new Swift file to CMakeLists.txt

3. Edit main.swift. Import and register your new Swift module.

I'd like to see this process automated with a simple script that takes the name of a new benchmark and correctly creates, sets up the files and inserts the benchmark in the correct and sorted position for example.

Would this makes sense? I have a PR ready to be opened with a WIP implementation.

cc: @milseman @palimondo

[palimondo]

I appreciate the effort Patrick! That’s in fact how I started contributing to Swift myself: #9193
But the script-generated approach was replaced sometime last year (can’t find the exact PR at the moment) in favor of the manual approach that’s in now. Maybe @eeckstein or @gottesmm can fill in the reasons behind that…

[swift-ci]

Comment by Patrick Balestra (JIRA)

@palimondo That's interesting. I opened a PR yesterday just to make the code visible, I am happy to close if it is not needed or it's not in the direction you want to go. #22016

[palimondo]

Personally I find your PR helpful, I just meant to warn you against automating too much as the previous scripting has been curbed.

[eeckstein]

In the long term we want to switch to swiftpm for building the benchmarks, which limits the possibility of generating sources, etc.
But your script can still be useful, because it's a manual process and not invoked from the build.