Multi year old Issues & PRs - Is BenchmarkDotNet in maintenance mode?

[agra6475]

I am having a hard time knowing the status of certain issues with attached PRs.
I will take these three PRs as examples:

#2111 No comments for the past two years. No status or comments in PR indicating whether it will ever be merged.
#2109 No comments at all.
#2563 No comments at all.

Perhaps discussion is ongoing outside of Github which would explain why there is no comments. That still does not explain why the issues attached to the PRs have been open for years.

Would be nice with updates on these issues & PRs whenever possible so users of BenchmarkDotNet can be aware of what the plan is. Then we know what bugs we permanently have to work around and which ones are only temporary.

It's not great when a users of a project starts talking about which bugs are permanent and which are temporary. That is why the title also asks if BenchmarkDotNet is in maintenance mode.

[adamsitnik]

To tell the long story short none of us (maintainers) is being paid to work on BenchmarkDotNet. We contribute in our free time.

In my case I always estimate what impact given bug has (is it common, how many users hit it), how complex is the fix (what risks it brings, is it going to do more harm than good) and if we should invest in given direction at all (when something is bad by design, I won't invest my free time into that).

To be specific:

• Reduce async overhead #2111 is fixing a bug with zero up-votes, introduces a lot of complexity and can affect the time reported by BDN which would affect all the users that have been running their benchmarks continuously and storing the results somewhere and using those results to detect regressions.
• Fix async GlobalSetup/Cleanup with InProcessEmit toolchain #2109 is fixing a bug with 10 upvotes (it's common) and it's quite simple, but it's investing time in InProcessToolchains which I believe is wrong because they are hard to keep in sync with other toolchains, can lead into bugs (no process level isolation) and should be replaced by source generator toolchain. But it maybe simple enough to be worth it, I am going to review it later today.
• Fix parallel builds #2563 - it brings a lot of complexity in an area where similar changes has brought new bugs in the past. Risk is too high for me and again, if we invest in source generator toolchains we won't need that.

[timcassell]

• Fix parallel builds #2563 - it brings a lot of complexity in an area where similar changes has brought new bugs in the past. Risk is too high for me and again, if we invest in source generator toolchains we won't need that.

Just a small correction about that. Source generator toolchains would not solve that particular issue. It's a quirk of MSBuild that is fixed by using <ArtifactsPath>. We would still have to rely on MSBuild for source generators if any build settings are modified per benchmark job (unless the source generator toolchains won't support custom build settings).

Also, the PR only got so complex because of the toolchains that it failed with in CI that I was unable to debug. If I am ever able to run those toolchains on my local machine, I would hopefully be able to fix them and simplify the complexity. Another option would be to re-open #2426, which is much simpler.

[agra6475]

@adamsitnik Thank you for the answer. I am surprised maintainers are not being paid.

I think i understand how you are prioritizing reviewing PRs. I want to explain what it looks like and how it affects an outsider.

I've made extensive use of BenchmarkDotNet for a while now.
I've wanted to contribute to this project for a while now but i've never done it. The reason has been the many year old open PRs with no responses. Why spend my time on a PR if no one will even look at it?

I think it's unfortunate because it probably demotivates quite a lot of people from contributing to the project.
Not sure what to do about it though. I can't ask you or any other maintainers to do more for the project since you aren't being paid to maintain it.

In any case thank you for reviewing #2109 i can finally get rid of a workaround to this issue in a bunch of benchmarks.
Thank you @timcassell for making all the PRs in the first place :)

Regarding source generators, is there any available plan or idea about how to use it in BenchmarkDotNet? I would love to see what the future might bring if possible.

[timcassell]

@agra6475 Please do send PRs. I am actively reviewing and merging PRs that folks have been sending. I just can't merge my own without another maintainer's review. 😉