Changes / Updates / Dates for Round 22

[NateBrady23]

Round 21 has concluded. The results will be posted shortly. We'd like to have a quicker turnaround for Round 22. We're hoping between Oct - Nov.

Checklist

• Update Citrine to Ubuntu 22.04 #7321

• Verify routing matches exact path #6967 (comment)

• Requirements in Fortunes test  #6883

• Some fraweworks don't follow the rules #6788

• [Requirements] Add that Response cache is not permitted in Fortunes test #6529

[billywhizz]

i was just reading a thread on reddit about latest results and a commenter mentioned that "Stripped" scores are included in the composite results. I didn't think this was allowed/possible, but it turns out this is in fact the case, for actix at least.

the /json and /plaintext results in the composite scores for actix are from the "server" configuration which is marked as "Stripped". is this correct or a mistake in the collation of the results?

[Yurunsoft]

The sql queries currently tested are returning too fast.
In the production environment, the table may have a lot of data, the query will not return so fast, and there are even slow queries.
Suggest adding slow IO test.

#6687

---

It is suggested to add memory usage indicator as one of the scoring criteria.

[NateBrady23]

@billywhizz You're correct. That shouldn't be the case. That implementation approach was changed and never updated for the composite scores. Will see what I can do.

[billywhizz]

@nbrady-techempower yes - i didn't think it was possible when i saw the comment so glad i checked. by my calculation this would give actix a composite score of 6939, moving it down to ninth place behind officefloor, aspnet.core, salvo and axum. i haven't checked if same is happening with any others.

[joanhey]

It is not problem only from the TechEmpower people.
We need to help all together for the health of the benchmark.

A lot of people don't understand a benchmark.
They don't use it for refactor their frameworks to be faster and learn from others.
They use it for make tricks and look faster, and not useful in a production app.
But they use it like Bench Marketing only.

[joanhey]

When I said that we need to clarify the rules is for that:

We are changing the rules for that people, but all need to follow the rules.

It's like with Faf #7402, we all can learn from that, for good or bad.
But nobody say anything about the process priority, it's OK but all or no fws. #6967 (comment)

For some time, the length of the server name is discussed but still without a solution. Before was a problem with the urls.
It's only some bytes, but for 7 millions req/s is enough difference.

etc, etc
But need to be in the rules, later they will be also tested.
What is not possible, is that because it is not in the rules, you can't report a fw. Like caching the response in fortunes.

[joanhey]

Also I want to see which frameworks are using pipeline in plaintext.
The same than we look for ORM or raw.

[joanhey]

Another big problem, that a lot of us have it is the servers.

We have enough information to inform the Ops.
From round 18 to 19, we all see a big drop in performance with the mitigations for spectre/meltdown.
If we change to the new kernel or new CPUs, we can help with the results to many company's.

Only with the kernel change #7321 or new servers.
Or with the new kernel with the MGLRU, or the patch for older.

They make a very big impact, more than the fw that we use.
And we need to inform and help about that.

[joanhey]

Another question,
It's possible to hardcode the Content-Lenght: ??
For me it's a stripped version.

[joanhey]

@billywhizz
always will be critics about any benchmark, but we need that be as few as possible.

[joanhey]

More questions

Is it realistic to have different variants and configs for every test??
Some use one for plaintext and json, and another for the db tests.
But some have 1 for plaintext, 1 for json, 1 for updates, ....
Anybody will do that for an app ??

If the fw is using JIT is very beneficial, but not realistic.
Also some are talking about that #7358.

[billywhizz]

@joanhey yes - i tend to think there should be a single entry/configuaration allowed per framework and it should be the same codebase that covers all the tests. this would be much more "realistic" and would also massively decrease the amount of time a full test run takes - some frameworks have 10 or more different configuarations that have to be tested!

[joanhey]

I understand variants for different db or driver.
But not per test (JIT) and also specific config.

In the same way, some fws use only 1 variant but the config to the bd is different for every test.
How anybody will do that in a real app ??

[billywhizz]

@joanhey good point re. different databases - but apart from that i think the number of configurations per framework should be minimised. i also think it would work better overall if a run was only triggered when a framework changed rather than continually running every framework end to end. if we only ran on every merge just for the changed framework then maintainers would have to wait a lot less time to see results of changes.

my worry about introducing too many and too complex rules is it will just discourage people from entering at all, so there is a balance to be found between too many rules and allowing for innovation in the approaches.

[franz1981]

Shared my thoughts on #7475 (comment) here #7358 (reply in thread) trying to both give my opinion but still convey that there are things, assuming what the purpose of the benchmark is, that should change a bit... although I agree with #7475 (comment) to not making it too complex to avoid folks to not get in.

[joanhey]

We can create an addendum, for those 1-2% devs who try to lie. With the more esoteric tricks.

About the run only with the changed frameworks.
The bench not only help to make faster code, it's also very useful to find bugs.
Now I'm searching for one with PHP8.1 and JIT, It's an intermittent bug.
Without change the code, not the php version, there are a ~15% drop from run to run.

[cirospaciari]

We can create an addendum, for those 1-2% devs who try to lie. With the more esoteric tricks.

About the run only with the changed frameworks. The bench not only help to make faster code, it's also very useful to find bugs. Now I'm searching for one with PHP8.1 and JIT, It's an intermittent bug. Without change the code, not the php version, there are a ~15% drop from run to run.

@nbrady-techempower
Robyn uses "const" with basically caches the string in Rust and avoid calling python at all, i think its not allowed right?
https://sansyrox.github.io/robyn/#/architecture?id=const-requests

https://github.com/TechEmpower/FrameworkBenchmarks/blob/master/frameworks/Python/robyn/app.py

[romanstingler]

Is it possible to get the tested framework version?

is it just me or should people be allowed to state
approach = "Realistic"
only when they use framework built-in functions/methods i.e. not allowed to rewrite the router, not allowed to write a custom JSON function, as this is not a realistic approach for each framework.
Otherwise, it is just a programming language benchmark with pulled in function of frameworks and minimal use of them or even overwriting them.

[volyrique]

The result visualization for every round has a link to the continuous benchmarking run that has been used (for example, round 21 is based on run edd8ab2e-018b-4041-92ce-03e5317d35ea). From the run you can get the commit ID, so that you can browse the repository at the respective revision. Then check the Dockerfile that corresponds to the test implementation you are interested in (and possibly any associated scripts in the implementation directory) to get the framework version that has been used. Unfortunately not all implementations keep their dependencies locked down properly - in that case your best bet is probably to check the build logs from the run. If that does not help, then I am afraid that other than making a guesstimate, you are out of luck.

[fafhrd91]

Hi, @nbrady-techempower do you guys decide on dates for Round 22?

[volyrique]

@nbrady-techempower I noticed an issue that seems to have appeared back in December after continouous benchmarking started running properly again - the Dstat data is missing.

[NateBrady23]

@fafhrd91 Nothing concrete yet. I've got to get in front of the servers and do some upgrades. I'd like to shoot for late March.

@volyrique Thanks, I'll take a look!

[volyrique]

@nbrady-techempower I had a closer look at the Dstat issue and it looks like a common problem. Unfortunately the tool appears to be unmaintained, and the closest thing to a drop-in replacement seems to be Dool.

[graemerocher]

@nbrady-techempower if there is not going to be an update to the benchmarks soon can you please remove micronaut from the dataset? Some random non-micronaut maintainer submitted a PR that was merged without the consent of the maintainers and artificially crippled our results (limited the connection pool size to 5) and we have had to live with people enquiring why it is so low in the results.

People unfortunately use these benchmarks to make technology decisions and if the data is wrong for long periods it is impacting us directly.

[NateBrady23]

@graemerocher I'm sorry to hear about this. I can get the round 21 results from micronaut removed as soon as I'm back next week. I'm having a hard time finding the commit for that. Would you mind linking it so I can see other activity from that user.

[graemerocher]

@nbrady-techempower the history of what happened is in this thread #7618

Thanks for helping.

[spericas]

@nbrady-techempower Is there a tentative date for Round 22?

[NateBrady23]

It's been almost a year since I said I'd like to start having more regular rounds... 😵‍💫

So, I think the biggest thing here was getting Citrine updated. Though I think all the things on the checklist are important, clarifying rules is a never-ending process, and if anyone thinks there's some in clear violation of any rules, please open a PR or an issue and ping me and the maintainers. Otherwise, I think we'll shoot for the first complete run in August.

[shaovie]

Good anticipation

[graemerocher]

@nbrady-techempower requesting this again as we keep getting questions. Please remove the invalid round 21 results for micronaut

[NateBrady23]

@graemerocher This is done.

Please note that we have added a "maintainers" property in the benchmark_config.json. Feel free to add yourself and any others that you want to be notified in case changes are made to your tests. It's impossible for us to track who the maintainers are for each project, so this will help catch this issue in the future.

https://github.com/TechEmpower/FrameworkBenchmarks/wiki/Codebase-Framework-Files

everyone:

We're waiting on a few missing pieces for the new rack in the new place. We're hoping to be up and running on Thursday.

[graemerocher]

Thanks

[alfiver]

May I ask when the new round of benchmark results will be released?

[NateBrady23]

@alfiver As soon as we get Citrine back online we're going to do a few full runs and then the next round. We'll be investigating the issues this week and have an update for everyone around Thursday.

[NateBrady23]

Sorry, folks. No good news yet. One of the machines won't stay on. We're looking into it and will update when we can.

[NateBrady23]

Ok, we got lucky! It was just the system battery. The servers are moved to their new home and looks like we're back up and running. Going to finish a full run to make sure everything looks good and then we'll send out a notice for when we're locking PRs for the this round.

[fakeshadow]

i was just reading a thread on reddit about latest results and a commenter mentioned that "Stripped" scores are included in the composite results. I didn't think this was allowed/possible, but it turns out this is in fact the case, for actix at least.

the /json and /plaintext results in the composite scores for actix are from the "server" configuration which is marked as "Stripped". is this correct or a mistake in the collation of the results?

I know this is a known issue and just want to point out the same is happening to xitca-web too. An unrealistic benchmark is counted towards total composite score unfortunately. It would be best if the misleading can be fixed in an official run. If a quick fix is not possible I suggest both xitca and actix mark their unrealistic bench as "broken" temporary until round 22 is finished.

[shaovie]

Ok, we got lucky! It was just the system battery. The servers are moved to their new home and looks like we're back up and running. Going to finish a full run to make sure everything looks good and then we'll send out a notice for when we're locking PRs for the this round.

Excuse me, do you have any good news?

[NateBrady23]

Unfortunately we had some other issues come up, but hopefully resolved. The latest is here.

[NateBrady23]

With the last run looking back to normal, it's time to actually set some dates for Round 22!

The run in progress will complete around 9/26. The following complete run will be a preview run. And we'll look to start the round run on 10/3.

We normally lock PRs down during the preview run. I would caution any maintainers on making adjustments to their frameworks during that time. As a reminder, we don't rerun individual frameworks for completed runs.

[CosminSontu]

Please wait for .NET 8 LTS release if it's not already accounted for. The release is planned for 14th of November this year.
Worst case scenario, please use a release candidate of .NET 8. Thanks!

[joanhey]

Wait, for the next run, than you don't know the results ....

[NateBrady23]

We had an internet outage that looks like it stopped the preview round run / communication to tfb-status. I'll be in the office tomorrow to see if the preview round completed successfully and kick of the official round.

[NateBrady23]

Someone was able to restart the service. Since the preview round wasn't able to complete, we'll do one more preview round and move the Round 22 official run to start around Oct 11th.

[graemerocher]

could someone review and merge #8478 before the run. Thanks.

[macel94]

Someone was able to restart the service. Since the preview round wasn't able to complete, we'll do one more preview round and move the Round 22 official run to start around Oct 11th.

@nbrady-techempower any news?

:D

[p8]

@macel94 It's running: https://tfb-status.techempower.com/