Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Changes / Updates for Upcoming Round #4630

Open
nbrady-techempower opened this issue Apr 9, 2019 · 25 comments

Comments

Projects
None yet
10 participants
@nbrady-techempower
Copy link
Member

commented Apr 9, 2019

I'll be editing this pinned issue to link to other issues or pull requests that may affect tests for this round.

We are working toward Round 18 now. Official results have been captured and will be available on the results website soon.

#4291 - Responses including cached date header
#4493 - Limit the amount of memory the framework container can use
#4601 - Cached queries tests were using the wrong levels
#4622 - Stricter Multiple Query test verification
#4636 - Enable accelerated networking in Azure
#4642 - PostgreSQL bumped to v11
#4834 - Actix fortunes explanation of results

We've also clarified some rules:

  • Use of single update statements that affect multiple rows is acceptable
  • Use of caching/memoization within or on top of JSON serializers is not allowed.

Tests that were in violation of the rules when captured:

@nbrady-techempower nbrady-techempower pinned this issue Apr 9, 2019

@mkurz

This comment has been minimized.

Copy link
Contributor

commented Apr 28, 2019

Any estimation when you will start the round?
BTW: https://tfb-status.techempower.com/ is down...

@nbrady-techempower

This comment has been minimized.

Copy link
Member Author

commented Apr 29, 2019

We'd like to pull the next round from continuous runs this week. Looks like there was an outage at the office this weekend; we'll get things back online Monday morning.

@mkurz

This comment has been minimized.

Copy link
Contributor

commented Apr 29, 2019

@nbrady-techempower Thanks. Can you please merge #4671 so it makes it into the next round? Thanks!

@volyrique

This comment has been minimized.

Copy link
Contributor

commented Apr 29, 2019

This probably got lost before, but I found another difference between the Terraform scripts and the Azure portal defaults - the OS disk type is "Standard HDD" instead of "Standard SSD". Are you going to change that too in the upcoming Azure runs?

@nbrady-techempower

This comment has been minimized.

Copy link
Member Author

commented Apr 29, 2019

@volyrique I want to point out that we're not necessarily shooting for the most optimized environment for azure; I think some of the point to the azure tests is to see how tests perform in a less optimized environment. With that being said, I believe we're using the temp storage drive in azure which is SSD (https://docs.microsoft.com/en-us/azure/virtual-machines/windows/sql/virtual-machines-windows-sql-performance#temporary-disk) I'll make sure to get the final specs posted soon.

@fafhrd91

This comment has been minimized.

Copy link
Contributor

commented Apr 29, 2019

seems tfb is stuck

@msmith-techempower

This comment has been minimized.

Copy link
Member

commented Apr 29, 2019

seems tfb is stuck

I checked the logs - it looks like it's running fine to me.

@fafhrd91

This comment has been minimized.

Copy link
Contributor

commented Apr 29, 2019

maybe something with cache?
https://tfb-status.techempower.com/results/019d3fd5-b7e0-41b2-b401-c42a382abf85 shows 16 completed for last 5 hours

@msmith-techempower

This comment has been minimized.

Copy link
Member

commented Apr 29, 2019

It has only been running for 2 hours.

It is still going and I see no reason to think anything is working improperly based on the logs and expected duration.

@nbrady-techempower

This comment has been minimized.

Copy link
Member Author

commented Apr 29, 2019

It does appear to be stuck; we'll take a look at it.

@volyrique

This comment has been minimized.

Copy link
Contributor

commented Apr 30, 2019

I want to point out that we're not necessarily shooting for the most optimized environment for azure...

@nbrady-techempower Sure, my main point was about reproducibility - I don't mind if a particular Azure option that may improve performance is not used, as long as it is clearly documented (especially if it is not the same as the default), so that people who try to reproduce the results are not left wondering why their data differs so much.

@jsongte

This comment has been minimized.

Copy link

commented Apr 30, 2019

I've added some more info on the wiki https://github.com/TechEmpower/FrameworkBenchmarks/wiki/Project-Information-Environment

Once I finalized the azure terraform setups and the scripts I'll make a reference of that on the wiki as well, since all the details can be found there.

@wangkaish

This comment has been minimized.

Copy link
Contributor

commented May 3, 2019

Seems Azure is stuck!

@nbrady-techempower

This comment has been minimized.

Copy link
Member Author

commented May 13, 2019

We may be pushing this off for just a little bit until we can resolve the Azure issues. Hopefully this week.

@nbrady-techempower

This comment has been minimized.

Copy link
Member Author

commented May 22, 2019

Still working on Azure. We're hoping to have Round 18 published by the end of next week, barring any more unforeseen issues (there are always some!)

Feel free to continue making adjustments, but I wouldn't make any experimental changes as they may get captured in official round results.

@zloster

This comment has been minimized.

Copy link
Contributor

commented Jun 3, 2019

Another note regarding Java versions - most images are using the Docker openjdk image. For quite long time it was based on the OpenJDK code in the Debian repositories. Very recently they've switched to AdoptOpenJDK builds.
11.0.3 tag is republished with the AdoptOpenJDK. This is not visible yet in the TravicCI builds.

@volyrique

This comment has been minimized.

Copy link
Contributor

commented Jun 25, 2019

It seems that the CVE fixes had a pretty significant effect on the JSON serialization, single query, and fortunes results in the latest Citrine run. I don't think the framework standings changed much, but the maximum requests per second dropped by 22.04%, 16.75%, and 22.05% respectively.

@zloster

This comment has been minimized.

Copy link
Contributor

commented Jun 25, 2019

Another important point about the MDS mitigations - it seems that the results are with activated hyper-threading.

h2o: Number of processors: 28

Search for the text above here. The Xeon Gold 5120 is having 14 physical cores.

I suspect there will be a run with hyper-threading disabled.

@volyrique

This comment has been minimized.

Copy link
Contributor

commented Jun 25, 2019

Apparently other projects suffered from regressions as well.

@msmith-techempower

This comment has been minimized.

Copy link
Member

commented Jun 25, 2019

It seems that the CVE fixes had a pretty significant effect on the JSON serialization, single query, and fortunes results in the latest Citrine run. I don't think the framework standings changed much, but the maximum requests per second dropped by 22.04%, 16.75%, and 22.05% respectively.

Yikes.

@joanhey

This comment has been minimized.

Copy link
Contributor

commented Jun 26, 2019

In the plaintext bench, the difference is near to 50% in some frameworks/platforms.

fw before after
h2o 5,724,772 3,085,341
nginx 3,668,042 2,267,821
php-ngx 2,679,720 1,746,828
php 282,415 172,840

I think like @zloster : run with hyper-threading disabled.
Info:
https://wiki.ubuntu.com/SecurityTeam/KnowledgeBase/MDS
https://www.phoronix.com/scan.php?page=news_item&px=MDS-Zombieload-Initial-Impact

@bhauer

This comment has been minimized.

Copy link
Member

commented Jul 3, 2019

If everything goes as planned, Round 18 will be published next week (approximately July 8). It will be based on the most recent Azure run and the Citrine run that used the same commit. These were captured prior to application of the CVE-2019-1147x mitigations.

@fafhrd91

This comment has been minimized.

Copy link
Contributor

commented Jul 3, 2019

@bhauer those runs still contain "onyx" framework, which violates db bench requirenments

@nbrady-techempower

This comment has been minimized.

Copy link
Member Author

commented Jul 3, 2019

@fafhrd91 See the original post. Anything that's listed as violating the rules won't be displayed on the official results page.

@nbrady-techempower

This comment has been minimized.

Copy link
Member Author

commented Jul 10, 2019

For those just following along here, Round 18 has been released. https://www.techempower.com/blog/2019/07/09/framework-benchmarks-round-18/

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.