Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add console app for running isolated benchmarks #32

Conversation

philipbelesky
Copy link
Collaborator

This adds a console app to the project that can run benchmarks outside of Rhino/GH over a set number of iterations. It does so by mocking parameters within the SandWorm component that are necessary to simulate acting on a mesh. The idea here is to be able to easily A/B changes without running Rhino/Grasshopper and to measure performance more reliably.

Currently this relates to #29 and #30 so the test functions and overall classes are setup to test the mesh coloring stage. If it is useful for optimising other parts of the code it should be able to be extended to cover different mock data/methods and the amount of mock data setup could be reduce by creating a shared class that is inherited by both SandwormComponent and MockMesh.

I'm a little split as to whether this should be in the main repo or kept on a branch. The former is a bit less clean, but is more convenient. If it proves useful it would also be easier to be able to improve the console app itself along the rest of the code. We could also just remove it later if it ends up being less useful?

Also a result of this, the console app needs to reference a Sandworm.dll so Sandworm needs to be built first (ideally as the released build) and the 'erase' step of the build event has been removed.

@philipbelesky philipbelesky force-pushed the feature/benchmarks-helper branch 4 times, most recently from 0f066d2 to 5d576a9 Compare September 14, 2019 05:39
@philipbelesky
Copy link
Collaborator Author

Oh, also, a quick note if testing this: the solution file was moved into the base directory, which means the packages folder also needs to move up a level.

@mariuszhermansdorfer
Copy link
Owner

Let's add it to the master for convenience and remove later when no longer needed

@mariuszhermansdorfer mariuszhermansdorfer merged commit d32be9c into mariuszhermansdorfer:master Sep 14, 2019
@mariuszhermansdorfer
Copy link
Owner

Great work on the console app @philipbelesky! It seems that there is a minor issue with the way times are measured, though. Currently the first test seems to suffer from a small overhead of approx. 6 ms. Easy to reproduce through reversing the order of tests in the list.
image
image

@philipbelesky
Copy link
Collaborator Author

Hmm, I noticed a much smaller discrepancy (sub 1ms avg) when running which seemed enough to ignore when profiling bigger functions. But I was mostly benchmarking within a VM which is probably a less typical environment.

However, it turns out the mocking class means that the benchmarks are now compatible with using benchmark.net now, so I've switched over to using that in 2381ef5. They take a little longer to run but the results are more consistent and all the timing code can be removed in favour of an attribute marker.

Screen Shot 2019-09-15 at 11 35 23 am

@philipbelesky philipbelesky deleted the feature/benchmarks-helper branch September 20, 2019 03:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants