Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Question: Any official way to benchmark same method between different assembly versions? #290

Closed
Shazwazza opened this Issue Oct 23, 2016 · 10 comments

Comments

Projects
None yet
5 participants
@Shazwazza
Copy link
Contributor

commented Oct 23, 2016

Hi,

I can't seem to find anything specifically about what I'd like to do, but if this has already been discussed some other place please let me know :)

Otherwise, what I'd like to do is benchmark the same method of an assembly but with different assembly versions. I know I can accomplish this in various ways and use reflection to load in the type/method based on different assembly files but am wondering if there is any official way to do this?

@adamsitnik

This comment has been minimized.

Copy link
Member

commented Oct 23, 2016

H! @Shazwazza !

Currenly we don't support it, but I am planning to add such feature.

My initial goal was to enable comparison for for assemblies that are available on the nuget. Sample scenario: what's the difference between current implementation and latest version available on nuget?

What would be your scenario?

@Shazwazza

This comment has been minimized.

Copy link
Contributor Author

commented Oct 23, 2016

yes that is exactly what I was planning on implementing :)

I haven't written anything quite yet but wanted to try to get something going as early as next week. The idea that I had was something along the lines of:

  • During the benchmark initialization, have some code config to point to either a referenced assembly or a nuget alias + version
  • Use Nuget API (or whatever) to go fetch the configured Nuget assembly versions from whatever Nuget sources are configured
  • Then load in the class/method from either the referenced assembly or the Nuget fetched assembly
  • Run the benchmark
@adamsitnik

This comment has been minimized.

Copy link
Member

commented Oct 24, 2016

One question: are you using .csprojs or .xprojs? I have an idea how dotnet cli could do the nuget restore for us.

If you are using csprojs then I think that the best thing to do would be to extend our RoslynToolchain. Here you can see how we build the code with Roslyn. I think that the solution would be to extend our Job class with list of assembiles/nugets that would be used by the RoslynGenerator

then it would a matter of executing all benchmarks in two configurations (by using two separate Jobs): without custom assemblies and with custom assemblies.

@Shazwazza

This comment has been minimized.

Copy link
Contributor Author

commented Oct 24, 2016

For the current project I'm working with it would be csproj. Thanks for the pointers, I'll fork the project and have a look and hopefully i can find some time :)

@GeertvanHorrik

This comment has been minimized.

Copy link

commented Nov 14, 2016

I am currently working on this for my own purposes. I will push this to the (open source) repository soon:

https://github.com/catel/catel.benchmarks

What I have in mind is to create separate console apps / version (so we can target specific versions without problems). Then I have a "wrapper runner" that runs the console apps one by one and stores the results inside results / version.

Once this is done, I "only" have to combine the results.

@GeertvanHorrik

This comment has been minimized.

Copy link

commented Nov 15, 2016

Here is the full repository:
https://github.com/catel/catel.benchmarks

Example of a generated comparison markdown:
https://github.com/Catel/Catel.Benchmarks/blob/develop/results/summary.md

In theory we could use a single app and load them into separate AppDomains. However I am not sure if defining the different versions via NuGet is the way to go (it makes testing against local builds harder).

For now the combiner runs all the different benchmarks and combines the results (reading all measurements and grouping them by version) which are easy to use for exporting purposes.

@Rbn3D

This comment has been minimized.

Copy link

commented Sep 4, 2018

Any updates on this? I'm trying to benchmark two versions of the same project (one as project reference and the other as nuget packageReference).

My purpose is to measure optimizations i'm making to that project (so I can compare them with the current version published in nuget easily).

@GeertvanHorrik

This comment has been minimized.

Copy link

commented Sep 6, 2018

@Rbn3D did you check out the project I mentioned? It does do benchmarks against each version and creates a fully summary of the delta (top 10 improvements and top 10 stuff becoming worse).

@Shazwazza

This comment has been minimized.

Copy link
Contributor Author

commented Oct 3, 2018

The PR for this is here #805

@Shazwazza

This comment has been minimized.

Copy link
Contributor Author

commented Oct 23, 2018

Updated PR is here #922

@adamsitnik adamsitnik added this to the v0.11.2 milestone Oct 25, 2018

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.