|
1 | | -Performance Tests |
| 1 | +Performance Tests |
2 | 2 | ====================== |
3 | 3 |
|
4 | 4 | This document contains instructions for building, running, and adding Performance tests. |
5 | 5 |
|
6 | | -Requirements |
7 | | --------------------- |
8 | | -### Windows |
9 | | -To run performance tests on Windows, .NET portable v5.0 is required. This library is included in [the Visual Studio Community 2015 download](https://www.visualstudio.com/products/visual-studio-community-vs). To get the correct packages during installation, follow these steps after opening the installer: |
10 | | -1. Select "Custom Installation" if no installation is present, or "Modify" otherwise |
11 | | -2. Check the "Universal Windows App Development Tools" box under the "Windows and Web Development" menu |
12 | | -3. Install |
13 | | -### Linux |
14 | | -Performance tests on Linux require all of the same steps as they do for regular xunit tests - see the linux instructions [here](https://github.com/dotnet/corefx/blob/master/Documentation/building/unix-instructions.md). Once you can have a directory on your Linux machine with a working corerun and xunit.console.netcore.exe (as well as the test dll containing your perf tests!), you only need to run the following command: |
15 | | - |
16 | | -`dnu commands install Microsoft.DotNet.xunit.performance.runner.dnx 1.0.0-alpha-build0021 -f https://dotnet.myget.org/F/dotnet-buildtools/api/v3/index.json` |
17 | | - |
18 | | -Be careful that your mscorlib, libcoreclr, and test dlls were compiled using the "/p:Configuration=Release" property. Otherwise you may get skewed results. |
19 | | - |
20 | | -Running the tests |
| 6 | +Building and Running Tests |
21 | 7 | ----------- |
22 | | -### Windows |
23 | | -Performance test files (if present) are stored within a library's ```tests/Performance``` directory and contain test methods that are all marked with a perf-specific *Benchmark* attribute. The performance tests will only be run if the ```performance``` property is set to ```true```. |
| 8 | +Performance test files (if present) are stored within a library's ```tests/Performance``` directory and contain test methods that are all marked with a perf-specific *Benchmark* attribute. |
24 | 9 |
|
25 | | -Before running the performance tests you must run ```build -release``` from the root folder. |
| 10 | +**Step # 1:** Prior to running performance tests, a full build from the repo root must be completed: ```build -release``` |
26 | 11 |
|
27 | | -To build and run the tests using msbuild for a project, run ```msbuild /t:BuildAndTest /p:Performance=true /p:ConfigurationGroup=Release /p:TargetOS=Windows_NT``` from the Performance directory with Admin privileges. If the v5.0 assemblies aren't installed on your system, an error will be raised and no tests will be run. |
| 12 | +**Step # 2:** Change directory to the performance tests directory: ```cd path/to/library/tests/Performance``` |
28 | 13 |
|
29 | | -Note: Because build.cmd runs tests concurrently, it's not recommended that you execute the perf tests using it. |
| 14 | +**Step # 3:** Build and run the tests: |
| 15 | + - Windows (using admin command shell): ```msbuild /t:BuildAndTest /p:Performance=true /p:ConfigurationGroup=Release /p:TargetOS=Windows_NT``` |
| 16 | + - Linux: ```<repo-root>/Tools/msbuild.sh /t:BuildAndTest /p:Performance=true /p:ConfigurationGroup=Release /p:TargetOS=Linux``` |
30 | 17 |
|
31 | | -results will be in: corefx/bin/tests/Windows_NT.AnyCPU.Release/TESTNAME/netcoreapp1.0 |
32 | | -### Linux |
33 | | -From your tests directory, run: |
34 | | -``` |
35 | | -xunit.performance System.Collections.Tests.dll -trait Benchmark=true -verbose -runner ./xunit.console.netcore.exe -runnerhost ./corerun -runid System.Collections.Tests.dll-Linux -outdir results |
36 | | -``` |
| 18 | +**Note: Because build-tests.cmd/sh runs tests concurrently, do not use it for executing performance tests.** |
37 | 19 |
|
38 | | -This will run the perf tests for System.Collections.Tests.dll and output the results in results/System.Collections.Tests.dll-Linux.xml and results/System.Collections.Tests.dll-Linux.csv |
| 20 | +The results files will be dropped in corefx/bin/tests/FLAVOR/TESTLIBRARY/TARGETFRAMEWORK. The console output will also specify the location of these files. |
39 | 21 |
|
40 | | -Adding new Performance tests |
| 22 | +Adding New Performance Tests |
41 | 23 | ----------- |
42 | | -Performance tests for CoreFX are built on top of xunit and [the Microsoft xunit-performance runner](https://github.com/Microsoft/xunit-performance/). |
| 24 | +Performance tests for CoreFX are built on top of xunit and [the Microsoft xunit-performance framework](https://github.com/Microsoft/xunit-performance/). |
43 | 25 |
|
44 | | -For the time being, perf tests should reside within their own "Performance" folder within the tests directory of a library (e.g. [corefx/src/System.IO.FileSystem/tests/Performance](https://github.com/dotnet/corefx/tree/master/src/System.IO.FileSystem/tests/Performance) contains perf tests for FileSystem). |
| 26 | +Performance tests should reside within their own "Performance" folder within the tests directory of a library (e.g. [corefx/src/System.IO.FileSystem/tests/Performance](https://github.com/dotnet/corefx/tree/master/src/System.IO.FileSystem/tests/Performance) contains perf tests for FileSystem). |
45 | 27 |
|
46 | 28 | It's easiest to copy and modify an existing example like the one above. Notice that you'll need these lines in the tests csproj: |
47 | 29 | ``` |
|
0 commit comments