[Question]Run with --cli

[MarcoRossignoli]

I'm trying to run benchmark with --cli options as outlined here https://github.com/dotnet/performance/blob/master/src/benchmarks/micro/README.md#dotnet-cli
However if I don't have installed correct machine wide sdk I get error:

D:\git\performance\src\benchmarks\micro (master -> origin)
λ dotnet run -c Release -f netcoreapp3.0 -- --join --filter System.Collections*.Dictionary* *.Perf_Dictionary.* --cli ..\..\..\..\netcorepreviewsdk\dotnet.exe
C:\Program Files\dotnet\sdk\2.2.100\Sdks\Microsoft.NET.Sdk\targets\Microsoft.NET.TargetFrameworkInference.targets(137,5): error NETSDK1045: The current .NET SDK does not support targeting .NET Core 3.0.  Either target .NET Core 2.2 or lower, or use a version of the .NET SDK that supports .NET Core 3.0. [D:\git\performance\src\benchmarks\micro\MicroBenchmarks.csproj]

The build failed. Please fix the build errors and run again.

But it works if I use downloaded(non machine wide) sdk:

D:\git\performance\src\benchmarks\micro (master -> origin)
λ ..\..\..\..\netcorepreviewsdk\dotnet.exe run -c Release -f netcoreapp3.0 -- --join --filter System.Collections*.Dictionary* *.Perf_Dictionary.* --cli ..\..\..\..\netcorepreviewsdk\dotnet.exe
// Validating benchmarks:
// ***** BenchmarkRunner: Start   *****
// ***** Building 1 exe(s) in Parallel: Start   *****

Is it expected?If so I propose to update guide with this scenario(I don't like install all preview machine wide), let me know if I can do a PR(if it makes sense to you).

/cc @adamsitnik @danmosemsft

[adamsitnik]

Hi @MarcoRossignoli

We have a nice python script that can download the right dotnet cli and run the benchmarks for you. I am gonna add some docs for it soon.

As of now you can use it in following way:

.\scripts\benchmarks_ci.py --frameworks netcoreapp3.0 --filter System.Collections*.Dictionary* *.Perf_Dictionary.* --bdn-arguments "--join"

/cc @jorive

[MarcoRossignoli]

for --CoreRun list can we append after --join?

[adamsitnik]

for --CoreRun list can we append after --join?

I am not sure if I understand correctly. You can combine both "--corerun" and "--join". The script that I have mentioned has native support for most of the bdn arguments like --filter --corerun but for some less popular like --join you need to pass it via --bdn-arguments option

[MarcoRossignoli]

The script that I have mentioned has native support for most of the bdn arguments like --filter --corerun

I meant that!Thank's!

[MarcoRossignoli]

@adamsitnik I get error:

D:\git\performancefork (dicperf -> origin)
λ .\scripts\benchmarks_ci.py --frameworks netcoreapp3.0 --filter System.Collections*.Dictionary* *.Perf_Dictionary.* --bdn-arguments "--join"
  File "D:\git\performancefork\scripts\benchmarks_ci.py", line 60
    architecture: str,
                ^
SyntaxError: invalid syntax

I'm not a python expert...could be version issue?

[jorive]

@MarcoRossignoli What version of Python are you running? We support >=3.5

[MarcoRossignoli]

Ah ok 2.7...

[jorive]

Python runtimes can live side-by-side. If you are running under Windows, then make sure you also install py.exe. This program knows what runtime to execute based on the shebang.

[MarcoRossignoli]

FYI @jorive @adamsitnik now it's ok with python runtime...however I've got issue with --bdn-arguments:

D:\git\performancefork (dicperf -> origin)                                                                                                       
λ .\scripts\benchmarks_ci.py --frameworks netcoreapp3.0 --filter System.Collections*.Dictionary* *.Perf_Dictionary.* --bdn-arguments "--join"    
usage: benchmarks_ci.py [-h] [--architecture {x64,x86,arm32,arm64}] -f                                                                           
                        {netcoreapp3.0,netcoreapp2.2,netcoreapp2.1,netcoreapp2.0,net461}                                                         
                        [{netcoreapp3.0,netcoreapp2.2,netcoreapp2.1,netcoreapp2.0,net461} ...]                                                   
                        [--incremental {yes,no}] [--enable-hardware-counters]                                                                    
                        [--category {coreclr,corefx}]                                                                                            
                        [--filter FILTER [FILTER ...]] [--corerun CORERUN]                                                                       
                        [--cli CLI] [--bdn-arguments BDN_ARGUMENTS]                                                                              
                        [--optimization-level {tiered,full_opt,min_opt}]                                                                         
                        [--cli-source-info {init-tools,repo,cli}]                                                                                
                        [--cli-branch CLI_BRANCH]                                                                                                
                        [--cli-commit-sha CLI_COMMIT_SHA]                                                                                        
                        [--cli-repository CLI_REPOSITORY]                                                                                        
                        [--cli-source-timestamp CLI_SOURCE_TIMESTAMP]                                                                            
                        [--generate-benchview-data]                                                                                              
                        [--upload-to-benchview-container UPLOAD_TO_BENCHVIEW_CONTAINER]                                                          
                        [--benchview-submission-name BENCHVIEW_SUBMISSION_NAME]                                                                  
                        [--benchview-run-type {rolling,private,local}]                                                                           
                        [--benchview-config-name BENCHVIEW_CONFIG_NAME]                                                                          
                        [--benchview-machinepool BENCHVIEW_MACHINEPOOL]                                                                          
                        [--benchview-config key value] [-q]                                                                                      
benchmarks_ci.py: error: argument --bdn-arguments: expected one argument                                                                         

I need to escape param value to make it work:

λ .\scripts\benchmarks_ci.py --frameworks netcoreapp3.0 --filter System.Collections*.Dictionary* *.Perf_Dictionary.* --bdn-arguments \"--join\"  

I don't know if it expected.

[MarcoRossignoli]

Issues with --corerun

1. If I try to run with custom corerun I get some errors(I tried to restart my laptop and to open only console to avoid involuntary file locking, without custom corerun all works well)

D:\git\performancefork (dicperf -> origin)
λ .\scripts\benchmarks_ci.py --frameworks netcoreapp3.0 --filter *.Perf_Dictionary.* --corerun D:\git\corefxupstream\artifacts\bin\testhost\netcoreapp-Windows_NT-Release-x64\shared\Microsoft.NETCore.App\9.9.9\CoreRun.exe --bdn-arguments \"--join\"
[2019/01/01 13:01:51][INFO] ----------------------------------------------
[2019/01/01 13:01:51][INFO] Initializing logger 2019-01-01 13:01:51.503711
[2019/01/01 13:01:51][INFO] ----------------------------------------------
[2019/01/01 13:01:51][INFO] Installing tools.
[2019/01/01 13:01:51][INFO] ----------------------
[2019/01/01 13:01:51][INFO] Downloading DotNet Cli
[2019/01/01 13:01:51][INFO] ----------------------
[2019/01/01 13:01:51][INFO] DotNet Install Path: 'D:\git\performancefork\tools\dotnet\x64'
[2019/01/01 13:01:51][INFO] Downloading https://raw.githubusercontent.com/dotnet/cli/master/scripts/obtain/dotnet-install.ps1
[2019/01/01 13:01:51][INFO] $ pushd "D:\git\performancefork"
[2019/01/01 13:01:51][INFO] $ powershell.exe -NoProfile -ExecutionPolicy Bypass D:\git\performancefork\tools\dotnet\x64\dotnet-install.ps1 -InstallDir D:\git\performancefork\tools\dotnet\x64 -Architecture x64 -Channel master
[2019/01/01 13:01:53][INFO] dotnet-install: .NET Core SDK version 3.0.100-preview-009990 is already installed.
[2019/01/01 13:01:53][INFO] dotnet-install: Adding to current process PATH: "D:\git\performancefork\tools\dotnet\x64\". Note: This change will not be visible if PowerShell was run as a child process.
[2019/01/01 13:01:53][INFO] $ popd
[2019/01/01 13:01:53][INFO] -----------------------------
[2019/01/01 13:01:53][INFO] Downloading BenchView scripts
[2019/01/01 13:01:53][INFO] -----------------------------
[2019/01/01 13:01:54][INFO] http://benchviewtestfeed.azurewebsites.net/api/v2/package/microsoft.benchview.jsonformat/0.1.0-pre029 -> D:\git\performancefork\tools\Microsoft.BenchView.JSONFormat
[2019/01/01 13:01:56][INFO] $ dotnet --info
[2019/01/01 13:01:56][INFO] .NET Core SDK (reflecting any global.json):
[2019/01/01 13:01:56][INFO]  Version:   3.0.100-preview-009990
[2019/01/01 13:01:56][INFO]  Commit:    c9eb469360
[2019/01/01 13:01:56][INFO]
[2019/01/01 13:01:56][INFO] Runtime Environment:
[2019/01/01 13:01:56][INFO]  OS Name:     Windows
[2019/01/01 13:01:56][INFO]  OS Version:  10.0.17134
[2019/01/01 13:01:56][INFO]  OS Platform: Windows
[2019/01/01 13:01:56][INFO]  RID:         win10-x64
[2019/01/01 13:01:56][INFO]  Base Path:   D:\git\performancefork\tools\dotnet\x64\sdk\3.0.100-preview-009990\
[2019/01/01 13:01:56][INFO]
[2019/01/01 13:01:56][INFO] Host (useful for support):
[2019/01/01 13:01:56][INFO]   Version: 3.0.0-preview-27218-01
[2019/01/01 13:01:56][INFO]   Commit:  d40b87f29d
[2019/01/01 13:01:56][INFO]
[2019/01/01 13:01:56][INFO] .NET Core SDKs installed:
[2019/01/01 13:01:56][INFO]   3.0.100-preview-009990 [D:\git\performancefork\tools\dotnet\x64\sdk]
[2019/01/01 13:01:56][INFO]
[2019/01/01 13:01:56][INFO] .NET Core runtimes installed:
[2019/01/01 13:01:56][INFO]   Microsoft.AspNetCore.App 3.0.0-preview-18579-0056 [D:\git\performancefork\tools\dotnet\x64\shared\Microsoft.AspNetCore.App]
[2019/01/01 13:01:56][INFO]   Microsoft.NETCore.App 3.0.0-preview-27218-01 [D:\git\performancefork\tools\dotnet\x64\shared\Microsoft.NETCore.App]
[2019/01/01 13:01:56][INFO]   Microsoft.WindowsDesktop.App 3.0.0-alpha-27218-5 [D:\git\performancefork\tools\dotnet\x64\shared\Microsoft.WindowsDesktop.App]
[2019/01/01 13:01:56][INFO]
[2019/01/01 13:01:56][INFO] To install additional .NET Core runtimes or SDKs:
[2019/01/01 13:01:56][INFO]   https://aka.ms/dotnet-download
[2019/01/01 13:01:56][INFO] -------------------------------
[2019/01/01 13:01:56][INFO] Restoring .NET micro benchmarks
[2019/01/01 13:01:56][INFO] -------------------------------
[2019/01/01 13:01:56][INFO] $ pushd "D:\git\performancefork\src\benchmarks\micro"
[2019/01/01 13:01:56][INFO] $ dotnet restore MicroBenchmarks.csproj --packages D:\git\performancefork\artifacts\packages
[2019/01/01 13:01:57][INFO]   Restoring packages for D:\git\performancefork\src\benchmarks\micro\MicroBenchmarks.csproj...
[2019/01/01 13:01:58][INFO]   Generating MSBuild file D:\git\performancefork\artifacts\obj\MicroBenchmarks\MicroBenchmarks.csproj.nuget.g.props.
[2019/01/01 13:01:58][INFO]   Generating MSBuild file D:\git\performancefork\artifacts\obj\MicroBenchmarks\MicroBenchmarks.csproj.nuget.g.targets.
[2019/01/01 13:01:58][INFO]   Restore completed in 467.21 ms for D:\git\performancefork\src\benchmarks\micro\MicroBenchmarks.csproj.
[2019/01/01 13:01:58][INFO] $ popd
[2019/01/01 13:01:58][INFO] --------------------------------------------------
[2019/01/01 13:01:58][INFO] Building .NET micro benchmarks for 'netcoreapp3.0'
[2019/01/01 13:01:58][INFO] --------------------------------------------------
[2019/01/01 13:01:58][INFO] $ pushd "D:\git\performancefork\src\benchmarks\micro"
[2019/01/01 13:01:58][INFO] $ dotnet build MicroBenchmarks.csproj --configuration Release --framework netcoreapp3.0 --no-restore
[2019/01/01 13:01:58][INFO] Microsoft (R) Build Engine version 16.0.284-preview+g7905402dc6 for .NET Core
[2019/01/01 13:01:58][INFO] Copyright (C) Microsoft Corporation. All rights reserved.
[2019/01/01 13:01:58][INFO]
[2019/01/01 13:02:02][INFO]   MicroBenchmarks -> D:\git\performancefork\artifacts\bin\MicroBenchmarks\Release\netcoreapp3.0\MicroBenchmarks.dll
[2019/01/01 13:02:02][INFO]
[2019/01/01 13:02:02][INFO] Build succeeded.
[2019/01/01 13:02:02][INFO]     0 Warning(s)
[2019/01/01 13:02:02][INFO]     0 Error(s)
[2019/01/01 13:02:02][INFO]
[2019/01/01 13:02:02][INFO] Time Elapsed 00:00:03.65
[2019/01/01 13:02:02][INFO] $ popd
[2019/01/01 13:02:02][INFO] -------------------------------------------------
[2019/01/01 13:02:02][INFO] Running .NET micro benchmarks for 'netcoreapp3.0'
[2019/01/01 13:02:02][INFO] -------------------------------------------------
[2019/01/01 13:02:02][INFO] $ pushd "D:\git\performancefork\src\benchmarks\micro"
[2019/01/01 13:02:02][INFO] $ dotnet run --project MicroBenchmarks.csproj --configuration Release --framework netcoreapp3.0 --no-restore --no-build -- --coreRun D:\git\corefxupstream\artifacts\bin\testhost\netcoreapp-Windows_NT-Release-x64\shared\Microsoft.NETCore.App\9.9.9\CoreRun.exe --filter *.Perf_Dictionary.* --join --packages D:\git\performancefork\artifacts\packages --runtimes netcoreapp3.0
[2019/01/01 13:02:09][INFO] // Validating benchmarks:
[2019/01/01 13:02:09][INFO] // ***** BenchmarkRunner: Start   *****
[2019/01/01 13:02:09][INFO] // ***** Building 1 exe(s) in Parallel: Start   *****
[2019/01/01 13:02:10][INFO] // start dotnet restore  /p:UseSharedCompilation=false in D:\git\performancefork\artifacts\bin\MicroBenchmarks\Release\netcoreapp3.0\2575d930-c448-4e52-b4d4-128c4580cbec
[2019/01/01 13:02:11][INFO] // command took 1.61s and exited with 0
[2019/01/01 13:02:11][INFO] // start dotnet build -c Release  /p:UseSharedCompilation=false in D:\git\performancefork\artifacts\bin\MicroBenchmarks\Release\netcoreapp3.0\2575d930-c448-4e52-b4d4-128c4580cbec
[2019/01/01 13:02:26][INFO] // command took 14.71s and exited with 1
[2019/01/01 13:02:26][INFO] // start dotnet build -c Release  --no-dependencies /p:UseSharedCompilation=false in D:\git\performancefork\artifacts\bin\MicroBenchmarks\Release\netcoreapp3.0\2575d930-c448-4e52-b4d4-128c4580cbec
[2019/01/01 13:02:29][INFO] // command took 3.16s and exited with 0
[2019/01/01 13:02:29][INFO] // start dotnet publish -c Release  /p:UseSharedCompilation=false in D:\git\performancefork\artifacts\bin\MicroBenchmarks\Release\netcoreapp3.0\2575d930-c448-4e52-b4d4-128c4580cbec
[2019/01/01 13:02:41][INFO] // command took 11.92s and exited with 1
[2019/01/01 13:02:41][INFO] // ***** Done, took 00:00:31 (31.96 sec)   *****
[2019/01/01 13:02:41][INFO] // Found benchmarks:
[2019/01/01 13:02:41][INFO] //   Perf_Dictionary.ContainsValue: Job-JRRDAT(Runtime=Core, Toolchain=CoreRun, IterationTime=250.0000 ms, MaxIterationCount=20, MinIterationCount=15, WarmupCount=1) [Items=3000]
[2019/01/01 13:02:41][INFO]
[2019/01/01 13:02:41][INFO] // Build Exception: Standard output:
[2019/01/01 13:02:41][INFO]  Microsoft (R) Build Engine version 16.0.284-preview+g7905402dc6 for .NET Core
[2019/01/01 13:02:41][INFO] Copyright (C) Microsoft Corporation. All rights reserved.
[2019/01/01 13:02:41][INFO]
[2019/01/01 13:02:41][INFO]   Restore completed in 40.34 ms for D:\git\performancefork\src\benchmarks\micro\MicroBenchmarks.csproj.
[2019/01/01 13:02:41][INFO]   Restore completed in 40.34 ms for D:\git\performancefork\artifacts\bin\MicroBenchmarks\Release\netcoreapp3.0\2575d930-c448-4e52-b4d4-128c4580cbec\BenchmarkDotNet.Autogenerated.csproj.
[2019/01/01 13:02:41][INFO] D:\git\performancefork\tools\dotnet\x64\sdk\3.0.100-preview-009990\Sdks\Microsoft.NET.Sdk\targets\Microsoft.NET.RuntimeIdentifierInference.targets(142,5): message NETSDK1057: You are using a preview version of .NET Core. See: https://aka.ms/dotnet-core-preview [D:\git\performancefork\artifacts\bin\MicroBenchmarks\Release\netcoreapp3.0\2575d930-c448-4e52-b4d4-128c4580cbec\BenchmarkDotNet.Autogenerated.csproj]
[2019/01/01 13:02:41][INFO] D:\git\performancefork\tools\dotnet\x64\sdk\3.0.100-preview-009990\Microsoft.Common.CurrentVersion.targets(4558,5): warning MSB3026: Could not copy "D:\git\performancefork\artifacts\obj\MicroBenchmarks\Release\netcoreapp3.0\MicroBenchmarks.exe" to "D:\git\performancefork\artifacts\bin\MicroBenchmarks\Release\netcoreapp3.0\MicroBenchmarks.exe". Beginning retry 1 in 1000ms. The process cannot access the file 'D:\git\performancefork\artifacts\bin\MicroBenchmarks\Release\netcoreapp3.0\MicroBenchmarks.exe' because it is being used by another process.  [D:\git\performancefork\src\benchmarks\micro\MicroBenchmarks.csproj]
[2019/01/01 13:02:41][INFO] D:\git\performancefork\tools\dotnet\x64\sdk\3.0.100-preview-009990\Microsoft.Common.CurrentVersion.targets(4558,5): warning MSB3026: Could not copy "D:\git\performancefork\artifacts\obj\MicroBenchmarks\Release\netcoreapp3.0\MicroBenchmarks.exe" to "D:\git\performancefork\artifacts\bin\MicroBenchmarks\Release\netcoreapp3.0\MicroBenchmarks.exe". Beginning retry 2 in 1000ms. The process cannot access the file 'D:\git\performancefork\artifacts\bin\MicroBenchmarks\Release\netcoreapp3.0\MicroBenchmarks.exe' because it is being used by another process.  [D:\git\performancefork\src\benchmarks\micro\MicroBenchmarks.csproj]
[2019/01/01 13:02:41][INFO] D:\git\performancefork\tools\dotnet\x64\sdk\3.0.100-preview-009990\Microsoft.Common.CurrentVersion.targets(4558,5): warning MSB3026: Could not copy "D:\git\performancefork\artifacts\obj\MicroBenchmarks\Release\netcoreapp3.0\MicroBenchmarks.exe" to "D:\git\performancefork\artifacts\bin\MicroBenchmarks\Release\netcoreapp3.0\MicroBenchmarks.exe". Beginning retry 3 in 1000ms. The process cannot access the file 'D:\git\performancefork\artifacts\bin\MicroBenchmarks\Release\netcoreapp3.0\MicroBenchmarks.exe' because it is being used by another process.  [D:\git\performancefork\src\benchmarks\micro\MicroBenchmarks.csproj]
[2019/01/01 13:02:41][INFO] D:\git\performancefork\tools\dotnet\x64\sdk\3.0.100-preview-009990\Microsoft.Common.CurrentVersion.targets(4558,5): warning MSB3026: Could not copy "D:\git\performancefork\artifacts\obj\MicroBenchmarks\Release\netcoreapp3.0\MicroBenchmarks.exe" to "D:\git\performancefork\artifacts\bin\MicroBenchmarks\Release\netcoreapp3.0\MicroBenchmarks.exe". Beginning retry 4 in 1000ms. The process cannot access the file 'D:\git\performancefork\artifacts\bin\MicroBenchmarks\Release\netcoreapp3.0\MicroBenchmarks.exe' because it is being used by another process.  [D:\git\performancefork\src\benchmarks\micro\MicroBenchmarks.csproj]
[2019/01/01 13:02:41][INFO] D:\git\performancefork\tools\dotnet\x64\sdk\3.0.100-preview-009990\Microsoft.Common.CurrentVersion.targets(4558,5): warning MSB3026: Could not copy "D:\git\performancefork\artifacts\obj\MicroBenchmarks\Release\netcoreapp3.0\MicroBenchmarks.exe" to "D:\git\performancefork\artifacts\bin\MicroBenchmarks\Release\netcoreapp3.0\MicroBenchmarks.exe". Beginning retry 5 in 1000ms. The process cannot access the file 'D:\git\performancefork\artifacts\bin\MicroBenchmarks\Release\netcoreapp3.0\MicroBenchmarks.exe' because it is being used by another process.  [D:\git\performancefork\src\benchmarks\micro\MicroBenchmarks.csproj]
[2019/01/01 13:02:41][INFO] D:\git\performancefork\tools\dotnet\x64\sdk\3.0.100-preview-009990\Microsoft.Common.CurrentVersion.targets(4558,5): warning MSB3026: Could not copy "D:\git\performancefork\artifacts\obj\MicroBenchmarks\Release\netcoreapp3.0\MicroBenchmarks.exe" to "D:\git\performancefork\artifacts\bin\MicroBenchmarks\Release\netcoreapp3.0\MicroBenchmarks.exe". Beginning retry 6 in 1000ms. The process cannot access the file 'D:\git\performancefork\artifacts\bin\MicroBenchmarks\Release\netcoreapp3.0\MicroBenchmarks.exe' because it is being used by another process.  [D:\git\performancefork\src\benchmarks\micro\MicroBenchmarks.csproj]
[2019/01/01 13:02:41][INFO] D:\git\performancefork\tools\dotnet\x64\sdk\3.0.100-preview-009990\Microsoft.Common.CurrentVersion.targets(4558,5): warning MSB3026: Could not copy "D:\git\performancefork\artifacts\obj\MicroBenchmarks\Release\netcoreapp3.0\MicroBenchmarks.exe" to "D:\git\performancefork\artifacts\bin\MicroBenchmarks\Release\netcoreapp3.0\MicroBenchmarks.exe". Beginning retry 7 in 1000ms. The process cannot access the file 'D:\git\performancefork\artifacts\bin\MicroBenchmarks\Release\netcoreapp3.0\MicroBenchmarks.exe' because it is being used by another process.  [D:\git\performancefork\src\benchmarks\micro\MicroBenchmarks.csproj]
[2019/01/01 13:02:41][INFO] D:\git\performancefork\tools\dotnet\x64\sdk\3.0.100-preview-009990\Microsoft.Common.CurrentVersion.targets(4558,5): warning MSB3026: Could not copy "D:\git\performancefork\artifacts\obj\MicroBenchmarks\Release\netcoreapp3.0\MicroBenchmarks.exe" to "D:\git\performancefork\artifacts\bin\MicroBenchmarks\Release\netcoreapp3.0\MicroBenchmarks.exe". Beginning retry 8 in 1000ms. The process cannot access the file 'D:\git\performancefork\artifacts\bin\MicroBenchmarks\Release\netcoreapp3.0\MicroBenchmarks.exe' because it is being used by another process.  [D:\git\performancefork\src\benchmarks\micro\MicroBenchmarks.csproj]
[2019/01/01 13:02:41][INFO] D:\git\performancefork\tools\dotnet\x64\sdk\3.0.100-preview-009990\Microsoft.Common.CurrentVersion.targets(4558,5): warning MSB3026: Could not copy "D:\git\performancefork\artifacts\obj\MicroBenchmarks\Release\netcoreapp3.0\MicroBenchmarks.exe" to "D:\git\performancefork\artifacts\bin\MicroBenchmarks\Release\netcoreapp3.0\MicroBenchmarks.exe". Beginning retry 9 in 1000ms. The process cannot access the file 'D:\git\performancefork\artifacts\bin\MicroBenchmarks\Release\netcoreapp3.0\MicroBenchmarks.exe' because it is being used by another process.  [D:\git\performancefork\src\benchmarks\micro\MicroBenchmarks.csproj]
[2019/01/01 13:02:41][INFO] D:\git\performancefork\tools\dotnet\x64\sdk\3.0.100-preview-009990\Microsoft.Common.CurrentVersion.targets(4558,5): warning MSB3026: Could not copy "D:\git\performancefork\artifacts\obj\MicroBenchmarks\Release\netcoreapp3.0\MicroBenchmarks.exe" to "D:\git\performancefork\artifacts\bin\MicroBenchmarks\Release\netcoreapp3.0\MicroBenchmarks.exe". Beginning retry 10 in 1000ms. The process cannot access the file 'D:\git\performancefork\artifacts\bin\MicroBenchmarks\Release\netcoreapp3.0\MicroBenchmarks.exe' because it is being used by another process.  [D:\git\performancefork\src\benchmarks\micro\MicroBenchmarks.csproj]
[2019/01/01 13:02:41][INFO] D:\git\performancefork\tools\dotnet\x64\sdk\3.0.100-preview-009990\Microsoft.Common.CurrentVersion.targets(4558,5): error MSB3027: Could not copy "D:\git\performancefork\artifacts\obj\MicroBenchmarks\Release\netcoreapp3.0\MicroBenchmarks.exe" to "D:\git\performancefork\artifacts\bin\MicroBenchmarks\Release\netcoreapp3.0\MicroBenchmarks.exe". Exceeded retry count of 10. Failed.  [D:\git\performancefork\src\benchmarks\micro\MicroBenchmarks.csproj]
[2019/01/01 13:02:41][INFO] D:\git\performancefork\tools\dotnet\x64\sdk\3.0.100-preview-009990\Microsoft.Common.CurrentVersion.targets(4558,5): error MSB3021: Unable to copy file "D:\git\performancefork\artifacts\obj\MicroBenchmarks\Release\netcoreapp3.0\MicroBenchmarks.exe" to "D:\git\performancefork\artifacts\bin\MicroBenchmarks\Release\netcoreapp3.0\MicroBenchmarks.exe". The process cannot access the file 'D:\git\performancefork\artifacts\bin\MicroBenchmarks\Release\netcoreapp3.0\MicroBenchmarks.exe' because it is being used by another process. [D:\git\performancefork\src\benchmarks\micro\MicroBenchmarks.csproj]
[2019/01/01 13:02:41][INFO]
[2019/01/01 13:02:41][INFO]
[2019/01/01 13:02:41][INFO]  Standard error:
[2019/01/01 13:02:41][INFO]
[2019/01/01 13:02:41][INFO]
[2019/01/01 13:02:41][INFO]
[2019/01/01 13:02:41][INFO] Run time: 00:00:00 (0.06 sec), executed benchmarks: 0
[2019/01/01 13:02:41][INFO]
[2019/01/01 13:02:41][INFO] // ***** BenchmarkRunner: Finish  *****
[2019/01/01 13:02:41][INFO]
[2019/01/01 13:02:41][INFO] // * Export *
[2019/01/01 13:02:41][INFO]   D:\git\performancefork\artifacts\bin\MicroBenchmarks\Release\netcoreapp3.0\BenchmarkDotNet.Artifacts\results\System.Collections.Tests.Perf_Dictionary-report.csv
[2019/01/01 13:02:41][INFO]   D:\git\performancefork\artifacts\bin\MicroBenchmarks\Release\netcoreapp3.0\BenchmarkDotNet.Artifacts\results\System.Collections.Tests.Perf_Dictionary-report-github.md
[2019/01/01 13:02:41][INFO]   D:\git\performancefork\artifacts\bin\MicroBenchmarks\Release\netcoreapp3.0\BenchmarkDotNet.Artifacts\results\System.Collections.Tests.Perf_Dictionary-report.html
[2019/01/01 13:02:41][INFO]
[2019/01/01 13:02:41][INFO] Unhandled Exception: System.ArgumentNullException: Value cannot be null.
[2019/01/01 13:02:41][INFO] Parameter name: source
[2019/01/01 13:02:41][INFO]    at System.Linq.Enumerable.Any[TSource](IEnumerable`1 source)
[2019/01/01 13:02:41][INFO]    at BenchmarkDotNet.Exporters.Json.JsonExporterBase.<>c__DisplayClass9_0.<ExportToLog>b__0(BenchmarkReport report)
[2019/01/01 13:02:41][INFO]    at System.Linq.Enumerable.SelectArrayIterator`2.MoveNext()
[2019/01/01 13:02:41][INFO]    at SimpleJson.SimpleJson.SerializeArray(IJsonSerializerStrategy jsonSerializerStrategy, IEnumerable anArray, StringBuilder builder)
[2019/01/01 13:02:41][INFO]    at SimpleJson.SimpleJson.SerializeValue(IJsonSerializerStrategy jsonSerializerStrategy, Object value, StringBuilder builder)
[2019/01/01 13:02:41][INFO]    at SimpleJson.SimpleJson.SerializeObject(IJsonSerializerStrategy jsonSerializerStrategy, IEnumerable keys, IEnumerable values, StringBuilder builder)
[2019/01/01 13:02:41][INFO]    at SimpleJson.SimpleJson.SerializeValue(IJsonSerializerStrategy jsonSerializerStrategy, Object value, StringBuilder builder)
[2019/01/01 13:02:41][INFO]    at SimpleJson.SimpleJson.SerializeObject(Object json, IJsonSerializerStrategy jsonSerializerStrategy)
[2019/01/01 13:02:41][INFO]    at SimpleJson.SimpleJson.SerializeObject(Object json)
[2019/01/01 13:02:41][INFO]    at BenchmarkDotNet.Exporters.Json.JsonExporterBase.ExportToLog(Summary summary, ILogger logger)
[2019/01/01 13:02:41][INFO]    at BenchmarkDotNet.Exporters.ExporterBase.ExportToFiles(Summary summary, ILogger consoleLogger)
[2019/01/01 13:02:41][INFO]    at BenchmarkDotNet.Exporters.CompositeExporter.<>c__DisplayClass5_0.<ExportToFiles>b__0(IExporter exporter)
[2019/01/01 13:02:41][INFO]    at System.Linq.Enumerable.SelectManySingleSelectorIterator`2.MoveNext()
[2019/01/01 13:02:41][INFO]    at BenchmarkDotNet.Running.BenchmarkRunner.PrintSummary(ILogger logger, IConfig config, Summary summary)
[2019/01/01 13:02:41][INFO]    at BenchmarkDotNet.Running.BenchmarkRunner.Run(BenchmarkRunInfo[] benchmarkRunInfos, IConfig commonSettingsConfig)
[2019/01/01 13:02:41][INFO]    at BenchmarkDotNet.Running.BenchmarkSwitcher.Run(String[] args, IConfig config)
[2019/01/01 13:02:41][INFO]    at MicroBenchmarks.Program.Main(String[] args) in D:\git\performancefork\src\benchmarks\micro\Program.cs:line 19
[2019/01/01 13:02:43][INFO] // * Artifacts cleanup *
[2019/01/01 13:02:44][ERROR] Process exited with status 3762504530
[2019/01/01 13:02:44][INFO] $ popd
Traceback (most recent call last):
  File "D:\git\performancefork\scripts\benchmarks_ci.py", line 512, in <module>
    __main(sys.argv[1:])
  File "D:\git\performancefork\scripts\benchmarks_ci.py", line 505, in __main
    micro_benchmarks.run(args.configuration, framework, verbose, *run_args)
  File "D:\git\performancefork\scripts\micro_benchmarks.py", line 262, in run
    BENCHMARKS_CSPROJ.run(configuration, framework, verbose, *args)
  File "D:\git\performancefork\scripts\dotnet.py", line 149, in run
    self.working_directory)
  File "D:\git\performancefork\scripts\performance\common.py", line 196, in run
    proc.returncode, quoted_cmdline)
subprocess.CalledProcessError: Command '$ dotnet run --project MicroBenchmarks.csproj --configuration Release --framework netcoreapp3.0 --no-restore --no-build -- --coreRun D:\git\corefxupstream\artifacts\bin\testhost\netcoreapp-Windows_NT-Release-x64\shared\Microsoft.NETCore.App\9.9.9\CoreRun.exe --filter *.Perf_Dictionary.* --join --packages D:\git\performancefork\artifacts\packages --runtimes netcoreapp3.0' returned non-zero exit status 3762504530.

D:\git\performancefork (dicperf -> origin)
λ

2. (FIXED IN Allow more than one CoreRun benchmarks_ci.py script #210 need a way to pass more than one custom corerun because I want compare pre and post updates. How can I pass "list" of arguments to --corerun?For instance

D:\git\performancefork (dicperf -> origin)
λ .\scripts\benchmarks_ci.py --frameworks netcoreapp3.0 --filter *.Perf_Dictionary.* --corerun D:\git\corefxupstream\artifacts\bin\testhost\netcoreapp-Windows_NT-Release-x64\shared\Microsoft.NETCore.App\9.9.9\CoreRun.exe D:\git\corefx\artifacts\bin\testhost\netcoreapp-Windows_NT-Release-x64\shared\Microsoft.NETCore.App\9.9.9\CoreRun.exe --bdn-arguments \"--join\"
usage: benchmarks_ci.py [-h] [--architecture {x64,x86,arm32,arm64}] -f
                        {netcoreapp3.0,netcoreapp2.2,netcoreapp2.1,netcoreapp2.0,net461}
                        [{netcoreapp3.0,netcoreapp2.2,netcoreapp2.1,netcoreapp2.0,net461} ...]
                        [--incremental {yes,no}] [--enable-hardware-counters]
                        [--category {coreclr,corefx}]
                        [--filter FILTER [FILTER ...]] [--corerun CORERUN]
                        [--cli CLI] [--bdn-arguments BDN_ARGUMENTS]
                        [--optimization-level {tiered,full_opt,min_opt}]
                        [--cli-source-info {init-tools,repo,cli}]
                        [--cli-branch CLI_BRANCH]
                        [--cli-commit-sha CLI_COMMIT_SHA]
                        [--cli-repository CLI_REPOSITORY]
                        [--cli-source-timestamp CLI_SOURCE_TIMESTAMP]
                        [--generate-benchview-data]
                        [--upload-to-benchview-container UPLOAD_TO_BENCHVIEW_CONTAINER]
                        [--benchview-submission-name BENCHVIEW_SUBMISSION_NAME]
                        [--benchview-run-type {rolling,private,local}]
                        [--benchview-config-name BENCHVIEW_CONFIG_NAME]
                        [--benchview-machinepool BENCHVIEW_MACHINEPOOL]
                        [--benchview-config key value] [-q]
benchmarks_ci.py: error: unrecognized arguments: D:\git\corefx\artifacts\bin\testhost\netcoreapp-Windows_NT-Release-x64\shared\Microsoft.NETCore.App\9.9.9\CoreRun.exe

D:\git\performancefork (dicperf -> origin)
λ

In both cases if I directly use BDN it works well i.e.:

..\..\..\..\netcorepreviewsdk\dotnet.exe run -c Release -f netcoreapp3.0 -- --join --filter *.Perf_Dictionary.* --cli ..\..\..\..\netcorepreviewsdk\dotnet.exe --coreRun D:\git\corefxupstream\artifacts\bin\testhost\netcoreapp-Windows_NT-Release-x64\shared\Microsoft.NETCore.App\9.9.9\CoreRun.exe D:\git\corefx\artifacts\bin\testhost\netcoreapp-Windows_NT-Release-x64\shared\Microsoft.NETCore.App\9.9.9\CoreRun.exe 

Thank's for your time, I hope to be useful to make this tool better!

[jorive]

@MarcoRossignoli when using the --bdn-arguments option, I will suggest using it with =. For example, --bdn-arguments="--join". This is a limitation on the argparse module, and there is a whole issue on it and the design decisions.

[MarcoRossignoli]

@jorive thank's a lot!We resolved point 2 of my list, I get again first one...strange error:

[2019/01/02 19:41:39][INFO] // Validating benchmarks:
[2019/01/02 19:41:39][INFO] // ***** BenchmarkRunner: Start   *****
[2019/01/02 19:41:39][INFO] // ***** Building 2 exe(s) in Parallel: Start   *****
[2019/01/02 19:42:14][INFO] // ***** Done, took 00:00:34 (34.98 sec)   *****
[2019/01/02 19:42:14][INFO] // ***** Failed to build in Parallel, switching to sequential build..   *****
[2019/01/02 19:42:14][INFO] // ***** Done, took 00:00:35 (35 sec)   *****
[2019/01/02 19:42:14][INFO] // Found benchmarks:
[2019/01/02 19:42:14][INFO] //   Perf_Dictionary.ContainsValue: Job-EHBXCI(Runtime=Core, Toolchain=\corefx\artifacts\bin\testhost\netcoreapp-Windows_NT-Release-x64\shared\Microsoft.NETCore.App\9.9.9\CoreRun.exe, IterationTime=250.0000 ms, MaxIterationCount=20, MinIterationCount=15, WarmupCount=1) [Items=3000]
[2019/01/02 19:42:14][INFO] //   Perf_Dictionary.ContainsValue: Job-VFUUXV(Runtime=Core, Toolchain=\corefxupstream\artifacts\bin\testhost\netcoreapp-Windows_NT-Release-x64\shared\Microsoft.NETCore.App\9.9.9\CoreRun.exe, IterationTime=250.0000 ms, MaxIterationCount=20, MinIterationCount=15, WarmupCount=1) [Items=3000]
[2019/01/02 19:42:14][INFO]
[2019/01/02 19:42:14][INFO] // Generate Exception: The file 'D:\git\corefx\artifacts\bin\testhost\netcoreapp-Windows_NT-Release-x64\shared\Microsoft.NETCore.App\744cacb6-58c7-4897-9cbe-0bf037a50b60\il\System.Private.CoreLib.dll' already exists.
[2019/01/02 19:42:14][INFO]
[2019/01/02 19:42:14][INFO] // Generate Exception: The file 'D:\git\corefxupstream\artifacts\bin\testhost\netcoreapp-Windows_NT-Release-x64\shared\Microsoft.NETCore.App\78566f6a-cefd-4969-840b-162f8afb74f5\il\System.Private.CoreLib.dll' already exists.
[2019/01/02 19:42:14][INFO]
[2019/01/02 19:42:14][INFO] Run time: 00:00:00 (0.07 sec), executed benchmarks: 0
[2019/01/02 19:42:14][INFO]
[2019/01/02 19:42:14][INFO] // ***** BenchmarkRunner: Finish  *****
[2019/01/02 19:42:14][INFO]
[2019/01/02 19:42:14][INFO] // * Export *
[2019/01/02 19:42:14][INFO]   D:\git\performance\artifacts\bin\MicroBenchmarks\Release\netcoreapp3.0\BenchmarkDotNet.Artifacts\results\System.Collections.Tests.Perf_Dictionary-report.csv
[2019/01/02 19:42:14][INFO]   D:\git\performance\artifacts\bin\MicroBenchmarks\Release\netcoreapp3.0\BenchmarkDotNet.Artifacts\results\System.Collections.Tests.Perf_Dictionary-report-github.md
[2019/01/02 19:42:14][INFO]   D:\git\performance\artifacts\bin\MicroBenchmarks\Release\netcoreapp3.0\BenchmarkDotNet.Artifacts\results\System.Collections.Tests.Perf_Dictionary-report.html
[2019/01/02 19:42:14][INFO]
[2019/01/02 19:42:14][INFO] Unhandled Exception: System.ArgumentNullException: Value cannot be null.
[2019/01/02 19:42:14][INFO] Parameter name: source
[2019/01/02 19:42:14][INFO]    at System.Linq.Enumerable.Any[TSource](IEnumerable`1 source)
[2019/01/02 19:42:14][INFO]    at BenchmarkDotNet.Exporters.Json.JsonExporterBase.<>c__DisplayClass9_0.<ExportToLog>b__0(BenchmarkReport report)
[2019/01/02 19:42:14][INFO]    at System.Linq.Enumerable.SelectArrayIterator`2.MoveNext()
[2019/01/02 19:42:14][INFO]    at SimpleJson.SimpleJson.SerializeArray(IJsonSerializerStrategy jsonSerializerStrategy, IEnumerable anArray, StringBuilder builder)
[2019/01/02 19:42:14][INFO]    at SimpleJson.SimpleJson.SerializeValue(IJsonSerializerStrategy jsonSerializerStrategy, Object value, StringBuilder builder)
[2019/01/02 19:42:14][INFO]    at SimpleJson.SimpleJson.SerializeObject(IJsonSerializerStrategy jsonSerializerStrategy, IEnumerable keys, IEnumerable values, StringBuilder builder)
[2019/01/02 19:42:14][INFO]    at SimpleJson.SimpleJson.SerializeValue(IJsonSerializerStrategy jsonSerializerStrategy, Object value, StringBuilder builder)
[2019/01/02 19:42:14][INFO]    at SimpleJson.SimpleJson.SerializeObject(Object json, IJsonSerializerStrategy jsonSerializerStrategy)
[2019/01/02 19:42:14][INFO]    at SimpleJson.SimpleJson.SerializeObject(Object json)
[2019/01/02 19:42:14][INFO]    at BenchmarkDotNet.Exporters.Json.JsonExporterBase.ExportToLog(Summary summary, ILogger logger)
[2019/01/02 19:42:14][INFO]    at BenchmarkDotNet.Exporters.ExporterBase.ExportToFiles(Summary summary, ILogger consoleLogger)
[2019/01/02 19:42:14][INFO]    at BenchmarkDotNet.Exporters.CompositeExporter.<>c__DisplayClass5_0.<ExportToFiles>b__0(IExporter exporter)
[2019/01/02 19:42:14][INFO]    at System.Linq.Enumerable.SelectManySingleSelectorIterator`2.MoveNext()
[2019/01/02 19:42:14][INFO]    at BenchmarkDotNet.Running.BenchmarkRunner.PrintSummary(ILogger logger, IConfig config, Summary summary)
[2019/01/02 19:42:14][INFO]    at BenchmarkDotNet.Running.BenchmarkRunner.Run(BenchmarkRunInfo[] benchmarkRunInfos, IConfig commonSettingsConfig)
[2019/01/02 19:42:14][INFO]    at BenchmarkDotNet.Running.BenchmarkSwitcher.Run(String[] args, IConfig config)
[2019/01/02 19:42:14][INFO]    at MicroBenchmarks.Program.Main(String[] args) in D:\git\performance\src\benchmarks\micro\Program.cs:line 19
[2019/01/02 19:42:17][INFO] // * Artifacts cleanup *
[2019/01/02 19:42:17][ERROR] Process exited with status 3762504530

I compare master(corefxupstream) with my "work in progress" repo corefx

[jorive]

Looking into it.