Skip to content

Conversation

@adamsitnik
Copy link
Member

I took the benchmarks created by @yaeldekel, solved merge conflict run them and changed the config a little bit to run just one warmup iteration.

@davidwrighton to run all the benchmarks you need to pass * as the filer:

    dotnet run -c Release -- -f * --join

Sample results:

BenchmarkDotNet=v0.11.0, OS=Windows 10.0.17134.228 (1803/April2018Update/Redstone4)
Intel Xeon CPU E5-1650 v4 3.60GHz (Max: 2.90GHz), 1 CPU, 12 logical and 6 physical cores
.NET Core SDK=2.1.301
  [Host] : .NET Core 2.1.1 (CoreCLR 4.6.26606.02, CoreFX 4.6.26606.05), 64bit RyuJIT

Toolchain=InProcessToolchain  MaxIterationCount=20  WarmupCount=1  
Type Method Mean Error StdDev AccuracyMacro Gen 0 Gen 1 Gen 2 Allocated
KMeansAndLogisticRegressionBench TrainKMeansAndLR 2,134.265 ms 164.3370 ms 189.2507 ms 0,98 16000.0000 9000.0000 3000.0000 49949.23 KB
StochasticDualCoordinateAscentClassifierBench TrainSentiment 2,130.503 ms 24.8173 ms 23.2141 ms 0,98 122000.0000 35000.0000 5000.0000 759772.8 KB
StochasticDualCoordinateAscentClassifierBench TrainIris 834.229 ms 254.5284 ms 293.1152 ms 0,98 6000.0000 1000.0000 - 12173.28 KB
StochasticDualCoordinateAscentClassifierBench PredictIris 2.472 ms 0.1202 ms 0.1384 ms 0,98 35.1563 15.6250 3.9063 123.24 KB
StochasticDualCoordinateAscentClassifierBench PredictIrisBatchOf1 2.712 ms 0.3276 ms 0.3773 ms 0,98 35.1563 15.6250 3.9063 123.2 KB
StochasticDualCoordinateAscentClassifierBench PredictIrisBatchOf2 2.370 ms 0.1334 ms 0.1482 ms 0,98 35.1563 15.6250 3.9063 123.31 KB
StochasticDualCoordinateAscentClassifierBench PredictIrisBatchOf5 2.492 ms 0.1678 ms 0.1865 ms 0,98 35.1563 15.6250 3.9063 123.61 KB

using Microsoft.ML.Runtime.Data;
using Microsoft.ML.Runtime.FastTree;
using Microsoft.ML.Runtime.Internal.Calibration;
using Microsoft.ML.Runtime.Learners;

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Learners [](start = 27, length = 8)

The changes to this file are probably not necessary. One of the tests was written here initially, but then moved to the Benchmarks project.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@yaeldekel good point! I have reverted this change

@eerhardt
Copy link
Member

eerhardt commented Aug 24, 2018

Is this a duplicate of #722?

@briancylui @danmosemsft

@briancylui
Copy link
Contributor

briancylui commented Aug 24, 2018

@eerhardt Seems possible to me, and I like the perf results presented above.

Regarding the perf results, KMeansAndLogisticRegression (KMeans+LR) shares the same AccuracyMacro with SDCA (0.98), but since the GetValue method of Program.cs:ClassificationMetricsColumn only references StochasticDualCoordinateAscentClassifierBench.s_metrics (link), which is irrelevant to KMeans+LR, it might be possible that KMeans+LR displayed SDCA's AccuracyMacro as its own metric in the perf results. When I ran dotnet run -c Release and chose KMeans+LR only, the AccuracyMacro was displayed as 0. It may have something to do with the added public variable ClassificationMetrics Empty.

This could be addressed either here or in #724.

@adamsitnik
Copy link
Member Author

Sorry for the confusion, I was told yesterday that this thing needs to be finished. I did not know that @briancylui is also working on this.

Closing since it's a duplicate.

@adamsitnik adamsitnik closed this Aug 24, 2018
@briancylui
Copy link
Contributor

Thank you @adamsitnik for your time and the good work - much appreciated.

@eerhardt
Copy link
Member

No worries, @adamsitnik. Thanks for the help.

@ghost ghost locked as resolved and limited conversation to collaborators Mar 29, 2022
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants