Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

PredictionEngine benchmark test #1013

Closed
najeeb-kazmi opened this issue Sep 24, 2018 · 6 comments

Comments

Projects
None yet
5 participants
@najeeb-kazmi
Copy link
Member

commented Sep 24, 2018

Add benchmark test to measure performance of single predictions made by PredictionEngine.

@justinormont justinormont added the perf label Sep 25, 2018

justinormont added a commit that referenced this issue Sep 26, 2018

Adding benchmark test for PredictionEngine (#1014)
Adds a benchmark test to measure performance of doing many single predictions with PredictionEngine.

Closes #1013
@najeeb-kazmi

This comment has been minimized.

Copy link
Member Author

commented Oct 2, 2018

Edit: Adding prediction runtimes for the legacy LearningPipeline API.

Benchmarks

In the ML.NET 0.6 release, we made a couple of performance improvements in making single predictions from a trained model. The first improvement comes from moving from the legacy LearningPipeline API to the new Estimators API. The second improvement comes from optimizing the performance of PredictionFunction in the new API.

Here is a comparison of runtimes for single predictions between the old LearningPipeline API, the new Estimators API with PredictionFunction, and the new Estimators API with the improved PredictionFunction. The benchmarks do a single prediction 10000 times on three different models, and each benchmark is run 20 times, with the average runtimes reported in the tables below, along with the standard deviations. We see the following speedups in average runtime, comparing the LearningPipeline with the improved PredictionFunction in the new Estimators API:

  • Predictions on Iris data: 3,272x speedup (29x speedup with the Estimators API, with a further 112x speedup with improvements to PredictionFunction).
  • Predictions on Sentiment data: 198x speedup (22.8x speedup with the Estimators API, with a further 8.68x speedup with improvements to PredictionFunction). This model contains a text featurizer, so it is not surprising that we see a smaller gain.
  • Predictions on Breast Cancer data: 6,541x speedup (59.7x speedup with the Estimators API, with a further 109x speedup with improvements to PredictionFunction).

Predictions with LearningPipeline API:

Method Mean Error StdDev
MakeIrisPredictions 9.879 s 0.1461 s 0.1295 s
MakeSentimentPredictions 10.225 s 0.0915 s 0.0856 s
MakeBreastCancerPredictions 8.850 s 0.1622 s 0.1518 s

Predictions with Estimators API, old PredictionFunction:

Method Mean Error StdDev
MakeIrisPredictions 338.0 ms 4.951 ms 4.389 ms
MakeSentimentPredictions 447.7 ms 7.453 ms 6.607 ms
MakeBreastCancerPredictions 148.2 ms 2.317 ms 2.054 ms

Predictions with Estimators API, new improved PredictionFunction:

Method Mean Error StdDev
MakeIrisPredictions 3.019 ms 0.0388 ms 0.0363 ms
MakeSentimentPredictions 51.575 ms 0.3298 ms 0.3085 ms
MakeBreastCancerPredictions 1.353 ms 0.0059 ms 0.0055 ms

cc: @GalOshri @TomFinley @Zruty0 @justinormont

@Zruty0

This comment has been minimized.

Copy link
Contributor

commented Oct 2, 2018

@najeeb-kazmi , is this LearningPipeline vs. PredictionFunction, or is it old PredictionFunction vs. new?

@najeeb-kazmi

This comment has been minimized.

Copy link
Member Author

commented Oct 2, 2018

This is old PredictionFunction vs new.

@najeeb-kazmi

This comment has been minimized.

Copy link
Member Author

commented Oct 2, 2018

@Zruty0 @GalOshri @shauheen @TomFinley @justinormont I've updated the numbers with prediction runtimes with the LearningPipeline API. The speedups are even more impressive when compared to how slow things were in 0.5.

@TomFinley

This comment has been minimized.

Copy link
Contributor

commented Oct 3, 2018

Um wow.

Still, 1.353 ms per 10k on BC means, 135 nanoseconds per cycle, which will correspond on a typical machine to around a few hundred CPU cycles. A few hundred CPU cycles for 9 multiplies, followed by 9 multiply-adds, followed by the application of a logistic function, is at least somewhat understandable, but suggests to me that there may be some additional speedups here and there to be had.

But at least the situation is not ridiculous.

@danmosemsft

This comment has been minimized.

Copy link
Member

commented Oct 3, 2018

@adamsitnik curious whether the traces you shared earlier are of the fast case above.

@TomFinley dowe have profile evidence that the time spent is dominated by "9 multiplies, followed by 9 multiply-adds, followed by the application of a logistic function"? If so that seems like something that may be amenable to eg @tannergooding looking at such factors as code gen and whether there is possibly another intrinsic we could use.

Zruty0 added a commit that referenced this issue Oct 5, 2018

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.