Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[FrontEnd] Add Model Tuner front-end tool. #3816

Closed
wants to merge 6 commits into from

Conversation

@mciprian13
Copy link
Contributor

mciprian13 commented Nov 25, 2019

Summary
Add a front-end tool for tuning (calibrating) the quantization parameters of a model. The parameters are tuned using the model accuracy as the optimization metric.

Motivation
The motivation for this tool is this: when using extremely brutal quantization schemes (for example the "SymmetricWithPower2Scale") the accuracy difference between the floating-point model and the quantized model can be pretty high (up to 10's of percents). One such example is an internally designed model which has:

  • 99 % accuracy for the floating-point model
  • 81 % accuracy for the (initial) quantized model using SymmetricWithPower2Scale schema
  • 98 % accuracy for the (final) quantized model after tuning

Attached here is the console output of this Tool after running it for the model:

Algorithm
The quantization parameters are initially chosen such that no saturation occurs (quantized range includes the min/max of the profile). For some of the tensors where the histogram exhibits some outlier values it might be better to use quantization parameters which saturate the outliers with the benefit of having smaller quantization step for the bulk of the histogram.
The tuning algorithm "plays" with the scale quantization parameter by trying sequentially for each node new values around the original value and picks the one which provides the best accuracy (it tries original_scale, original_scale/2 and original_scale/4). One might say that there is a philosophical problem with this approach: the algorithm is over-fitting the quantization parameters for a given dataset. But from a practical point of view it is better to have maybe an over-fitting of a couple of percent than have an under-fitting of 10s of percent for the quantization mechanism.

Refactorizations

  • Had to refactor a a little the Loader class in order to provide more information through its API for the tools which is using the Loader.
  • Refactored Base.cpp to provide a function for validating the quantization parameters.
  • Refactored ProtobufLoader.cpp to add function to retrieve the unique input placeholder for a model with a single input placeholder.

Documentation
doc/ModelTuner.md

Test Plan
None

…Base.cpp. Fixes in ProtobufLoader.cpp.
@mciprian13 mciprian13 changed the title Add Model Tuner front-end tool. [FrontEnd] Add Model Tuner front-end tool. Nov 25, 2019
mciprian13 added 2 commits Nov 25, 2019
@mciprian13

This comment has been minimized.

Copy link
Contributor Author

mciprian13 commented Dec 3, 2019

Question: what is the model-runner front-end application intended for? It assumes the graph inputs are constants so I`m not sure what is the purpose of that.

@stale

This comment has been minimized.

Copy link

stale bot commented Dec 18, 2019

This issue has been automatically marked as stale because it has not had recent activity. It will be closed in 15 days if no further activity occurs. Thank you for your contributions.

@jfix71
jfix71 approved these changes Dec 18, 2019
Copy link
Contributor

jfix71 left a comment

@mciprian13 Thanks for contributing this, looks pretty cool! Just a few small comments.

CC: @rdzhabarov who built most of the initial quantization support in Glow. I'm curious if you all tried adjusting the selection of the quantization range based on the actual histogram as opposed to the simple min/max which we currently do.

Question: what is the model-runner front-end application intended for? It assumes the graph inputs are constants so I'm not sure what is the purpose of that.

It's used just for testing purposes, so the model is standalone/has no other inputs you need to add.

include/glow/Importer/ProtobufLoader.h Outdated Show resolved Hide resolved
tools/loader/CMakeLists.txt Outdated Show resolved Hide resolved
tools/loader/ImageClassifier.cpp Outdated Show resolved Hide resolved
tools/loader/LoaderUtils.cpp Outdated Show resolved Hide resolved
@stale stale bot removed the stale_will_be_closed label Dec 18, 2019
Copy link

facebook-github-bot left a comment

@jfix71 has imported this pull request. If you are a Facebook employee, you can view this diff on Phabricator.

mciprian13 added 3 commits Dec 19, 2019
…tobufLoader.h" to "ProtobufLoader.cpp".
Copy link

facebook-github-bot left a comment

@jfix71 has imported this pull request. If you are a Facebook employee, you can view this diff on Phabricator.

@facebook-github-bot

This comment has been minimized.

Copy link

facebook-github-bot commented Dec 19, 2019

@jfix71 merged this pull request in 41d3e30.

@mciprian13 mciprian13 deleted the mciprian13:Add_Model_Tuner_Tool branch Dec 20, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
3 participants
You can’t perform that action at this time.