Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Heterogeneous partitioning and quantization have issues when used together #3112

Closed
opti-mix opened this issue Jun 14, 2019 · 2 comments

Comments

Projects
None yet
1 participant
@opti-mix
Copy link
Contributor

commented Jun 14, 2019

I observe that an attempt to run an NN model on heterogeneous devices using a quantization profile produced before on a single device (e.g. CPU) does not always work as expected because after applying the heterogeneous partitioning some nodes are not found in the saved quantization profile and thus cannot be quantized.

@opti-mix

This comment has been minimized.

Copy link
Contributor Author

commented Jun 14, 2019

I had a brief discussion with @jfix71 and @beicy about this issue and how to solve it.

The solution we discussed looks something like this:

  1. The Partitioner should perform its backend kind based partitioning as usual and create multiple partitions
  2. Then if -dump-profile mode is used, the Partitioner should assign CPU to the partitions created in the previous step and do an early exit
  3. The provisioner would then call compiler for each of those partitions and use the -dump-profile mode, which would result in instrumenting the graph to collect quantization profiles for each of the partitions
  4. After the run, the quantization profiles for each partition will be produced and dumped (as a single or multiple profiles)

Later on, to run the quantized model, one would use the usual command-line with -load-profile and this time the partitioner would perform the real heterogeneous partitioning as usual. Since all the nodes in different partitions were recorded in the profile, there should be no problems this time with finding them in the profile and quantization process should succeed.

@opti-mix

This comment has been minimized.

Copy link
Contributor Author

commented Jun 14, 2019

cc @rdzhabarov

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.