|
1 | | -# SigOpt Strategy |
| 1 | +SigOpt Strategy |
| 2 | +============ |
2 | 3 |
|
3 | | -[SigOpt](https://app.sigopt.com/) is available via online platform and can be used for model development and performance. [Optimization Loop](https://app.sigopt.com/docs/overview/optimization) is the backbone of using SigOpt, we can set metrics and realize the interaction between online platform and tuning configures based on this mechanism. |
| 4 | +1. [Introduction](#introduction) |
4 | 5 |
|
5 | | -## Preparation |
| 6 | + 1.1 [Preparation](#preparation) |
| 7 | + |
| 8 | + 1.2 [SigOpt Platform](#sigopt-platform) |
| 9 | + |
| 10 | + 1.3 [Neural Compressor Configuration](#neural-compressor-configuration) |
6 | 11 |
|
7 | | -Before using `SigOpt` strategy, a SigOpt account is necessary. |
8 | | -- Each account has its own api token. Find your api token and then fill in the configure item `sigopt_api_token`. |
9 | | -- Create a new project and write the corresponding name into the configure item `sigopt_project_id`. |
10 | | -- Set the name for this experiment in configure item `sigopt_experiment_id`, the default is nc-tune. |
| 12 | +2. [Performance](#performance) |
| 13 | + |
| 14 | + 2.1 [Benefit of SigOpt Strategy](#benefit-of-sigopt-strategy) |
11 | 15 |
|
12 | | -### SigOpt introduction |
| 16 | + 2.2 [Performance Comparison of Different Strategies](#performance-comparison-of-different-strategies) |
13 | 17 |
|
14 | | -If you are using SigOpt products for the first time, please [sign-up](https://app.sigopt.com/signup), if not, please [login](https://app.sigopt.com/login). It is free to apply for an account. Although there are certain restrictions on the model parameters and the number of experiments created, it is sufficient for ordinary customers. If you want higher capacity, please contact support@sigopt.com. |
| 18 | +## Introduction |
15 | 19 |
|
16 | | -After logging in, you can use `the token api` to connect the local code and the online platform, corresponding to the configure item `sigopt_api_token`, it can be obtained [here](https://app.sigopt.com/tokens/info). |
| 20 | +[SigOpt](https://app.sigopt.com/) is an online model development platform that makes it easy to track runs, visualize training, and scale hyperparameter optimization for any type of model. [Optimization Loop](https://app.sigopt.com/docs/overview/optimization) is the backbone of using SigOpt. We can set metrics and realize the interaction between the online platform and tuning configurations based on this mechanism. |
17 | 21 |
|
18 | | -SigOpt has two concepts: [project](https://app.sigopt.com/projects) and [experiment](https://app.sigopt.com/experiments). Create a project before experimenting, corresponding to `sigopt_project_id` and `sigopt_experiment_name`, Multiple experiments can be created in each project. After creating experiment, run through these three simple steps, in a loop: |
| 22 | +### Preparation |
19 | 23 |
|
20 | | -- Receive a Suggestion from SigOpt |
21 | | -- Evaluate your metric |
22 | | -- Report an Observation to SigOpt |
| 24 | +Before using the `SigOpt` strategy, a SigOpt account is necessary. |
| 25 | +- Each account has its own API token. Find your API token and then fill it in the `sigopt_api_token` field. |
| 26 | +- Create a new project and fill the corresponding name into the `sigopt_project_id` field. |
| 27 | +- Set the name of this experiment in `sigopt_experiment_id` field optionally. The default name is "nc-tune". |
23 | 28 |
|
24 | | -In Neural Compressor sigopt strategy, the metrics add accuracy as constraint and optimize for latency. |
| 29 | +### SigOpt Platform |
25 | 30 |
|
26 | | -### Neural Compressor configuration |
| 31 | +If you are using the SigOpt products for the first time, please [sign-up](https://app.sigopt.com/signup), if not, please [login](https://app.sigopt.com/login). It is free to apply for an account. Although there are certain restrictions on the model parameters and the number of experiments created, it is sufficient for ordinary customers. If you want higher capacity, please contact support@sigopt.com. |
27 | 32 |
|
28 | | -Compare to `Basic` strategy, `sigopt_api_token` is necessary for `SigOpt` strategy. Create the corresponding project name `sigopt_project_id` in the account before using the strategy. |
| 33 | +After logging in, you can use `the token api` to connect the local code to the online platform, corresponding to `sigopt_api_token`. It can be obtained [here](https://app.sigopt.com/tokens/info). |
29 | 34 |
|
30 | | -```yaml |
31 | | -tuning: |
32 | | - strategy: |
33 | | - name: sigopt |
34 | | - sigopt_api_token: YOUR-ACCOUNT-API-TOKEN |
35 | | - sigopt_project_id: PROJECT-ID |
36 | | - sigopt_experiment_name: nc-tune |
37 | | - accuracy_criterion: |
38 | | - relative: 0.01 |
39 | | - exit_policy: |
40 | | - timeout: 0 |
41 | | - random_seed: 9527 |
| 35 | +SigOpt has two concepts: [project](https://app.sigopt.com/projects) and [experiment](https://app.sigopt.com/experiments). Create a project before experimenting, corresponding to `sigopt_project_id` and `sigopt_experiment_name`. Multiple experiments can be created on each project. After creating the experiment, SigOpt will execute three simple steps below in a loop: |
42 | 36 |
|
43 | | -``` |
| 37 | +- Receive a Suggestion from SigOpt; |
| 38 | +- Evaluate your metrics; |
| 39 | +- Report an Observation to SigOpt; |
| 40 | + |
| 41 | +In our build-in sigopt strategy, the metrics add accuracy as a constraint and optimize for latency. |
| 42 | + |
| 43 | +### Neural Compressor Configuration |
44 | 44 |
|
| 45 | +Compare to `Basic` strategy, `sigopt_api_token` and `sigopt_project_id` is necessary for `SigOpt` strategy. Before using the strategy, it is required to create the project corresponding to `sigopt_project_id` in your account. |
| 46 | + |
| 47 | +```python |
| 48 | +from neural_compressor import config |
| 49 | + |
| 50 | +conf = config.PostTrainingQuantConfig( |
| 51 | + tuning_criterion=config.TuningCriterion( |
| 52 | + strategy="sigopt", |
| 53 | + strategy_kwargs={ |
| 54 | + "sigopt_api_token": "YOUR-ACCOUNT-API-TOKEN", |
| 55 | + "sigopt_project_id": "PROJECT-ID", |
| 56 | + "sigopt_experiment_name": "nc-tune", |
| 57 | + }, |
| 58 | + ), |
| 59 | +) |
| 60 | +``` |
45 | 61 |
|
46 | 62 | ## Performance |
47 | 63 |
|
48 | | -### Benefit for Sigopt strategy |
| 64 | +### Benefit of SigOpt Strategy |
49 | 65 |
|
50 | | -- Metric based the SigOpt is better than self-define and easy to use. you can read details from [here](https://app.sigopt.com/docs/overview/metric_constraints). |
51 | | -- Through the token api, the results of each experiment are recorded in your account. You can use the SigOpt data analysis function to analyze the results, such as drawing a chart, calculating F1 score, etc. |
| 66 | +- Metric based SigOpt is better than self-defining and easy to use. You can read the details [here](https://app.sigopt.com/docs/overview/metric_constraints). |
| 67 | +- With the token api, results of each experiment are recorded in your account. You can use the SigOpt data analysis function to analyze the results, such as drawing a chart, calculating the F1 score, etc. |
52 | 68 |
|
53 | | -### Performance comparison of different strategies |
| 69 | +### Performance Comparison of Different Strategies |
54 | 70 |
|
55 | | -MobileNet_v1 tensorflow |
| 71 | +- MobileNet_v1(tensorflow) |
56 | 72 |
|
57 | | -|strategy|FP32 baseline|int8 accuracy|int8 duration(s)| |
58 | | -|--------|-------------|-------------|----------------| |
59 | | -| basic | 0.8266 | 0.8372 | 88.2132 | |
60 | | -| sigopt | 0.8266 | 0.8372 | 83.7495 | |
| 73 | + |strategy|FP32 baseline|int8 accuracy|int8 duration(s)| |
| 74 | + |--------|-------------|-------------|----------------| |
| 75 | + | basic | 0.8266 | 0.8372 | 88.2132 | |
| 76 | + | sigopt | 0.8266 | 0.8372 | 83.7495 | |
61 | 77 |
|
62 | | -ResNet50_v1 tensorflow |
| 78 | +- ResNet50_v1(tensorflow) |
63 | 79 |
|
64 | | -|strategy|FP32 baseline|int8 accuracy|int8 duration(s)| |
65 | | -|--------|-------------|-------------|----------------| |
66 | | -| basic | 0.8299 | 0.8294 | 85.0837 | |
67 | | -| sigopt | 0.8299 | 0.8291 | 83.4469 | |
| 80 | + |strategy|FP32 baseline|int8 accuracy|int8 duration(s)| |
| 81 | + |--------|-------------|-------------|----------------| |
| 82 | + | basic | 0.8299 | 0.8294 | 85.0837 | |
| 83 | + | sigopt | 0.8299 | 0.8291 | 83.4469 | |
68 | 84 |
|
0 commit comments