Week of July 13th
- Detects non-defective classes accurately, as the number of defects increases, the prediction accuracy, pd and pf gets worse.
- My Solution: Use raw bugs on;y while building the trees, otherwise use boolean values.
| Bugs | Pd | Pf |
|---|---|---|
| 0 | 0.71 | 0.33 |
| 1 | 0.27 | 0.76 |
| 2 | 0.03 | 0.95 |
| 3 | 0.19 | 0.88 |
| 4 | 0.07 | 0.97 |
| 5 | 0.00 | 1.00 |
| 6 | 0.00 | 1.00 |
| Bugs | Pd | Pf |
|---|---|---|
| 0 | 0.77 | 0.34 |
| 1 | 0.35 | 0.77 |
| 2 | 0.03 | 0.99 |
| 3 | 0.12 | 0.92 |
| 4 | 0.067 | 0.99 |
| 5 | 0.0 | 0.99 |
| 6 | 0.0 | 0.99 |
| Bugs | Pd | Pf |
|---|---|---|
| 0 | 0.67 | 0.34 |
| 1 | 0.27 | 0.74 |
| 2 | 0.10 | 0.91 |
| 3 | 0.19 | 0.88 |
| 4 | 0.07 | 0.97 |
| Bugs | Pd | Pf |
|---|---|---|
| 0 | 0.75 | 0.31 |
| 1 | 0.27 | 0.79 |
| 2 | 0.03 | 0.97 |
| 3 | 0.19 | 0.89 |
| 4 | 0.07 | 0.97 |
| 5 | 0.00 | 1.00 |
| 6 | 0.00 | 1.00 |
| Bugs | Pd | Pf |
|---|---|---|
| 0 | 0.79 | 0.37 |
| 1 | 0.34 | 0.77 |
| 2 | 0.00 | 0.99 |
| 3 | 0.19 | 0.94 |
| 4 | 0.07 | 0.99 |
| 5 | 0.00 | 1.00 |
| 6 | 0.00 | 1.00 |
- Ran CROSSTREES with 3 baseline studies:
- Apply the ∆ in the mean/centroid as a baseline -> (Clusters-Centroid-∆s).
- (Clusters-Centroid-∆s)+InfoPrune.
- Decision Trees w/o clustering.
- CROSSTREES (with/without WHERE) is generally better than baseline. Especially, given how few changes are suggested.
- I don't see the need for Continuous classes, except for Ivy the improvement is minimal. In fact, using that makes Lucene worse.
Continuous Classes:
rank , name , med , iqr
----------------------------------------------------
1 , CART , 0.63 , 0.16 (* | ), 0.63, 0.63, 0.79
1 , CART + WHERE , 0.70 , 0.06 ( * | ), 0.70, 0.70, 0.76
1 , Baseline (No Prune) , 0.72 , 0.16 ( --------* | ), 0.65, 0.72, 0.81
2 , Baseline (Prune) , 0.86 , 0.04 ( | ---* ), 0.84, 0.86, 0.87
Continuous Classes:
rank , name , med , iqr
----------------------------------------------------
1 , CART, 0.43 , 0.14 (* | ), 0.43, 0.43, 0.57
1 , (CART WHERE) , 0.43 , 0.00 (* | ), 0.43, 0.43, 0.43
2 , Baseline (Prune) , 0.86 , 0.02 ( | * ), 0.85, 0.86, 0.87
3 , Baseline (No Prune) , 0.91 , 0.03 ( | -* ), 0.89, 0.91, 0.92
Continuous Classes:
rank , name , med , iqr
----------------------------------------------------
1 , CART + WHERE , 0.74 , 0.06 ( -* | ), 0.73, 0.74, 0.79
1 , CART , 0.78 , 0.06 ( -------*| ), 0.73, 0.78, 0.80
2 , Baseline (Prune) , 0.86 , 0.02 ( | -* ), 0.85, 0.86, 0.87
2 , Baseline (No Prune) , 0.87 , 0.02 ( | ---* ), 0.85, 0.87, 0.88
Continuous Classes:
rank , name , med , iqr
----------------------------------------------------
1 , Baseline (No Prune) , 0.77 , 0.14 ( --------* | ), 0.70, 0.77, 0.84
2 , Baseline (Prune) , 0.85 , 0.04 ( | --* ), 0.84, 0.85, 0.88
2 , CART + WHERE , 0.87 , 0.02 ( | -* ), 0.86, 0.87, 0.88
2 , CART , 0.88 , 0.02 ( | ---* ), 0.86, 0.88, 0.88
Boolean Classes:
rank , name , med , iqr
----------------------------------------------------
1 , Baseline (No Prune) , 0.77 , 0.14 ( -------* | ), 0.70, 0.77, 0.84
2 , Baseline (Prune) , 0.85 , 0.04 ( | --* ), 0.84, 0.85, 0.88
3 , CART , 0.89 , 0.05 ( | ---* ), 0.86, 0.89, 0.91
3 , CART + WHERE , 0.91 , 0.04 ( | -* ), 0.90, 0.91, 0.94
Continuous Classes:
rank , name , med , iqr
----------------------------------------------------
1 , Baseline (Prune) , 0.87 , 0.02 ( ---* | ), 0.85, 0.87, 0.87
2 , Baseline (No Prune) , 0.88 , 0.02 ( --* | ), 0.87, 0.88, 0.89
2 , CART , 0.89 , 0.02 ( ---* | ), 0.88, 0.89, 0.90
3 , CART + WHERE , 0.92 , 0.02 ( | -* ), 0.92, 0.92, 0.94
Boolean Classes:
rank , name , med , iqr
----------------------------------------------------
1 , CART + WHERE , 0.73 , 0.16 ( -----*| ), 0.67, 0.73, 0.83
1 , CART , 0.78 , 0.13 ( ----|--* ), 0.70, 0.78, 0.83
2 , Baseline (Prune) , 0.87 , 0.02 ( | -* ), 0.85, 0.87, 0.87
3 , Baseline (No Prune) , 0.88 , 0.02 ( | -* ), 0.87, 0.88, 0.89
- Ran CROSSTREES on XOMO and POM3. Individually for each objective, and then for all the objectives considered at once.
- Used the continuous domination formula
sum(exp(objs)) - Individual is better than All-Objectives.
- Unable to optimize some objectives in XOMO — Months and Risk. Using xtrees makes it worse.
rank , name , med , iqr
----------------------------------------------------
1 , -Defects , 0.50 , 0.05 ( -* | ), 0.48, 0.50, 0.53
2 , -Effort , 0.68 , 0.02 ( * | ), 0.67, 0.68, 0.70
3 , All Obj , 0.87 , 0.00 ( | * ), 0.87, 0.87, 0.87
4 , -Months , 1.02 , 0.01 ( | -* ), 1.01, 1.02, 1.02
5 , -Risk , 1.10 , 0.01 ( | *), 1.10, 1.10, 1.11
rank , name , med , iqr
----------------------------------------------------
1 , -Cost , 0.46 , 0.01 ( * | ), 0.45, 0.46, 0.46
2 , -Idle , 0.95 , 0.08 ( | --* ), 0.90, 0.95, 0.97
2 , +Completion , 0.97 , 0.03 ( | -* ), 0.96, 0.97, 0.99
2 , All Obj , 0.98 , 0.01 ( | * ), 0.98, 0.98, 0.99
- I believe the poor performance is due to the Regression.
- I'd like to discretize using FI discretizer and use classifier instead. Not sure if this is viable..
rank , name , med , iqr
----------------------------------------------------
1 , X264 , 0.81 , 0.08 ( ----* | ), 0.77, 0.81, 0.84
2 , BDBJ , 0.95 , 0.01 ( | -* ), 0.94, 0.95, 0.95
3 , Apache , 0.99 , 0.01 ( | * ), 0.99, 0.99, 0.99
3 , SQL , 0.99 , 0.00 ( | -* ), 0.99, 0.99, 0.99
3 , LLVM , 1.00 , 0.00 ( | * ), 1.00, 1.00, 1.00