AIK causes odd downstream effects, breaks algorithm

[GoogleCodeExporter]

What steps will reproduce the problem?
1. Set 'MinimizeFeatures' to true
2. Run demo3.m (binary feature example)
3. Note odd behaviour

What is the expected output? What do you see instead?
When using 'MinimizeFeatures'=false, the algorithm quickly converges to the a 
feature set containing the 3 features used in the output. However, it also 
contains extra, useless features. Thus an AIK penalty seems logical, but causes 
odd effects.

1) With the AIK criterion, the cost value is now different from expected (i.e. 
AUROC costs go from 0.8 -> -2.3, etc)
2) The AIK cost causes the plot to look odd
3) The algorithm no longer converges to a feature set. In fact, it removes all 
features.

Likely all of the above is due to the nature of arbitrarily setting a threshold 
in the fs_opt function. The level is too high compared to a cost like the 
AUROC, so the algorithm determines that no features + 0 AUROC is better than 1 
feature + non-zero AUROC. 

Action plan: Try to dynamically set the threshold values based on the problem 
set, i.e. t = 1/num_features, or something. Requires literature review. For 
now, use 'MinimizeFeatures' with care, and only to compare with algorithm runs 
without 'MinimizeFeatures'.

Further work: Need to find some way to have the GA optimize a minimal feature 
set with good cost evaluation, rather than just good cost evaluation. This 
would remove the need to have GA repetitions.

Original issue reported on code.google.com by alistair...@gmail.com on 9 Dec 2011 at 11:47