Relative Gain

I've been reporting gain as a relativistic measure, which is obtained as below: The reason for my using relative gain is as follows — If I were to compare ‘actual' with ‘after', there is always going to be questions regarding the accuracy of CART as a predictor, there is always an uncertainty wrt to how our oracle (which has no domain knowledge) does its predictions. Also, the numbers we report would be under an inherent assumption that CART is 100% accurate, which is clearly not true.

Now, in order to combat this, I use a 'relative gain'. Here I compare the predictions of oracle before and after applying the deltas from our contrast set. Now, since I do not change any of CART’s attributes and due to the fact that CART is a deterministic classifier, the relative gain that I report would be a measure of the efficacy of contrast set learner and would be devoid of misclassifications. If there were indeed some misclassifications, both before and after would face the same problem.

So, does this seem kosher to you? Or would you rather I use some other measure?