How to improve counterfactuals

[somahota]

Hi @amit-sharma @gaugup
I am working on a dummy HR dataset to generate counterfactuals to prevent employee attrition. 0 means no attrition and 1 means attrition. I am trying to generate counterfactuals from 1(original outcome) to 0.

It is generating incorrect counterfactuals as seen in the screenshot - eg. instead of increasing satisfaction score, it is decreasing it.

https://drive.google.com/file/d/1Hs7jfBFSjE_Ed6jRBCTjiFFzOsl7gMde/view?usp=sharing

I have attached my dataset, would be great if you could take a look.

[gaugup]

@somahota, could you share sample that you are using to generate the counterfactuals?

For the binary classification scenario, you should use desired_class as "opposite".

Regards,

[somahota]

Hi, @gaugup , do you mean share the sample dataset? I have shared it on my original post above. Here, it is https://drive.google.com/file/d/1Hs7jfBFSjE_Ed6jRBCTjiFFzOsl7gMde/view

Also, I have made a small change which is giving me relatively better results but still not good enough (screenshots attached below).
Here's the code to generate counterfactuals:

pred_test = pd.read_csv('modified_hrm.csv')
pred_test = pred_test.loc[pred_test['prediction'] == 1]

pred_test = pred_test.drop(['recently_promoted','last_evaluation_missing','filed_complaint','low_performance'],axis=1)
pred_test = pred_test.drop(['prediction'],axis=1)
e1 =  exp.generate_counterfactuals(pred_test[0:159],
                total_CFs=3,desired_class=0,
                proximity_weight=0.5,
                diversity_weight=1.0, categorical_penalty=0.1,                   
                features_to_vary=["avg_monthly_hrs","last_evaluation","n_projects","satisfaction","salary"])

e1.visualize_as_dataframe(show_only_changes=True)

As you can see, for my 1st query, my counterfactuals increased the last evaluation score which is great but as you can see, the 2nd counterfactual (for my 1st query) set the satisfaction score to 0 instead of increasing it.

Similarly, for the 2nd query, we can clearly see that the only reason why the employee is predicted to leave is because they have a low salary. However, my counterfactuals don't account for that. The expected counterfactual result was to see a medium-high salary.

[gaugup]

Thanks for the sample code and the screen shots @somahota. For the original question of incorrect counterfactuals, when you have a binary classification scenario you may want to set "desired_class" as "opposite".

Happy to see that you were able to improve the counterfactuals.

I am trying to understand the motive behind the counterfactual generation here. Looking at the dataset, I don't think that a higher value of satisfcation is correlated with employee staying. There are samples where employee has left even though the satification rating was high. One such sample below:-

1 300 0 0.957338138 4 0 0.694679799 5 0 0 0 Medium

For the second instance it seems like the last_evaluation and satification did indeed help flip the outcome. However, are these two features HR controlled or employee controlled? I can imagine that features like n_projects and salary is something HR might be able to control, but features like satisfacation might be something that may not be in hands of HR in order to keep the employee from leaving. satification may not be the causal feature in your problem scenario, IMHO. If these are indeed features that HR can control, then this indicates problems with your trained model. Dice-ml simply uses your model for prediction of the sample points in generates.

This is my understanding of your data. Please correct if this is misunderstood here.

Keeping the above two points in mind, In think you should remove satification and/or last_evaluation from your feature_to_vary list and give a try again.

P.S. Please bear in mind that this library solely gives you a way of generating counterfactuals that help you achieve the desired outcome. What counterfactual we end up generating may or may not be useful to you as an end user. It is upto your use case and your interpretation of these results.

[somahota]

Thank you @gaugup for your elaborate response, super helpful. You brought up a great point regarding features being controlled by the HR vs employee. last_evaluation is HR controlled, however, satisfaction is employee controlled.
However, strangely, I am getting better results including both those features in my features_to_vary list.

Also, initially, I did set my "desired_class" as "opposite" but the results were not good/interpretable. Since my desired outcome is to flip from 1 to 0, hardcoding my desired_class to 1 did the job. Not sure why setting it to "opposite" failed.

[gaugup]

@somahota, I would suggest investigating the dataset and the model in more details to see why it might be that including those features help your cf generation. I feel some causal analysis could help you figure out if those features actually affect your outcome and based on the causal analysis you could determine your feature for cf genrration.

Regarding setting the desired_class as opposite could you share sample outputs on what failed for you? Maybe there is bug in this library which we can investigate.

Regards,
Gaurav

[gaugup]

@somahota, if there is no further enquiries on the issue, could you maybe close this issue? It seems we don't have anything to fix in the dice-ml library. What do you think?

Regards,

[somahota]

@gaugup, sorry about the delay in response. I'd be more than happy to share my output setting desired_class as opposite for further investigation, however, it's been a hectic week so I'd only be able to do so by coming weekend.

Let me know if I should leave it open till I share that output or close it.

[gaugup]

No worries. We can keep this open until you post the requested outputs.

Regards,

[somahota]

Hi, @gaugup . I have attached the screenshot of the output when I set the desired_class to "opposite". My understanding was that if it encounters '0' as my original outcome, then it should automatically predict for '1' (new outcome) but it seems it's predicting for 0 from 0. It seems to work the reverse case though (original outcome:1 and new outcome:0). Just not sure why it wouldn't work likewise for original outcome:0.

Also, I encountered another error when trying other parameters (proximity_weight) to improve my counterfactuals.

Am I setting the parameter incorrectly?

[gaugup]

@somahota Thanks for the outputs.

For the first, do you mind sharing a notebook? I see that you have shared the dataset. We should look in more detail on why this should happen.

For second one, are you trying to use "genetic" method within "sklearn" model type? IF so, then we have a bug in our code which doesn't allow to propagate "proximity_weight" to the dice_genetic.py code.

Regards,

[somahota]

@gaugup, sure. Here's the notebook
Github_Notebook.pdf

I tried using those parameters with both "genetic" and "random" method but still the error.

[gaugup]

Thanks @somahota for the notebook. For fixing the error I have created PR #251. Please bear in mind the the parameters like "proximity_weight" only apply to "genetic" and not to "random"/"kdtree".

I am yet to debug issue where setting "opposite" wouldn't generate expected counterfactuals.