Will DRPolicyForest control for confounder in Treatments that occured under different seasons?

[titubs]

@kbattocchi Hi Keith,
I wanted to understand how the DRPolicyForest would behave if I had 2 treatments in which one treatment is randomized data collected during December and the other treatment is randomized data collected during May.

If I were to specify confounder such as seasonal factors, would the DRPolicyForest be able to control for differences in seasons among the two treatments and make them more comparable?

For example:

• Treatment 1 from May generated $5 revenue per user
• Treatment 2 from December generated $8 revenue per user

We cannot compare $5 with $8 directly due to seasonal confounders.

Is there a way with the DRPolicyForest to show case how the adjusted figures for revenue per user would look like?
I.e. User A's observed Treatment 1 generated $4 in revenue. However, User's A de-biased revenue under Treatment 1 would be $3 (after controlling for confounder)?

Is that possible as I am only able to get the counterfactual effect values (model.predict_value(X_test)) for all treatments BUT the observed one. Or is there anything using the counterfactual values to showcase this point?

[kbattocchi]

I'm afraid I'm not fully sure I understand what you're asking. It sounds like since you have (different) randomized treatments in each period, you can compute the treatment effect in each period, but that you worry that the treatment effect from one season can't be assumed to apply to the other. This seems like a very reasonable concern.

If you are willing to assume that "season" acts on the outcome via some other measured covariates (say, temperature and rainfall), then if you include those covariates in W or X (depending on whether you think the season affects only the outcome or possibly also the strength of the effect) and fit a model on the combined data then the model could take those factors into account. However, I would be concerned that there might not be enough in-season variation in these factors to extrapolate from one season to another, and also that "seasonal" factors might include a whole host of other things that you haven't measured (e.g. maybe there are more tourists in town at one time than another), so I wouldn't take any such estimates very seriously unless you're confident that you understand the mechanism by which the season affects the outcomes, that you have measured all necessary factors, and that those factors affect the outcome in a way which an ML model will be able to generalize from one season to the other.

[titubs]

@kbattocchi Thank you and yes, you understood this question correctly. To make sure I understood your reply, let me repeat it back:

• Theoretically, the DRPolicyForest should control for confounder (i.e. seasonal difference between those variants).
• The concern is that there may not be sufficient in-season variation to extrapolate from one season to another. Can you give an example what you mean? Why would there be not enough variation?
• The other concern is that there may be more confounder that are unobserved and may bias treatment effect.

Can you confirm/clarify above?

Furthermore, the issue with the unobserved confounders (3rd bullet) is "we dont know what we dont know", right? Is there any test that you recommend to check how "good" the adjusted treatment effect is (from the DR model) given that I have ground truth for both experiments (since those were randomized trials)? In other words, do you have any idea how we could leverage ground truth (for each variant) to check how good the model controls for confounder? Is this possible? I want to get a temperature if we can trust the model in cases where we train the policy on randomized data that came from difference periods/seasons.

[kbattocchi]

For your second bullet, imagine, for example, that ice cream demand is high when temperatures are above 50 degrees Fahrenheit and low when temperatures are below 50 degrees. Then if summer temperatures are always above 50 and winter temperatures are always below 50 then any attempt to extrapolate from one season to the other will fail. So you either need to ensure that there is overlap (the same situations occur in both seasons, but maybe with different frequencies), or that the functional relationship varies smoothly in a way that you can safely extrapolate from one season to the other.

For your last question, I don't think it's possible, both because of the extrapolation issue (your Y(treatment, weather) model can have zero error but still not generalize correctly out-of-domain), as well as the fact that if the Y model does not fit perfectly, there's no way to tell if this is due to harmless random noise or due to a missing confounder that would actually lead you to compute a different treatment effect if accounted for.