How to evaluate the model?

[jayjoshi33]

I'm currently testing the Meridian MMM model on real-world data. In my dataset, there are two marketing channels:

Channel 1: YouTube, accounting for 92% of total spend

Channel 2: Google Ads, with only 8% of total spend

Using the default priors, the model estimated:

ROI: YouTube = 15, Google Ads = 1.5

R² = 80%, MAPE = 26%

Total contribution = 162%, with YouTube contributing 156%

This led to a negative baseline, which is problematic.

I’m using 90 weeks of data and have included 2–3 control variables in the model.

To address the unrealistic contribution and ROI from YouTube, I tightened the prior on its ROI by reducing mu_sigma from 0.9 to 0.2. This resulted in:

YouTube contribution dropped to 14%

Total contribution from all channels = 17%

The baseline is now positive, which is good

However, R² dropped to 52% and MAPE increased to 55%

YouTube’s ROI fell to 1.4

My Questions:
How should I evaluate the model performance in this case?

Is a higher R² always better even if the baseline becomes unrealistic?

Or should I prioritize realistic contributions and ROIs, even at the cost of lower R²?

How do I know which ROI estimates are “best” or most trustworthy?

Should I always tune priors until the baseline becomes positive and contributions are reasonable?

[cpulavarthi]

Hello @jayjoshi33,

Thank you for contacting us!

It's important to note that Meridian models often benefit from more extensive data. We typically recommend using a minimum of two years' worth of weekly data for geo-level models and three years' of data for national-level models. Your 90 weeks of data (approx. 1.7 years) might be a contributing factor to some of the challenges you're observing, particularly with complex relationships and limited signal for some channels.

I would like to point you towards the documentation on assessing the baseline. This page has detailed debugging steps to mitigate low or negative baseline which can help you resolve your issue. It is great that you have already added control variables to your model, please make sure you aren’t missing any confounders because this is one of the common causes for low/ negative baseline. Another common cause of incorrect baseline attribution is improper prior setting. Prior standard deviation (sigma) needs to be guided by your confidence in the prior estimate of the ROI. Blindly reducing the sigma may bias the model. Even if it is giving plausible ROI estimates, it may not generalize well.

Please consider using suitable ROI priors for your model. You can use past experiments’ data or even your expectations to guide the prior selection. Check our documentation on ROI priors and calibration for guidance regarding prior setting. It may also be worth exploring total media contribution priors which allows you to set the expectations of the contribution towards the target KPI (specifically a non-revenue metric) from all media channels combined.

Meridian uses Bayesian Causal Inference and attempts at modeling causal relationships between the treatments and the target variable. It is difficult to say which predictive accuracy metric is better for evaluating the model performance as the main goal of Meridian isn’t prediction, but rather understanding the causal impact and attribution of marketing efforts to inform strategic decisions. I would recommend using all the metrics, i.e., R^2, MAPE and wMAPE, along with holdout sets to assess model performance, don’t prioritize one over the other unless it makes business sense. But the first step would be to check for convergence using r-hat. While predictive accuracy metrics give valuable insights, also consider thorough analysis of the results using the methods from the Analyzer module to evaluate model’s estimates of various metrics like percentage contributions, effectiveness, etc. You may check our documentation on assessing the model fit and the results.

Coming to your questions:

1. Evaluating model performance is subjective, and we recommend a holistic approach using all the post-modeling metrics. You may choose to prioritize some depending on the business use case.
2. Higher R^2 doesn’t always mean a better model, use other methods from the Analyzer module to check the ROI, contributions, effectiveness, etc estimates. Negative baseline is unrealistic, so this should be investigated further.
3. You may prioritize realistic contributions over lower R^2, but make sure you are not biasing the model towards realistic contributions, i.e., ensure that your modeling choices aren’t purely driven by your need to achieve certain estimates. Please note that simply tightening the YouTube ROI prior would make the model constrained to produce a more “reasonable” ROI which might inherently reduce the model’s ability to perfectly fit the historical data, which explains the lower R^2.
4. Use holdout sets to evaluate model performance on unseen data, use various visualizations and summaries from the Analyzer module to understand your results and evaluate if they make sense to you. You may use the following check-list of questions as a guide:
   • Business plausibility: Do your results align with your domain knowledge and expectations?
   • Consistency with other diagnostics: Do they make sense in conjunction with other metrics like contribution percentages, etc.
   • Robustness to sensitivity analysis: Are the estimates relatively stable to changes? Ensure that the results are not just due to priors’ influence.
5. Setting correct priors is one way to mitigate the low/ negative baseline problem but tuning them until the baseline is positive seems like an outcome-driven approach. Please make sure to consider the validity of the prior itself while tuning. Some more suggestions are given in the documentation on debugging steps to mitigate negative or low baseline.

I hope this answers your questions. Feel free to reach out for any further questions or suggestions regarding Meridian.

Google Meridian Support Team