Low baseline, ROI Consistency Check failing

[superli3]

Reference Discord thread: https://discord.com/channels/1397974595952185365/1446607859235164160

I built a Meridian MMM for one of our clients (in retail). The brand is well established, but there were some concerns about the baseline being too low (57%) - the client's existing MMM provider has shown a baseline of 83%.

1. Is there a way to methodically determine if one is more correct than the other.

Upon further investigation, using the built-in Post-modeling quality checks, everything passes except for the following:
ROIConsistency Check:
Status: REVIEW
Recommendation: We've detected an unusually low ROI estimate (for channels Channel_A, Channel_B, Channel_C) where the posterior point estimate falls into the extreme tail of your custom prior. Please review this result to determine if it is reasonable within your business context.

2. Are the ROI priors being set too high, or too low?

---

debugging info

1. Data Volume & Periodicity: How much data do you have? What type of periodicity (daily, weekly, monthly), etc.

2.5 years of data at a weekly level

2. Data granularity:Are you running a geo or national level model? How many geos?

National level model

3. Number/types of channels: What is the # of channels? Can you give a breakdown? Any non media treatments or organic media?

12 Channels:
'affiliates',
'digital_flyers',
'dooh',
'magazine',
'online_audio',
'online_display',
'online_video',
'paid_search',
'paid_social',
'print_flyer',
'radio',
'tv'

I considered using 2 non-media treatments: Promotion Redemptions and Email Sends, however, they did not help the model
No organic media

4. What is the outcome/KPI you are measuring?: What are your outcome/KPI that you are measuring? Revenue or non revenue

Revenue

5. Data quality : Can you describe the process in which you constructed the channels? Do you have high confidence in the data quality?

The channels were constructed and validated by the client. There is high confidence in the data quality

6. Detailed Priors Usage: Are you using priors? What priors they're using (parameterization, custom vs. default)

Custom priors for ROI

7. Controls: Are you using controls? If so, which controls, how many?

Selecting from a pool of 15 controls available. They are broadly categorised under the following:
Housing resales
Mobile app events
Emails sent
Promotions
Commodity prices
Holidays
Special economic events like Black Friday
Interest Rates
Weather
GQV
8) Model quality report card (code snippet here)

I'm attaching this in a followup

9. Priors: If using custom priors, how informative are they?

Strongly informative

10. Adstock & saturation choices?

Adstock - Binomial
Media Effect Dist - Log Normal

11. Sparsity: What percentage of the data contains zeros? (Part of the EDA report coming in Jan).

<1.6%

12. Mitigations you have tried: What other mitigations have you tried to resolve the issue so far?

Not much other than tweaking the priors. Want to understand the cause.

13. More detail is better (i.e. on multicollinearity): Do you observe any specific combination of channels which demonstrate multicollinearity? Are there some channels which are more problematic than others

We conducted VIF checks, where all columns have a score between 1-5
Model Quality Report:

========================================
Model Quality Checks

Overall Status: PASS
Summary: Passed with reviews: Review is needed.

Check Results:

Convergence Check:
Status: PASS
Recommendation: The model has likely converged, as all parameters have R-hat values < 1.2.

Baseline Check:
Status: PASS
Recommendation: The posterior probability that the baseline is negative is 0.00. We recommend visually inspecting the baseline time series in the Model Fit charts to confirm this.

BayesianPPP Check:
Status: PASS
Recommendation: The Bayesian posterior predictive p-value is 1.00. The observed total outcome is consistent with the model's posterior predictive distribution.

GoodnessOfFit Check:
Status: PASS
Recommendation: R-squared = 0.9753, MAPE = 0.0272, and wMAPE = 0.0259. These goodness-of-fit metrics are intended for guidance and relative comparison.

PriorPosteriorShift Check:
Status: PASS
Recommendation: The model has successfully learned from the data. This is a positive sign that your data was informative.

ROIConsistency Check:
Status: REVIEW
Recommendation: We've detected an unusually low ROI estimate (for channels affiliates, digital_flyers, print_flyer, dooh, magazine, online_audio, online_display, online_video, paid_search, paid_social, radio, tv) where the posterior point estimate falls into the extreme tail of your custom prior. Please review this result to determine if it is reasonable within your business context.

[cpulavarthi]

Hello @superli3,

Thank you for contacting us and sharing detailed information regarding your data and modeling choices!

Here is a summary of the information you provided and my comments:

1. You are running a national model with 2.5 years of weekly data - We typically recommend using a minimum of two years' worth of weekly data for geo-level models and three years' data for national-level models, however in your case the data seems to be sufficient because the model is converging to a separate posterior distribution despite strongly informative ROI priors which most likely don’t agree with the data.
2. 12 Media Channels, 2 Non-Media treatments, 0 Organic Media - Though the number of channels isn’t directly a concern, given the feedback from the model quality report regarding some channels having very low attributions towards the ROI, you may want to consider merging some of the low performing channels together. Do consider business relevance before proceeding with this.
3. Confidence in data quality is high. The data sparsity is <1.6% zeros. - The data being used seems to follow Meridian’s requirements.
4. Control variables are selected from a pool of 15 variables. - While you shared the categories of the controls, you didn’t mention how many of them you are actually using while modeling. Not including all important confounding variables as controls is one of the main causes of low-baseline attribution. Please ensure that you are including all appropriate confounders while modeling.
5. Revenue model with strongly informative ROI priors. - You are using strongly informative ROI priors but the posterior estimate is falling on the extreme tail of your custom prior. This suggests that the information from data is contradicting the choice of priors. You may want to manually evaluate the relevance of your custom priors and see if they actually reflect business expectations.
6. VIF score is between 1-5 for all channels implying that there is no multicollinearity issue.
7. You have experimented with tweaking the priors to address the issue of low baseline attribution. - This is again one of the recommended steps for debugging a low baseline.

To answer your question regarding how to determine which model better captures the ground truth, I would suggest using out-of-sample prediction metrics to compare the models. You may compare metrics such as R-squared, MAPE, wMAPE, and any other relevant predictive accuracy metrics. It is also worth evaluating the r-hat and Bayesian PPP values of the other model if applicable (i.e., if the other model also uses Bayesian Causal Inference like Meridian).

Regarding the ROI priors, the model quality report combined with the information you shared, suggest that you are using strong ROI priors at the higher end of the spectrum and still the point estimate of the converged posterior is much lower and falls on the extreme tail of the custom prior. While I cannot comment on the accuracy of the chosen priors because this depends on your specific use case, it is worth evaluating business relevance of the chosen custom prior distributions.

While I can’t tell if the baseline estimate of 57% is low (as this depends on your specific business use case), if you believe that this value is low as per your expectations, you may check our documentation on how to mitigate negative or low baseline which has detailed guidance on debugging this issue. The baseline is computed by subtracting individual channel contributions from 100%. The information you provided indicates that the chosen custom priors have much higher point estimates than the posteriors model converged to. Which means you are indirectly nudging the model towards a low baseline (as the ROI priors are high). The model warning ("Unusually low ROI") means the data is trying to pull the ROI down (and the baseline up), but the "Strong" priors are preventing it from moving far enough. This is happening for all 12 of your channels which further suggests that you need to evaluate the relevance of your priors.

I hope this answers all your questions. Feel free to reach out if you have any further queries regarding this.

Google Meridian Support Team