Baseline contribution - Investigating Differences

[ani-rag]

We recently migrated from Lightweight MMM to the Meridian package and have built separate geo-level models for different business regions (historical data - using last 3 yrs, weekly data). When analyzing baseline contribution over time, we've observed an interesting divergence: in one region, the baseline has shown a steady upward trend over the past 12 months, whereas in another region, it has remained relatively flat.

We're hypothesizing that this difference could stem from changes in underlying market dynamics, but I'm specifically interested in what insights we can glean directly from the Meridian model outputs to further investigate this.

Given Meridian’s support for knots and time-varying coefficients, what diagnostics or outputs (e.g., posterior summaries, coefficient trends, contribution decompositions) would be most helpful to understand why the baseline has generally grown in one region but not the other?

Would appreciate any guidance or best practices on how to explore this using the model artifacts? Thanks!

[cpulavarthi]

Hello @ani-rag,

Thank you for contacting us!

Meridian doesn’t generate a separate baseline for each geo in a geo-level model. Are you comparing the baselines from different national-level models built for each geo? And if so, are you using the same control variables in the different models? 3 years of weekly data is ideally sufficient for a Meridian model.

The baseline in Meridian is the expected outcome under the counterfactual scenario where all treatment variables are set to their baseline values. For paid and organic media, the baseline values are zero. For non-media treatment variables, the baseline value can be set to the observed minimum value of the variable (default), the maximum, or a user-provided float. You may check our documentation on assessing the baseline for debugging steps related to incorrect baseline estimates.

The increase in baseline over time indicates that the media contribution towards the target KPI has decreased over time. This could be due to lower media execution, control variables having a greater impact, or even an underlying growth in demand in that region, independent of marketing activities. The model attempts to capture non-attributable trends in the baseline. Since you are comparing separate models built for each geographic region, I would recommend you perform an exploratory analysis on the input data, analyzing the trends in the media execution, organic media and non-media treatments for each geo. Also check how your controls are changing over time and in different regions if they are geo-specific.

To analyze the outputs from the model, you can use Analyzer class methods for both model diagnostics (rhat_summary, predictive_accuracy, etc) and result analysis (baseline_summary_metrics, expected_vs_actual_data, incremental_outcome, etc). Be sure to set the aggregate_times attribute to False in order to avoid aggregation over the time dimension. You may also explore visualizations such as plot_channel_contribution_area_chart, plot_adstock_decay, plot_hill_curves, and plot_response_curves to gain more insights into the media contributions and effects. The above mentioned suggestions are not an exhaustive list, and you would need to perform appropriate analysis based on what you think might be causing the deviations in baseline attribution.

Knots in Meridian are set during the modeling phase. The default value for knots is 1 for a national model and n_times for a geo-level model. Adjusting the knots parameter could be a potential debugging step if you see unreasonable variations in the media effect estimates, especially based on your initial exploratory analysis. You may also choose to specify knot locations and increase the number of knots during the periods with high variations in trends.

Feel free to reach out if you have any further questions regarding this.

Google Meridian Support Team

[ani-rag]

Hi @cpulavarthi - Thanks for the details. I’ll dig deeper into the input data.

I’m currently comparing baselines from two different models(both are geo level). Both use the same control variables. I'm interested in understanding what additional diagnostics I can run to analyze the learned model parameters. I’ll explore the methods in the Analyzer class as a next step.