Varying population

[shaco-git]

Hello,

In our business context the geo population varies significantly over time because of expansions and shutdowns of certain cities/hubs. Meridian only accepts one population average for all time periods. In this way values before the expansion are scaled down too heavily and values after the expansion are scaled down too little, right ?
Is there any way around it, like scaling my media and KPI data manually beforehand ?

[cpulavarthi]

Hello @shaco-git,

Thank you for contacting us!

Meridian's underlying model (GBHMMM) assumes that geographies are stable and that their populations are constant throughout the modeling period. Though Meridian accepts population data at geo and time granularity, i.e., population values can be input for each geo and each time period, the data loader internally averages the population values over all times for each geo before using them for scaling. This is based on the underlying model assumptions. Population is an objective geo-specific feature, and should not be subject to the business expansions/ shutdowns. Meridian estimates the average ROI of a given time period. So any expansions/ shutdowns are already considered in the final results. You may check the paper on Geo-level Bayesian Hierarchical MMM to better understand the model’s framework.

Since in your situation the population values change significantly over time due to expansions and shutdowns, this averaging would lead to improper scaling and may influence model performance as you suspected. To resolve this, I would recommend defining geos which are large enough so that the population stays relatively stable over time. This would be the best approach and will ensure that the model’s assumptions aren’t violated.

In case maintaining stable population sizes for your defined geos isn’t practically feasible for your use case, you may consider scaling the media and KPI data manually and attempt modeling. You would be defining per capita variables instead of spend, impressions, sales, etc., for modeling and you may set the population value to be 1 in the input data so that the population scaling doesn’t end up happening twice. Please note that while this pre-scaling solves the mathematical scaling issue, the model's internal logic still assumes a stable population pool. Rapid population changes are a dynamic the model isn't explicitly designed to handle, so the stability of the model's results may not be guaranteed.

Do reach out if you have any further questions or suggestions for us!

Google Meridian Support Team

[devin30000]

Thank you for the quick respone,

I have tried the approach with pre scaling and setting the population to 1. I have tried 2 things.
a) set kpi_type to non_revenue and add a revenue_per_kpi. I first tried it without the manual pre scaling and the results were okay. But then the contribution of the Media channels converged towards 0, while non-media treatments had all the contribution. Is there something else I have to consider when doing this approach? I reconstructed the scaling and adstock and hill transformation and it looked fine, but something after that seems to be off.
b) i set the kpi to revenue and this time the results looked better, but instead of having revenues of thousands i was now into the millions.

Can you tell me what the problems are here?

in case this approach simply doesnt work, I also thought of 2 other ideas:
1.) scale KPI and media inputs by a factor relative to the the reference population. e.g. a population increases by 50% then I would scale down the KPI and media after that by 1/(1+0.5) = 2/3.
2.) leave KPI and media as it is, but include this factor as a control/non-media variable.

I would appreciate if you could help me solve this problem!

[cpulavarthi]

Hello @devin30000,

Thank you for contacting us! I will try to answer each of your questions in the same order you posed them:

1. You mentioned trying two different models, one with a non-revenue KPI and Revenue per KPI conversion factor and the other using Revenue directly as the target KPI. The differences in the contributions observed aren't expected to be as drastic as you mentioned because Meridian estimates the causal effect of media on revenue in both these cases. You can find a detailed discussion of the differences between these two choices in Discussion #1020. It is possible that the difference you observed could be because of different variable scaling approaches, please ensure you use per-capita variables for the Media input as well if you are pre-scaling the target KPI by population. I would recommend that you ensure the pre-scaling used in both cases is identical and see if the differences in the results still persist.

2. While my previous response suggested using pre-scaling to resolve mathematical scaling issue in the case of varying population sizes as a last resort (because Geo-level modeling assumes constant population during the modeling period), we strongly recommend not to perform any other media transformations manually before loading the data. Performing Adstock and Hill transformations manually may result in the same transformation happening twice because Meridian performs these transformations internally before modeling.

3. Regarding 2 ideas of using a scaling factor either to scale the target KPI or as an additional variable in the model, I would recommend that you only scale the target KPI in your proposed way if it makes business sense, i.e., if there is a logic to scaling in that particular way because typical geo-level modeling only scales the variables with the population (assuming population is constant) and not with population increase or any other custom scaling factor. The scaling with population is done because per-capita values make more sense for comparing impact rather than direct values, i.e., a geo with lesser population may have lower revenue/ sales - this is accounted for while population scaling.

   You may choose to include your proposed scaling factor as a control variable. The scaling factor should be included as a non-media treatment only if your population expansions are intervenable, else it may be added as a control.

We strongly recommend defining your modeling approach (like scaling choices and which control variables to include) based on business logic before you run the model. Iteratively changing your pre-processing steps or adding/removing control variables based on how the model's results "look" can lead to a form of manual overfitting which may compromise its generalizability.

Hope this answers all your questions. Do reach out if you have any further queries regarding this.

Google Meridian Support Team