Budget Allocation: budget constraint not hold

[JiaMeihong]

Project Robyn

As suggested by @kyletgoldberg, I create this issue below, which is likely to be caused by some bug of allocator.

Describe issue

I’ve tried both scenarios of history-based optimization and future optimization with given budget.
My constraint over channels is (0.8,1.5) for each channel.

I found

1. in both of these cases, the optmResponseUnitTotalLift is negative. Meaning the uplift is negative after optimization.
2. in the future optimization case, I believe expSpendUnitTotal >= optmSpendUnitTotal should always hold, but the result shows differently.

Provide dummy data & model configuration

I've check the model result, it has a R2 above 0.75, and no coefficients of our interest is zero.
So I've detected no abnormality in the model.

Environment & Robyn version

R version (R --version): 4.1.2
Robyn version: latest version

[JiaMeihong]

@kyletgoldberg
I think I found some clues. It turns out the order of channel names in the model, and order of channel names in allocator are different. And I think in budge allocator the spend share is matching to wrong channels names.

I've checked input of my model, and found nothing wrong there. paid_media_spends, paid_media_vars and hyperparamters assignment are in the same channel name order.

In the allocator, there's no function to assign channel names externally, so I think it should be the problem of allocator code iteself. After examining the code in allocator, I found the media_vars and media_spends have been reordered in channel names, but the model coefficients seem to be ordered in another way here. I guess this inconsistency in the order caused the weird allocation result.

(All coefficient for channels are positive in my model.)
Below I attach an example of budget allocation visualization.

[laresbernardo]

Hi @JiaMeihong I've been re-coding some important stuff in robyn_allocator() function and I think this issue might have been already solved. Could you please try installing Robyn from "allocation_range" branch and running your pipeline again to check if that's the case?
To install that version: remotes::install_github('facebookexperimental/Robyn/R@allocation_range')
Appreciate your tests and following comments. Once you confirm if this has been solved we can land it to "main" branch.

[laresbernardo]

X-ref: Possibly connected with the same issue

[JiaMeihong]

@laresbernardo
Hi thanks for your reply! I've tested using the branch you shared.
Part of problem is solved: Now the spend share are matching to the right channel names.
But two problems persist:

1. in the max_history case, after optimization, we increase spend share on channel 1, but the mean response of channel 1 decreased. This visualization is still confusing as some points are not lying on the response curve.
2. in the max future expense case, still the optimizationUnitTotal > expenseUnitTotal. I wonder is it because my expenseUnitTotal is much smaller than Initial Spend unit Total?

[laresbernardo]

Hi @JiaMeihong
0. Glad to know we at list fixed that first issue :)

1. I can't seem to replicate those cases where the values are plotted outside the curves. I even set channel_constr_low = runif(5, 0, 1) and channel_constr_up = runif(5, 1, 5) and ran 10 random simulations and none plotted values outside the curve. I've also re-checked the code if we are sorting the variables and constraints before optimizing. I've slightly changed some code in that branch, maybe try updating the branch version again and retry, please. Also, check the print(AllocatorCollect) results to numerically check results. Let me know how can I replicate your case if it's still occurring.
2. What do you mean? optmSpendUnitTotal values and expSpendUnitTotal (exp for expected, not expense) right? So the first one mentioned is the sum of the optimized results (optimal spend) and the second the sum of the expected spend for the period specified. They both depend on your expected_spend and expected_spend_days inputs. Do they make sense?

[JiaMeihong]

Hi @laresbernardo
Thanks for your reply!

1. I've tried with new branch version for case of max_history, but still in my data, the dot is outside the line. In addition, I've tried with test data provided by Robyn, and for test data, it seems to work fine, and the plot of budget allocator looks nice. Would it be possible I share with you my data, and you test with it?
2. Yes, I give the expected spend and expected inputs, and expect the optimized budget <= total budget give, and in my understanding, expSpendUnitTotal >= optmSpendUnitTotal should always hold, but the result shows differently. Is it a bug or it's designed to be so?

[laresbernardo]

Yes, I think sharing your data will def help me debug this; I'll do my best to understand what's happening there. Share the data file and script you are running so I can replicate it exactly, please.
On the 2nd point, please read this reply provided to another user with same question.

[laresbernardo]

Feel free to re-open if the error or issue persists @JiaMeihong