Budget Allocator issue

[prashanthin]

Project Robyn

Hi team, I am doing a case study using Robyn package.

Describe issue

Facing issue in Budget allocator allocation - both initial and optimized budget allocation are same.
and also why my optimised response is greater than initial response? Am i missing anything while allocating budget of 1000000

Provide dummy data & model configuration

code -
AllocatorCollect <- robyn_allocator(
InputCollect = InputCollect
, OutputCollect = OutputCollect
, select_model = select_model
, scenario = "max_response_expected_spend"
, channel_constr_low = c(0.7, 0.7, 0.7,0.7)
, channel_constr_up = c(1.5, 1.5, 1.5, 1.5)
, expected_spend = 1000000 # Total spend to be simulated
, expected_spend_days = 7 # Duration of expected_spend in days
)

Allocated values -

Model ID: 4_39_5
Total Spend Increase: 6400% (440K)
Total Response Increase (Optimized): 60.9%

Allocation Summary:

• DISPLAY_ad_spend:
  Spend Share: Initial (avg) = 4.29% -> Optimized = 4.29%
  Mean Response (per time unit): 52,112 -> Optimized = 75,148
  Response: 9.436K -> Optimized = 14.15K (Delta = 50%)

• PAIDSEARCH_ad_spend:
  Spend Share: Initial (avg) = 62.8% -> Optimized = 62.8%
  Mean Response (per time unit): 348,889 -> Optimized = 434,106
  Response: 138K -> Optimized = 207K (Delta = 50%)

• PAIDSOCIAL_ad_spend:
  Spend Share: Initial (avg) = 17.9% -> Optimized = 17.9%
  Mean Response (per time unit): 5,930 -> Optimized = 18,568
  Response: 39.28K -> Optimized = 58.92K (Delta = 50%)

• PARTNERSHIPMARKETING_ad_spend:
  Spend Share: Initial (avg) = 15% -> Optimized = 15%
  Mean Response (per time unit): 272,771 -> Optimized = 565,608
  Response: 33.02K -> Optimized = 49.52K (Delta = 50%)

Environment & Robyn version

R version (R latest version)
Also Latest Robyn version

Thanks in advance.

[laresbernardo]

Hi @prashanthin thanks for reporting this. This is def a bug we have in this late version. Can you please check this comment and try this out? There are several users that have reported this and we are working on a solution in a separate branch before we merge to master. Please, let me know if this particular issue has been fixed for you on the "allocation_range" branch. Thanks!

[prashanthin]

Hi Bernardo,

Thanks for the reply. I tried running code updating Robyn running below - remotes::install_github('facebookexperimental/Robyn/R@allocation_range')

But Still I am getting same error. when I ran the case example already installed with Robyn its showing the changed values -

But for my case its showing same. Neither optimised and intial budget allocation changing, also opimised response is higher for all the channels.

Also I have one question when I compared 2 graphs - why for me total spend increases is 50% while in case example its zero? Can you guide me what can I do next.

Thanks.
Prashant

[laresbernardo]

Hey @prashanthin it seems you're not loading the updated "allocation_range" branch version. I know this because we changed some of the formattings on those plots. Please, try again by updating the Robyn version as you did, resetting your R session, and retrying. Let me know

[prashanthin]

Hey Bernardo, Thanks for the reply. I have used the updated Robyn package ( from allocation branch ) as you said, but it has many problems/bugs. Some of which are mentioned below -

1. I have total four spend channels for my case - and here are the historical spend based on raw data -

problem - but when I ran the robyn_allocator with max_historical_response - I am getting below data in my file -

Its clear that historical spend are not aligned to channels , as figures are mismatched, Can you please fix/provide solution to this issue ?

2. In above graph- highlighted in red - I am having Paidsearch ad spend apprearing twice, this might be because my Display_ad_spend variable is having zero ROI, possibly this may be also a bug.

3. For Budget allocator - max_response_expected_spend - I am trying below version where I am allocating 500000 as my expected spend -

AllocatorCollect <- robyn_allocator(
  InputCollect = InputCollect
  , OutputCollect = OutputCollect
  , select_model = select_model
  , scenario = "max_response_expected_spend"
  , channel_constr_low = c(0.8, 0.8, 0.8,0.8)
  , channel_constr_up = c(1.5, 1.5, 1.5, 1.5)
  , expected_spend = 500000 # Total spend to be simulated
  , expected_spend_days = 7 # Duration of expected_spend in days
)

Problem - I am getting negative spend increase and hence optimum budget allocation is not changing -

bugs - 1) why my expspend unit total is coming as 71428, instead of spending 500000 as total spend for all channels.
2) How is init unit spend units (210321) are calculated?
3) Because of this my optimum spend is -20% , why my spend is decreasing inspite of giving high input figure ?

Below is the budget allocation graph for exp spend as 500000

Can you please help me in resolving these issues?
Thanks,

Prashant

[laresbernardo]

Would you be able to share with me your dataset and script so it can help me debug easier these bugs you mention? Thanks a lot for taking the time to double-check this branch's changes. Please add #349 in the email so I know you are referring to this issue.

[laresbernardo]

Ok. Received. I'll start by sharing some observations @prashanthin

• I ran the script as-is (500 iterations and 5 trials) and none of the 2 objectives converged. I'd increase the number of iterations here
• I got 12 clusters and selected a model with a channel with ROI = 0: 3_31_12 (for some reason 2_50_8 was not available)

That said, I share the results I was able to replicate.

1. The channels seemed to have the same issue as you had when one of the channels had ROI = 0. I've fixed it in this "allocation_range" branch now. You'll now get only 3 rows instead of 4 as DISPLAY_ad_spend will be excluded correctly.
2. For the "max_response_expected_spend" scenario:

• We calculate expSpendUnitTotal = expected_spend / (expected_spend_days / InputCollect$dayInterval). So 500000 / (7 / 1), given that your data granularity is provided daily.
• initSpendUnitTotal is calculated summing the means of non-zero spend values for each channel: sum(sum(x) / sum(x > 0))
• Regarding the negative result, this is a perfect example of economies of scale (law of increasing returns): with decreasing budget on S-curve you can move out of the proportional/linear part into the inefficient section (lower part of S). If you spend too little you never reach economies of scale, if you spend too much you reach diminishing returns. I ran a couple of scenarios and they check out: if you increase your budget x10, then you'll get (positive results) with an additional 50% spend and 89% response increases. On the other hand, if you open the upper and lower bounds to (0.01, 5) with the budget proposed, you'll get a 67% increase in response with -66% spend.

Please, update with the most recent version of "allocation_range" and re-run everything to check out the fix. Also, please let us know if it makes sense so we can merge with "main" branch and deploy this to other users as well. Thanks!

[HasanShaukat]

Just an opinion and a question, if the total spend increases and the spend share is the same, isn't it natural for the total response to increase?

[kyletgoldberg]

Hey @HasanShaukat - in most cases yes, unless the response curve were horizontal. do you mind elaborating on where this may be impacting this issue?

[HasanShaukat]

I believe the origin of this issue was that the percentages of budget allocated are the same for initial and optimal, and that we're seeing higher budget for the optimal setting. But the text above the graph says that the budget volume set is 50% higher, and I believe that should account for a higher volume in response.

[laresbernardo]

Feel free to re-open if the error or issue persists @HasanShaukat @prashanthin