-
Notifications
You must be signed in to change notification settings - Fork 327
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Robyn_refresh error when collection results #307
Comments
Hey, this is very strange. I can't think of any possible cause out of the box. We'll need a reproducible dataset to replicate this error, including a dataset (true value masked) and your demo.R file for model specification. If you don't want to share publicly, please send it via email @laresbernardo and me (bernardolares@fb.com, gufeng@fb.com) |
what should i mask the data? By true value you mean features and target? |
Users usually don't share real data with us but somehow randomised dataset. |
ok, sent it to you email |
Hi @ohad-monday |
@laresbernardo thank you, im checking right now. |
hey, yes there're actually major improvement in optimisation, see here. Please also check out the new demo.R guide that introduces new functionalities and workflows. You should be also seeing new convergence message after the model runs. How many iterations are you running? Now for the simulated dataset it converges at 1.5-2k iterations. |
Can you share this plot with us: OutputCollect$OutputModels$convergence$moo_distrb_plot |
@gufengzhou I don't see a this plot here: |
@ohad-monday can you please check on |
@laresbernardo sure! (it's from the robyn_run()) |
Thanks for sharing. as you see, model hasn't converged, if you compare this with the example plot here. Esp. NRMSE hasn't moved from the right side and it reflects exactly the bad fitting plot you showed before. I recommend you to run higher iterations to try out. You can do for example 5k with 1 trial first to see if 5k converges. The reason for this change of convergence speed is that we've added lambda as extra hyperparameter to enable its automatic optimum selection. We'll fine-tune it over time to improve convergence speed. |
FYI I've just committed a fix that should accelerate convergence. You should be seeing the "hills" in the plot moving left earlier compared to before. Let us know if it works. |
@gufengzhou Found the reason you weren't able to reproduce the error i got. the problem is with a model with a 0 coefficient that also has calibration data for this channel. might be somehow related to the mape lift calculation for the nevegrad optimization? |
Uhh, bummer. Would you be able to send me (laresbernardo @gmail.com) a CSV with anonymized data and the .R file you're using to run Robyn so I can replicate exactly your issue? It'd be really useful to debug this error in case you're not able to do it yourself and make a pull request. I'd be happy to check given that Gufeng will be on leave for some months. |
@laresbernardo thanks! i shared an updated params.R in email with calibration data. the refresh supposed to end with an error. than if you will exclude the f6 feature from the calibration data, it will work. |
Hi @laresbernardo @kyletgoldberg Correct me if I'm wrong, my understanding is that previous version 3.5.0 choose best lambda of Ridge regression using 10 fold cv on window period. What has changed in the method to tune lambda in 3.6.0 version? and why do you decide to make the change? Thanks very much for your help! I'm really learning a lot from your product development and discussion here! |
@JiaMeihong could you check to see if any of the calibration data you are using is also corresponding to a channel that was assigned a 0 coefficient? Seems like that was causing the issue for @ohad-monday so that would be a good place to start. |
@kyletgoldberg Hi, thanks for your reply! I've checked the coeffecient in "pareto_aggregated.csv". The channels of interest don't have a 0 coefficient. Though after iterations it seems that results are still far from convergence, all channels in my model result have positive coef. |
Thanks for taking a look - how many different calibration inputs are you using? this is a difficult one to replicate on our end since we don't have data to recreate, but if you don't have too many calibrations, would it be possible to take them out one at a time and see if it is one particular calibration input causing an issue? I suspect some interaction between the calibration and the lambda hyperparameter optimization is causing the issue |
@kyletgoldberg My calibration data is by month, but my input data is by week. Does that possibly impact the result? Could you briefly explain what has changed in this Robyn version 3.6.0 about calibration? Cus when using previous version with calibration, the result worked fine. I think by understanding the logic of change may help me to detect the bug further. Thanks! |
@JiaMeihong That shouldn't be an issue - nothing changed in 3.6 with respect to calibration, but in 3.6 we added lambda hyperparameter to the nevergrad optimization rather than using the CV method. This should lead to better results by allowing more flexibility in learning the hyperparameters, but also seems to be leading to some convergence issues when paired with calibration data at times. It seems like what is essentially happening here is that we are running into cases where the calibration data is so at odds with how the model wants to fit that it is never going to converge, which is a difficult problem to solve. I saw you had edited a comment that had initially said that removing one of the calibrations with the lowest lift had got the model to work again - was that the case? It could be worth digging into how exactly the test was set up vs. how the data is collected for the channel - i.e. does the test encompass all of the spend you are measuring? etc. to ensure that they are as aligned as possible. Would be great to understand any info you have on this part to keep working to figuring this out. Thanks for your patience! |
@kyletgoldberg Thanks very much for your detailed explanation! For the lambda part, I understand the version difference now. What I found for now is, yes there's a converging issue when I try to add calibration, even just one channel of one testing period won't converge in my case; and I also played with another model without calibration, but a lot more media channels. The latter case also haven't converge after many iterations. I got into a headache of dying kernel when iterations are too many, and thus feel that simply increasing iterations in hope of achieving convergence may not be a good idea. Would appreciate it very much if there're other workarounds to get convergence. |
@JiaMeihong do you mind sharing what you get when you run the following code after the non-convergent models in both calibration and non-calibration cases?
How many media channels are you including in that second case? We generally recommend to have at least 10 observations per independent variable so that may also be adding some difficulty. Thanks again for your patience |
@kyletgoldberg
I have added 7 channels in total, and each have more one year weekly data. Though I specify it to just learn the most recent one year in the window function. |
@JiaMeihong could you also run that without the calibration and share how it looks? |
@JiaMeihong can you give us a bit more context on your calibration inputs?
|
@laresbernardo
Yes
there's fluctuation in spending, but I tried to transform the AB test result by multiplying it by the ratio of average spending/ actual spending.
Most of the results are from AB test, so it should be trustable. Even without calibration, in another model where I try to include more channels (7 in total) to the model, it fails to converge as well. |
If we run a model to convergence, does the Pareto Front determination exclude solutions which are pre-convergence? |
Hi @extrospective |
We are running Robyn 3.6.2 We had been using with calibration successfully, but we changed our target variable and are now encountering exactly the error mentioned here.
If there is any wisdom on what causes that we would appreciate it, as we're on a tight deadline to turn around this model. At first I thought it was that we had too few iterations, so we increased to 1 trial x 6000 iterations. Here is stdout. As you can see, we did not run the 20,000 total iterations suggested, partly because this was already 6 hours and we just wanted to see if the code ran before scaling up.
|
@extrospective could you share your session info please? i've seen this error once before when a user was running on GCP, are you running locally or on a cloud based solution? |
we are running in databricks |
Databricks runtime 10.2 Here is the r version info:
We pulled from the git repo 3.6.2 roughly 2 weeks ago. checking now for details |
r 4.1.2 is the maximum available in databricks runtimes at this time. |
@extrospective would it be possible to try running locally to see if you get the same error? i don't think you need to run as many trials we should be able to see whether it works or not with fewer. I suspect this may be an issue with Robyn having some issues with cloud platforms so that would be a good one to rule out if we can |
We will try some further tests. not sure why "cloud platforms" should be a source of error, but as cloud platforms may have different versions of libraries, I think a library comparison between what works for you and what does not work for us is helpful. Our next runs include:
|
good point. |
I am not sure the April 12 commit is in our copy. The prior commits should be in our copy of Robyn. |
This is the sessionInfo() for our run.
|
One change which was made on our end from a prior run was the addition of SparkR, so we will also test if that library is the source of an issue. --> We were wrong about this, sparkR has been in the session all along (see next comment). |
Databricks has sparkR in an empty notebook by default. We showed that if we turn calibration off, we do not encounter the error mentioned above. [From this research we have learned about library doRNG and rngtools which can be used to trace through errors (for those who encounter this error in the future and want to investigate more rapidly.)] |
I suspect there is error in liftAbs in the calibration data -> found some NAs I assume robyn_run() would have noticed this earlier then at the final pareto generation step; as it's input to objective function. Can checks be added for NA? But first, I'll verify this is the source of the problem. |
We have confirmed this error for us as a data issue. Based on this, I might recommend:
This assertion would then avoid an unusual and weird error. And then I think this ticket might be closed with a hypothesis that this same issue caused the other errors reported with this same symptom. |
Thanks for confirming @extrospective - we will add a check for that and then close the issue out with that fix. |
Thanks for the feedback and for checking the source of this issue @extrospective You're recommendation has been implemented! Feel free to close this ticket if you consider it's been fixed |
I do not have the close ticket available to me. I recommend closing. |
Project Robyn
Describe issue
issue - when running robyn_refresh with more than 22 incremental days i get this error below
error message:
Provide dummy data & model configuration
what else is needed?
Environment & Robyn version
R version : R version 4.0.3 (2020-10-10)
Robyn version: Robyn_3.4.8
The text was updated successfully, but these errors were encountered: