-
Notifications
You must be signed in to change notification settings - Fork 2.3k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
splitTestGroup ads are shown when AdvertiserSplitTestStudy is not defined in seed #15164
Comments
This is intended behaviour as we were optimising for pacing hence allowing the ad if a client didn't sign up. @jsecretan @tmancey what do you think? |
@moritzhaller Agreed. |
Ok I actually think the behavior you were mentioning is the right one @btlechowski, if a client doesn't have the group defined, it shouldn't participate in the campaign. This should be rare in practice, so I'll put a low priority on this one. |
Verification passed on
Verified test plan from the description
Verified splitTestGroup ads are shown when group is defined in the seed: |
Follow up to #14613
This could skew the a/b testing results. For example when seed becomes available, but some split testing ads have already been shown.
The splitTestGroup ads should only be shown when the Group is defined in
AdvertiserSplitTestStudy
in the seed variation.Steps to Reproduce
Notification ad - AdvertiserSplitTestStudy is not defined in seed
from https://github.com/brave/internal/issues/762Actual result:
splitTestGroup ad is shown.
Expected result:
splitTestGroup ads are filtered out
Reproduces how often:
Easily reproduced
Brave version (brave://version info)
cc @moritzhaller @tmancey @jsecretan @brave/legacy_qa
The text was updated successfully, but these errors were encountered: