Split Dashboard showing negative NON-FINISHED value

[chrisatboilerjuice]

Hello!

We have encountered a bit of strange behavior on the split dashboard where we can get a negative value for the non-finished. Please find the attached image which shows a recent test we were testing.

We aren't quite sure how to reproduce this but I do know our QA team has been using Selenium to test the Split test, so they don't have to manually go through the test 100 times and make sure the split test is set up correctly.

I setup the test for the user calling this bit of code:
ab_test(:Service_fee_test) do |_alternative, meta| result = meta['value'] end

Just before calling the ab_finished i do call ab_test(:Service_fee_test) again (as I need the value again near the end of the journey), which should give me the same result.

The ab_finished(:Service_fee_test, reset: true) tag is called on the post to our complete page, and once that is done it does a GET to the SHOW action on the complete page, so it can't be called again.

So how on earth is this happening???

Any advice would be most appreciated.

Thanks,
Chris

[SxDx]

Hey, we are seeing the same issue with 4.0.2.
It seems like there was fix quite a while ago in #408 and I'm pretty sure I did not see this behaviour at all pre 4.x.

We are using the following configuration:

Split.configure do |config|
  config.enabled = true
  config.winning_alternative_recalculation_interval = 3600
  config.allow_multiple_experiments = true
  config.store_override = true
end

Split.configuration.experiments = {
  pixel_info: {
    alternatives: %w[visible hidden],
    resettable: false,
    combined_experiments: %w[pixel_info_sd_download pixel_info_hd_cta pixel_info_hd_download],
  }
}

ab_finished(:pixel_info_hd_download, reset: false)