Layers shared between different test runs when using zio.test.check

[jdsalchow]

When using zio.test.check my expectation was that provided layers would not be shared between test runs. In my actual case I use it for e2e testing with a database that I want to be be in a pre-defined state for the test. A silly reproduction is here:

import zio.*
import zio.test.*

object LayersShared extends ZIOSpecDefault {
  override def spec =
    test("foo") {
      check(Gen.int(0, 10)) { testInt =>
        for {
          currentValue <- ZIO.serviceWithZIO[ScopedRef[Int]](_.get) // "should" be 0 for every test
          _            <- ZIO.serviceWithZIO[ScopedRef[Int]](_.set(ZIO.succeed(testInt + currentValue)))
          newValue     <- ZIO.serviceWithZIO[ScopedRef[Int]](_.get)
        } yield assertTrue(newValue == testInt)
      }
    }.provideSome[Scope](ZLayer(ScopedRef.make(0)))
}

I got:

  Test failed after 2 iterations with input: 0
  Original input before shrinking was: 6
  ✗ 15 was not equal to 0
  newValue == testInt
  newValue = 15

[adamgfraser]

This is not correct. This is a single test and the layer is properly constructed once for each test. There is no sharing between tests. If you want to construct the layer for each random sample you can do so by providing the layer to the workflow in check.

import zio._
import zio.test._

object ExampleSpec extends ZIOSpecDefault {

  def spec = test("foo") {
    check(Gen.int(0, 10)) { testInt =>
      (for {
        currentValue <- ZIO.serviceWithZIO[ScopedRef[Int]](_.get) // "should" be 0 for every test
        _        <- ZIO.serviceWithZIO[ScopedRef[Int]](_.set(ZIO.succeed(testInt + currentValue)))
        newValue <- ZIO.serviceWithZIO[ScopedRef[Int]](_.get)
      } yield assertTrue(newValue == testInt)).provideSome[Scope](ZLayer(ScopedRef.make(0)))
    }
  }
}

[swoogles]

@jdsalchow To expand a bit more, you shouldn't consider the generator something that produces/runs many different test cases. Everything happens within one test case - "foo".

Since you were providing your dependency at the level of the test case, it was only "shared" across the different situations produced by the generator.

[jdsalchow]

Is there an obvious reason that checks should not "run in isolation"? My whole point here is that it's not the behaviour I would have expected :)

[987Nabil]

@jdsalchow Really? I mean you provide on the brace that surrounds all the checks. Seems quite okay to me. And if providing to all the checks one layer should be possible, where else should you put it? 🤔
And it is also consistent with how you would run an app. You provide once on the outside, and every effect that requires the layer gets the same layer.

[adamgfraser]

The semantics of provide is that it provides a separate copy of the layer to each test. A sample in a property based test is not a test. The test is the thing labeled foo that gets reported in your test output. If a property based fails after 42 iterations the result is that there is 1 test failure, not 42 test failures and 1 test success.

I understand your desired behavior is to provide a separate copy of the layer for each sample, which is easily accomplished by providing the layer inside of check and follows normal scoping rules. However, other users do not desire this . For example, their samples do not interfere with each other or they do necessary cleanup and they do not want to make their property based tests slower by building the layer for each sample. Your change would make that impossible.

Hope this helps!