Conditional QuickConfiguration / Extensible example flags

[darronschall]

I'd like to request a (better) way to create a configuration that applies to only certain examples.

I have some examples that require a Core Data stack to be setup. I have some examples that don't. I have some examples that require not just a Core Data stack, but a stack that is persisted on-disk due to a Core Data bug with aggregate operations (for most examples, an in-memory stack works great, and it's a fine default).

So, I'm working with a few different configurations:

1. Do not set up a core data stack if the example doesn't need it
2. Set up a core data stack in-memory if the example needs core data
3. Set up a core data stack on-disk if the example needs to work around the internal Core Data bug I mentioned

With XCTestCase, I solved this with a CoreDataTestCase subclass. I could opt-in to CoreData by changing the test case to extend CoreDataTestCase. Additionally, I could override an inMemoryStore variable I crated to control behavior of the stack during setup.

I had a hard time porting this logic to Quick. My first attempt was to create a CoreDataQuickSpec that extended QuickSpec that did the same sorts of things (overriding setup to set up the core data stack, etc). It worked, but it didn't feel right (e.g. it wasn't idiomatic).

My next approach was to create a CoreDataQuickConfiguration. This mostly worked, but I wasn't able to separate the different configurations on a per-example basis. That is, with this approach, every single example had a Core Data stack setup via the configuration and it had to be always set up on disk (because that worked around the Core Data bug across all examples). In the interest of making my specs as fast as possible, my configuration was working against me.

To solve this, I tried to tap into flags, e.g.:

describe( "a spec that uses core data with an on-disk store", flags: [ "coreDataStack": true, "inMemoryStore": false ] )

... but I quickly ran into a problem trying to read those flags from within the configuration. ExampleMetaData -> Example does not expose filterFlags at the level of visibility I needed within my QuickConfiguration subclass. Also, I'm not sure that filterFlags would be the same semantically here as perhaps configurationFlags or even perhaps an exampleParameters would be.

I'm coming to Quick with experience in Ruby/RSpec. In RSpec, there's user-defined metadata available: https://relishapp.com/rspec/rspec-core/docs/metadata/user-defined-metadata

Using metadata, I can easily extend my examples, e.g.:

describe "a behavior", freeze_time_at: "06/05/2018"

.. and then create an RSpec configuration that leverages my user-defined extension:

RSpec.configure do |config|
  # ...
  
  # Automatically triggered when `:freeze_time_at` is detected in the example's metadata.
  config.around(:example, :freeze_time_at) do |example|
    TimecopHelpers.freeze_example_at(example.metadata[:freeze_time_at], example)
  end
end

What I eventually settled on feels like I'm sort of on the right path, but it also feels like a workaround. I decided to just add Ruby-like symbols into the describe string, and use substring matching in the QuickConfiguration subclass to alter behavior. Brittle, yes. But, it works for now at least.

For example:

override class func configure( _ configuration: Configuration )
{
	// ...
	configuration.beforeEach
	{ exampleMetadata in
		guard usesCoreData( exampleMetadata ) else
		{
			return
		}
		
		// The default is in-memory store. Allow examples to prefer on-disk
		// store by specifying ":onDiskStore" in their name.
		if usesOnDiskStore( exampleMetadata )
		{
			setupCoreDataStack()
		}
		else
		{
			setupInMemoryCoreDataStack()
		}
	}
	// ...
}

class func usesCoreData( _ exampleMetadata: ExampleMetadata ) -> Bool
{
	return exampleMetadata.example.name.contains( ":coreDataStack" )
}

... which lets me write my examples like this:

describe( "a non core data example" ) { ... }
describe( "a core data example that defaults to an in memory store :coreDataStack" ) { ... }
describe( "a core data example with an on-disk store :coreDataStack :onDiskStore" ) { ... }

I could go further down this path and perhaps write some parsing logic in my configuration to try to pull out a parameter after my fake Ruby-like symbols, but so far I haven't had a need for it... and I'd rather try to modify Quick to add this behavior vs. continuing down an obvious workaround path.

Now... all of that said... is this something that Quick already supports and I am just missing the right way to achieve this? Has this already been discussed and I didn't have the right keywords to search for to find the discussion? If so, I'm happy to submit a PR to update the documentation if someone points me in the right direction.

Otherwise, I'd enjoy seeing this feature implemented. I suspect this is something that could be perhaps easily supported by either exposing filterFlags to the configuration, or creating a new parameters [String: Any] dictionary or something for each example, visible to the configuration through ExampleMetadata to take the appropriate action...

[ikesyo]

FilterFlags is not for such usecases, but for the test filtering feature (focused tests, pending tests, excluded tests). So

I suspect this is something that could be perhaps easily supported by either exposing filterFlags to the configuration

that is no go.

[paulz]

we use example flags to filter some of our specs.
Our use cases:

1. filter to verify that there is no extra test artifacts (e.g. snapshot tests PNG files)
2. filter to enable SwiftUI view stability for specs (by removing shadows that use Gaussian randomness)

[paulz]

We simply do testable import Quick to get access to flags.

Would be nice to use a public API.

[jessesquires]

Hey there! 👋🏼

I'm a new maintainer for this project and I'm trying to get the next release out ASAP and also clean up old issues and old PRs. I'm closing all issues that no longer seem relevant or are very old / stale.

This does not mean this issue is necessarily being rejected. If you are still interested in contributing the changes specified in this issue, please comment to request it be reopened.

We appreciate you opening this issue and acknowledge the time and effort you put in to contribute! You're awesome. 💯 However, we are all volunteers here with limited capacity working for free. Unfortunately, that means we must close out stale issues and PRs to move the project forward in a reasonable way.

We apologize for the inconvenience and appreciate your understanding! 😎 ✌🏼