Add aroundEach #820

pcantrell · 2018-09-24T15:23:25Z

This is a proposed feature. It’s not yet ready to merge; I’m submitting the PR to see if Quick’s maintainers are amenable to the feature before completing the tests and documentation.

This PR adds an aroundEach decoration, which accepts the individual example as a callback:

aroundEach { example in
    ...
    example()
    ...
}

See the similar features in Rspec and JUnit.

Motivation

There are a few test setup/cleanup scenarios the existing beforeEach/afterEach don’t handle, or don’t handle well.

Some test decorations can’t split into a separate beforeEach and afterEach. The original motivation for adding aroundEach is Siesta checking for leaks in specs. To do this, it needs to wrap each spec in its own autorelease pool, then verify that objects are no longer retained after the pool is drained:

it("does this") {
    autoreleasepool {
        doThis()
    }
    checkObjectsNoLongerRetained()
}

it("does that") {
    autoreleasepool {
        doThat()
    }
    checkObjectsNoLongerRetained()
}

I’d like to pull out both the autoreleasepool block and checkObjectsNoLongerRetained() so they’re not repeated in each spec. (The two are tightly coupled, and it makes sense to factor them out together.) This is not possible with beforeEach/afterEach, but with aroundEach it is:

aroundEach { example in
    autoreleasepool {
        example()
    }
    checkObjectsNoLongerRetained()
}

it("does this") {
    doThis()
}

it("does that") {
    doThat()
}

Other methods that take a closure/block present a similar problem: XCTest’s measure, for example.

Even in cases where it is possible to split the decoration into beforeEach and afterEach, it’s not always desirable. Grouping tightly coupled before and after behavior in a single aroundEach can help ensure proper nesting of operations.

Questions

First, is this a feature Quick’s maintainers would consider merging?

Second, does this introduce any regressions? I rewrote beforeEach and afterEach in terms of afterEach. This is necessary to ensure proper mutual ordering between the three; Quick’s existing strategy of keeping separate before and after lists won’t work when “around” needs to be ordered properly with respect to both kinds.

Doing this actually cleaned up the code a bit, but I'm a little concerned about accidental breakage. The existing specs all pass, but I would want to double check that this doesn’t subtly change the ordering of spec decorators in some subtle circumstance. There’s also the possibility that it lowers the effective limit on how many before/after blocks there can be. I'd be surprised if anybody has enough of them to overflow the stack, but who knows? People do wild things!

Finally, does it need Obj-C support?

Merge checklist

Tests
Documentation
Objective-C support
Ensure new implementation of before/after does not break existing public behavior
Appease SwiftLint
Is this a new feature (Requires minor version bump)?

houndci-bot · 2018-09-24T15:23:44Z

Sources/Quick/Hooks/ExampleHooks.swift

-        phase = .beforesExecuting
-        for before in befores {
-            before(exampleMetadata)
+        wrappers.prepend { exampleMetadata, runExample in


Unused Closure Parameter Violation: Unused parameter "exampleMetadata" in a closure should be replaced with _. (unused_closure_parameter)

houndci-bot · 2018-09-24T15:23:44Z

Sources/Quick/Hooks/ExampleHooks.swift

    }

    internal func appendBefore(_ closure: @escaping BeforeExampleClosure) {
-        befores.append { (_: ExampleMetadata) in closure() }
+        wrappers.append { exampleMetadata, runExample in


Unused Closure Parameter Violation: Unused parameter "exampleMetadata" in a closure should be replaced with _. (unused_closure_parameter)

houndci-bot · 2018-09-24T15:23:45Z

Sources/Quick/Example.swift

+        let allWrappers = group!.wrappers + world.exampleHooks.wrappers
+        let wrappedExample = allWrappers.reduce(runExample) {
+            closure, wrapper in
+            { wrapper(exampleMetadata, closure) }


Opening Brace Spacing Violation: Opening braces should be preceded by a single space and on the same line as the declaration. (opening_brace)

houndci-bot · 2018-09-24T15:23:45Z

Sources/Quick/Example.swift

-            after(exampleMetadata)
+        let allWrappers = group!.wrappers + world.exampleHooks.wrappers
+        let wrappedExample = allWrappers.reduce(runExample) {
+            closure, wrapper in


Closure Parameter Position Violation: Closure parameters should be on the same line as opening brace. (closure_parameter_position)

houndci-bot · 2018-09-24T15:23:45Z

Sources/Quick/Example.swift

@@ -79,21 +79,23 @@ final public class Example: _ExampleBase {
            world.currentExampleMetadata = nil
        }

-        world.exampleHooks.executeBefores(exampleMetadata)
+


Vertical Whitespace Violation: Limit vertical whitespace to a single empty line. Currently 2. (vertical_whitespace)

QuickBot · 2018-09-24T15:28:50Z

	2 Warnings
⚠️	Need to add an unit test if you’re modifying swift source
⚠️	Big PR

Generated by 🚫 Danger

pcantrell · 2018-12-27T21:33:44Z

I’ve been using this successfully in Siesta’s specs for some time now. It works well.

Project maintainers, questions for you:

Is this a feature you would consider merging?
Does this need additional regression testing beyond the specs?
Does it need Obj-C support?

If the answer to 1 is yes, I'll work on tidying up and fixing the assorted CI issues.

tmandry · 2019-01-21T02:38:33Z

Would one of the maintainers answer the author's question, please?

I could really use this myself. In my case, I'd like to run some of my tests on a thread other than the main thread. beforeEach/afterEach obviously can't do that, but aroundEach could.

I also really like and would use the approach to testing for leaks!

pcantrell · 2019-01-21T15:53:12Z

@tmandry While the project maintainers attend to other things (I can sympathize!), you can use my fork of Quick with this patch (the around-each branch).

I don't keep the fork up-to-date with new Quick commits all that aggressively, but it is an active dependency of Siesta, and I'll keep the fork working as long as I'm depending on it.

ikesyo · 2019-04-02T16:44:09Z

I'm sorry for the late response! I'll answer as an active maintainer.

Is this a feature you would consider merging?

I'm open to this feature, that looks lovely 🤩

Does this need additional regression testing beyond the specs?

I think the existing test suite would be good enough.

Does it need Obj-C support?

I think so 😎

devxoul · 2020-04-05T17:03:08Z

I hope this PR gets merged. It seems that it is verified at least one and half year since siesta started using it.

pcantrell · 2020-04-05T17:42:34Z

I’ve been putting in some open source hours of late, so I’ll bet I could find time to add ObjC support, and also make sure it works in an ObjC-free SwiftPM environment. It would indeed be nice to get it working.

pcantrell · 2020-04-10T00:29:52Z

Added an ObjC API and fixed the SwiftLint errors. I also decided that adding some regression tests for this would indeed be worthwhile.

Writing those tests raised a sticky question: execution order. Suppose we have the following code:

context("foo") {
  beforeEach { before0() }
  afterEach { after0() }

  aroundEach { run in
      around0Prefix()
      run()
      around0Suffix()
  }

  beforeEach { before1() }
  afterEach { after1() }

  aroundEach { run in
      around1Prefix()
      run()
      around1Suffix()
  }

  beforeEach { before2() }
  afterEach { after2() }

  it("says hi") {
    print("hi")
  }
}

What order should the various before / after / around callbacks run in?

Quick’s documentation gives us fairly free reign on that question:

An example group may contain an unlimited number of beforeEach/afterEach. They'll be run in the order they're defined, but you shouldn't rely on that behavior.

The most logical order for them to run in would be nesting in and then nesting out:

// [order 0]
before0
  around0prefix
    before1
      around1prefix
        before2
        after2
      around1suffix
    after1
  around0suffix
after0

However, Quick throws a wrench into this by executing afterEach block in declaration order, instead of the reverse declaration order that nesting would demand. My existing implementation of afterEach preserves this by pushing all the afterEach blocks to the end:

// [order 1]
before0
  around0prefix
    before1
      around1prefix
        before2
      around1suffix
  around0suffix

after0
after1
after2

We could switch to order 0 above (which is actually a bit simpler to implement), but it would change Quick’s behavior for any existing code that uses multiple afterEach callbacks in the same context, even if it doesn’t use aroundEach.

FWIW, I did a quick survey of approaches in other frameworks. JUnit reorders its before/after blocks in a way that it deterministic but intentionally nonsensical, lest tests rely on order. RSpec does something very close to the nested approach, except that it forces all the around(:each) blocks to the outside (which curiously is exactly the opposite of what the RSpec docs say it does).

Preserving Quick’s behavior and keeping order 1 seems like the safest option, but I thought I’d bring it up.

Replicated non-check for proper context from beforeEach and afterEach

pcantrell · 2020-04-10T03:46:43Z

Added docs.

Fixed a Linux + Swift 4.2 issue, and ~~CI now appears to be passing~~. Addendum: I spoke too soon. Is this just one of those spurious Travis failures that disappears when you rerun it? Or is there a real problem here?

I think this is ready to merge, pending resolution of the ordering question above.

ikesyo · 2020-06-01T09:26:08Z

About ordering

I read RSpec's documentation and assume that beofreEach/afterEach hooks are wrapped by aroundEach hooks, so

Suppose we have the following code:

in that case

around0prefix
  around1prefix
    before0
    before1
    before2
    hi
    after0
    after1
    after2
  around1suffix
around0suffix

is what I expect.

around hooks are nested in declaration order
before/after hooks are wrapped in the example argument of aroundEach { example in.. } and executed when example() is called

1900cc6?w=1 is a quick sketch of this behavior.

pcantrell · 2020-06-01T15:48:57Z

Note from my comment above that RSpec actually does the exact opposite of what the docs say!

RSpec does something very close to the nested approach, except that it forces all the around(:each) blocks to the outside (which curiously is exactly the opposite of what the RSpec docs say it does).

ikesyo · 2020-06-01T16:22:50Z

Let's see rspec/rspec-core#2729, the following description is actually wrong:

WARNING: around hooks will execute after any before hooks, and before any after hooks regardless of the context they were defined in.

and the Per example hooks are wrapped by the `around` hook is expected/correct behavior.

pcantrell · 2020-06-01T21:46:50Z

If we're willing to revisit execution order, I'd strongly recommend preserving source order like this:

// [order 0]
before0
  around0prefix
    before1
      around1prefix
        before2
        after2
      around1suffix
    after1
  around0suffix
after0

This means:

all setup happens in source order, top to bottom, and
and cleanup happens in source order, bottom to top,
regardless of type of block used.

That last point is nice because it doesn’t reorder setup/cleanup operations if, for example, somebody needs to add cleanup to a beforeEach and promotes it to an aroundEach.

It’s also an argument in favor of consistency, I think, that the order above is the natural behavior of this PR if you remove the additional work I did to preserve Quick’s current order.

So if we are willing to break the existing ordering, this is the one I would advocate.

pcantrell · 2020-10-31T02:44:49Z

I’ve updated the PR to match the latest from the main branch.

I’d love to get this PR merged. Given any further thought to #989? You do make a really good point in that PR that the more logical ordering I proposed above would be consistent with addTeardownBlock. If you want to go with that more logical ordering before (or after) merging this, it’s a pretty small change, and I’d be happy to make it and update the tests.

pcantrell · 2020-10-31T14:32:34Z

P.S. I pushed the changes to switch to logical afterEach ordering in a separate commit off this PR, in case you want to see how it impacts the code and tests: pcantrell@c0de5a5

pcantrell · 2021-10-14T19:33:14Z

I’d really, really like to get this merged. Here is a proposal to resolve the ordering question:

Quick provides a flag to switch between the legacy behavior and the more logical proposed behavior.
The legacy behavior is the default.
If the flag is set to legacy behavior, then aroundEach prints a warning message that
1. briefly explains how the current behavior may produce undesired results,
2. specifies how to change the flag, and
3. links to this discussion.

That would allow this feature to be merged without resolving #989, and could also serve to pave an incremental adoption path for #989.

What do you think? I’m happy to code this up if the approach meets with the maintainers’ approval.

jessesquires · 2022-04-13T17:52:43Z

Hey there! 👋🏼

I'm a new maintainer for this project and I'm trying to get the next release out ASAP and also clean up old issues and old PRs. I'm closing all PRs older than 1-2 years, but that doesn't mean these changes are necessarily being rejected. If you are still interested in contributing these changes, please submit a new PR branching from the top of main.

We appreciate your work here and acknowledge the time and effort you put in to contribute! You're awesome. 💯 However, we are all volunteers here with limited capacity working for free. Unfortunately, that means we must close out stale issues and PRs to move the project forward in a reasonable way.

We apologize for the inconvenience and appreciate your understanding! 😎 ✌🏼

This is a reopening of #820, brought up to date with the latest main. Here is an updated summary; full historical discussion is in the original PR. I’d sure love to get this merged before it hits the 4-year mark! --- This PR adds an `aroundEach` decoration, which accepts the individual example as a callback: ```swift aroundEach { example in ... example() ... } ``` See the similar features in [Rspec](https://relishapp.com/rspec/rspec-core/v/3-0/docs/hooks/around-hooks) and [JUnit](https://junit.org/junit4/javadoc/4.12/org/junit/rules/RuleChain.html#around(org.junit.rules.TestRule)). ## Motivation There are a few test setup/cleanup scenarios the existing `beforeEach`/`afterEach` don’t handle, or don’t handle well. Some test decorations can’t split into a separate `beforeEach` and `afterEach`. The original motivation for adding `aroundEach` is [Siesta checking for leaks in specs](https://github.com/bustoutsolutions/siesta/blob/2b7bff709e0938a593de1f08415e905b4eece3ab/Tests/Functional/ResourceSpecBase.swift#L115-L145). To do this, it needs to wrap each spec in its own autorelease pool, then verify that objects are no longer retained _after_ the pool is drained: ```swift it("does this") { autoreleasepool { doThis() } checkObjectsNoLongerRetained() } it("does that") { autoreleasepool { doThat() } checkObjectsNoLongerRetained() } ``` I’d like to pull out both the `autoreleasepool` block _and_ `checkObjectsNoLongerRetained()` so they’re not repeated in each spec. (The two are tightly coupled, and it makes sense to factor them out together.) This is not possible with `beforeEach`/`afterEach`, but with `aroundEach` it is: ```swift aroundEach { example in autoreleasepool { example() } checkObjectsNoLongerRetained() } it("does this") { doThis() } it("does that") { doThat() } ``` Other methods that take a closure/block present a similar problem: XCTest’s `measure`, for example. Even in cases where it is possible to split the decoration into `beforeEach` and `afterEach`, it’s not always desirable. Grouping tightly coupled before and after behavior in a single `aroundEach` can help ensure proper nesting of operations. ## Implementation notes This PR works to preserve Quick’s somewhat illogical before/after hook execution order. (See #989.) This PR reimplements _before_ and _after_ hooks using the new feature. Internally, there is now only _around_. This appears to work; all existing specs still pass! ## Merge checklist - [x] Tests - [x] Documentation - [x] Objective-C support - [x] Ensure new implementation of before/after does not break existing public behavior - [x] Appease SwiftLint - [x] Is this a new feature (Requires minor version bump)?

houndci-bot reviewed Sep 24, 2018

View reviewed changes

VojtaStavik mentioned this pull request May 9, 2019

Add (before|after)EachAutoreleasepool hooks #887

Closed

4 tasks

pcantrell added 3 commits March 8, 2020 19:32

Added aroundEach

a1b3d49

Reimplemented beforeEach/afterEach using aroundEach

24d8110

Renamed arounds → wrappers

d91676d

pcantrell force-pushed the around-each branch from eeaddb1 to d91676d Compare March 9, 2020 00:37

Fixed lint warnings

b607a2b

pcantrell changed the title ~~[RFC] Add aroundEach~~ [WIP] Add aroundEach Apr 9, 2020

pcantrell added 3 commits April 9, 2020 19:17

Started tests for aroundEach, verified it’s used in proper context

5213cbb

Added tests for (debatable) ordering of before/after/around

bf0afc1

Added ObjC API for aroundEach

5481400

pcantrell added 3 commits April 9, 2020 22:12

Fixed aroundEach on Linux

28666c7

Replicated non-check for proper context from beforeEach and afterEach

Docs for aroundEach

26fe5f9

ObjC docs for aroundEach

39b75c1

pcantrell changed the title ~~[WIP] Add aroundEach~~ Add aroundEach Apr 10, 2020

pcantrell mentioned this pull request Apr 10, 2020

SwiftPM Quick dependency fix bustoutsolutions/siesta#309

Merged

ikesyo mentioned this pull request Jun 1, 2020

Reconsider afterEach execution order #989

Open

Merge version 3.0 into around-each

da801e9

Base automatically changed from master to main March 2, 2021 13:54

jessesquires closed this Apr 13, 2022

pcantrell mentioned this pull request Apr 14, 2022

Add aroundEach #1132

Merged

6 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add aroundEach #820

Add aroundEach #820

pcantrell commented Sep 24, 2018 •

edited

Loading

houndci-bot Sep 24, 2018

houndci-bot Sep 24, 2018

houndci-bot Sep 24, 2018

houndci-bot Sep 24, 2018

houndci-bot Sep 24, 2018

QuickBot commented Sep 24, 2018 •

edited

Loading

pcantrell commented Dec 27, 2018 •

edited

Loading

tmandry commented Jan 21, 2019 •

edited

Loading

pcantrell commented Jan 21, 2019

ikesyo commented Apr 2, 2019

devxoul commented Apr 5, 2020

pcantrell commented Apr 5, 2020

pcantrell commented Apr 10, 2020 •

edited

Loading

pcantrell commented Apr 10, 2020 •

edited

Loading

ikesyo commented Jun 1, 2020 •

edited

Loading

pcantrell commented Jun 1, 2020

ikesyo commented Jun 1, 2020

pcantrell commented Jun 1, 2020

pcantrell commented Oct 31, 2020 •

edited

Loading

pcantrell commented Oct 31, 2020 •

edited

Loading

pcantrell commented Oct 14, 2021 •

edited

Loading

jessesquires commented Apr 13, 2022

Add aroundEach #820

Add aroundEach #820

Conversation

pcantrell commented Sep 24, 2018 • edited Loading

Motivation

Questions

Merge checklist

houndci-bot Sep 24, 2018

Choose a reason for hiding this comment

houndci-bot Sep 24, 2018

Choose a reason for hiding this comment

houndci-bot Sep 24, 2018

Choose a reason for hiding this comment

houndci-bot Sep 24, 2018

Choose a reason for hiding this comment

houndci-bot Sep 24, 2018

Choose a reason for hiding this comment

QuickBot commented Sep 24, 2018 • edited Loading

pcantrell commented Dec 27, 2018 • edited Loading

tmandry commented Jan 21, 2019 • edited Loading

pcantrell commented Jan 21, 2019

ikesyo commented Apr 2, 2019

devxoul commented Apr 5, 2020

pcantrell commented Apr 5, 2020

pcantrell commented Apr 10, 2020 • edited Loading

pcantrell commented Apr 10, 2020 • edited Loading

ikesyo commented Jun 1, 2020 • edited Loading

About ordering

pcantrell commented Jun 1, 2020

ikesyo commented Jun 1, 2020

pcantrell commented Jun 1, 2020

pcantrell commented Oct 31, 2020 • edited Loading

pcantrell commented Oct 31, 2020 • edited Loading

pcantrell commented Oct 14, 2021 • edited Loading

jessesquires commented Apr 13, 2022

pcantrell commented Sep 24, 2018 •

edited

Loading

QuickBot commented Sep 24, 2018 •

edited

Loading

pcantrell commented Dec 27, 2018 •

edited

Loading

tmandry commented Jan 21, 2019 •

edited

Loading

pcantrell commented Apr 10, 2020 •

edited

Loading

pcantrell commented Apr 10, 2020 •

edited

Loading

ikesyo commented Jun 1, 2020 •

edited

Loading

pcantrell commented Oct 31, 2020 •

edited

Loading

pcantrell commented Oct 31, 2020 •

edited

Loading

pcantrell commented Oct 14, 2021 •

edited

Loading