Add BlockListener support... #1575

leonard84 · 2023-02-16T18:56:24Z

This feature allows extension authors to register a IBlockListener for
a feature to observe the execution of a feature in more detail.
This surfaces some of Spock's idiosyncrasies, for example interaction
assertions are actually setup right before entering the preceding
when-block as well as being evaluated on leaving the when-block
before actually entering the then-block.

The only valid block description is a constant String, although some
users mistakenly try to use a dynamic GString. Using anything other
than a String, will be treated as a separate statement and thus ignored.

fixes #538
fixes #111

codecov · 2023-02-16T19:03:18Z

Codecov Report

Attention: Patch coverage is 79.57746% with 29 lines in your changes missing coverage. Please review.

Project coverage is 81.85%. Comparing base (2c7db77) to head (d6f4c9d).
Report is 100 commits behind head on master.

Files	Patch %	Lines
...rg/spockframework/runtime/DataIteratorFactory.java	40.00%	9 Missing ⚠️
.../java/org/spockframework/runtime/ErrorContext.java	63.15%	7 Missing ⚠️
...main/java/org/spockframework/compiler/AstUtil.java	50.00%	2 Missing ⚠️
...ava/org/spockframework/compiler/SpecAnnotator.java	33.33%	1 Missing and 1 partial ⚠️
...java/org/spockframework/compiler/SpecRewriter.java	94.73%	0 Missing and 2 partials ⚠️
...ockframework/runtime/extension/IBlockListener.java	0.00%	2 Missing ⚠️
...va/org/spockframework/runtime/model/BlockInfo.java	66.66%	2 Missing ⚠️
.../java/org/spockframework/compiler/model/Block.java	83.33%	1 Missing ⚠️
...org/spockframework/runtime/PlatformSpecRunner.java	75.00%	1 Missing ⚠️
...va/org/spockframework/runtime/model/ErrorInfo.java	75.00%	1 Missing ⚠️

Additional details and impacted files

@@             Coverage Diff              @@
##             master    #1575      +/-   ##
============================================
+ Coverage     80.44%   81.85%   +1.40%     
- Complexity     4337     4603     +266     
============================================
  Files           441      448       +7     
  Lines         13534    14444     +910     
  Branches       1707     1821     +114     
============================================
+ Hits          10888    11823     +935     
+ Misses         2008     1950      -58     
- Partials        638      671      +33

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

spock-core/src/main/java/org/spockframework/runtime/extension/IBlockListener.java

kriegaex · 2023-02-17T10:35:00Z

Hi @leonard84. I cannot perform a formal code review, because I am not a committer, but I hope that during the weekend I can find a small time slice to build and play around with it. I simply wanted to say thanks in advance for taking care of this feature request. It has not gone unnoticed.

Quick question: Are you planning to add more commits to this PR? Code changes? User manual? I am just asking, not demanding anything. I simply do not want to start testing too early. As for the user manual, I can of course take a look at the unit tests and take it from there. But that might not be true for all future extension developers, I am just speaking for myself.

leonard84 · 2023-03-10T10:47:32Z

@kriegaex I'll write some documentation for it, but I mainly wanted to get feedback on the feature first, like the one from @szpak concerning the usability.

leonard84 · 2023-03-10T10:50:34Z

...ecs/src/test/groovy/org/spockframework/smoke/ast/condition/ExceptionConditionsAstSpec.groovy

+        try {
+            org.spockframework.runtime.SpockRuntime.callEnterBlock(this.getSpecificationContext(), new org.spockframework.runtime.model.BlockInfo(org.spockframework.runtime.model.BlockKind.WHEN, []))
+            foobar = this.foobar()
+            org.spockframework.runtime.SpockRuntime.callExitBlock(this.getSpecificationContext(), new org.spockframework.runtime.model.BlockInfo(org.spockframework.runtime.model.BlockKind.WHEN, []))


💬 this won't be called, fixing this makes for some awkward interactions between the individual AST transformations,

I tried moving the blockListener logic where the blocks are written back in SpecRewriter#L388, however this didn't work for some cases, e.g. cleanup as it already replaced all Blocks with an anonymous block.

spock-specs/src/test/groovy/org/spockframework/smoke/ast/mock/MocksAstSpec.groovy

spock-core/src/main/java/org/spockframework/runtime/model/ErrorInfo.java

spock-core/src/main/java/org/spockframework/runtime/extension/IBlockListener.java

szpak · 2023-10-02T15:42:21Z

As this PR seems a little bit stalled, I pushed the code I initially created ~6 months ago as a PoC for the BlockListener support. It a very raw version (with a lot of diagnostic stuff to be amended in the future), but functional in some basic scenarios. The JPA session, where requested, is flushed after the when block.

https://github.com/szpak/spock-jpa-flush-enforcer/tree/preview1

I would like to awake discussion about future shape of this PR.

@leonard84 @kriegaex WDYT?

leonard84 · 2023-10-06T14:23:29Z

It mostly hangs on the problems that adding the exit listener introduces #1575 (comment)

leonard84 · 2024-03-21T17:26:41Z

I think I've fixed the issues, but I need another fresh set of eyes to verify that everything looks good, I've stared at too many snapshots already.

spock-core/src/main/java/org/spockframework/runtime/ErrorContext.java

spock-core/src/main/java/org/spockframework/runtime/SpecificationContext.java

spock-core/src/main/java/org/spockframework/runtime/SpockRuntime.java

spock-core/src/main/java/org/spockframework/runtime/model/FeatureInfo.java

spock-core/src/main/java/org/spockframework/runtime/model/IErrorContext.java

spock-core/src/main/java/org/spockframework/runtime/ErrorContext.java

spock-core/src/main/java/org/spockframework/runtime/extension/IBlockListener.java

leonard84 · 2024-03-21T19:09:55Z

@AndreasTu, as I mentioned earlier, I'll write some documentation and polish it later. Thanks for your comments in any case.

However, I'm looking for a review of the correctness of the implementation and generated code and general usability.

szpak

I've switched my experimental (and heavily work-in-progress) extension to the latest version and it still works as expected without any modification. Nice.

I will think about the possible corner cases, I might encounter there.

Btw, I wonder, if it is still a recommended way to decide if the block listener was intended for the current iteration?

spock-core/src/main/java/org/spockframework/runtime/SpockRuntime.java

spock-core/src/main/java/org/spockframework/runtime/extension/IBlockListener.java

spock-specs/src/test/groovy/org/spockframework/smoke/parameterization/DataProviders.groovy

spock-core/src/main/java/org/spockframework/runtime/model/BlockInfo.java

Vampire

Some things missing in multiple places:

license headers
JavaDoc
documentation
@Beta
@since

spock-core/src/main/java/org/spockframework/compiler/AstUtil.java

spock-core/src/main/java/org/spockframework/compiler/DeepBlockRewriter.java

spock-core/src/main/java/org/spockframework/compiler/SpecRewriter.java

spock-core/src/main/java/org/spockframework/runtime/model/ErrorInfo.java

spock-core/src/main/java/org/spockframework/runtime/model/FeatureInfo.java

spock-core/src/main/java/org/spockframework/runtime/model/IErrorContext.java

Vampire · 2024-03-21T22:06:11Z

Btw. could we also support where (and upcoming filter) blocks in a sensible way?

leonard84 · 2024-03-22T00:46:42Z

Btw. could we also support where (and upcoming filter) blocks in a sensible way?

I don't think it would be intuitive or really helpful.
They run outside the normal iteration.
When would you enter or leave the block?

only once when creating the data provider
on every data iterator's iteration?

However, you are most familiar with that part of the code.

Vampire · 2024-03-22T03:57:47Z

When would you enter or leave the block?

No idea. :-D
I guess multiple times.
Maybe with an additional phase enum.
One for creating the data iterators.
One for getting the next set of data variable values.

But we can also start without and see where and how need arises.
But then it needs time to land in production version with our release cadence. :-D

leonard84 · 2024-04-19T16:13:06Z

Forgot to publish my responses 😅

spock-core/src/main/java/org/spockframework/compiler/SpecRewriter.java

spock-core/src/main/java/org/spockframework/runtime/ErrorContext.java

spock-core/src/main/java/org/spockframework/runtime/SpecificationContext.java

spock-core/src/main/java/org/spockframework/lang/ISpecificationContext.java

leonard84 · 2024-04-22T17:50:04Z

Btw, I wonder, if it is still a recommended way to decide if the block listener was intended for the current iteration?

@szpak, can you motivate this use case more? At first glance, I'd try to register a block listener that can handle all iterations instead of one per iteration.

szpak · 2024-04-25T20:00:09Z

@szpak, can you motivate this use case more? At first glance, I'd try to register a block listener that can handle all iterations instead of one per iteration.

Hmm, I think the main problem was to get the invocation instance to obtain the current field instance. As a result, I put IBlockListener inside AbstractMethodInterceptor which provides invocation. Would you propose @leonard84 to pass the invocation instance to the listener in any simpler way?

leonard84 · 2024-05-14T18:19:25Z

@szpak, can you motivate this use case more? At first glance, I'd try to register a block listener that can handle all iterations instead of one per iteration.

Hmm, I think the main problem was to get the invocation instance to obtain the current field instance. As a result, I put IBlockListener inside AbstractMethodInterceptor which provides invocation. Would you propose @leonard84 to pass the invocation instance to the listener in any simpler way?

We could easily pass in the current instance. The question is whether we should.

I've also been debating whether I should give access to the current ISpockExecution so that a listener can get an IStore.

szpak · 2024-05-18T18:19:42Z

Hmm, I think the main problem was to get the invocation instance to obtain the current field instance. As a result, I put IBlockListener inside AbstractMethodInterceptor which provides invocation. Would you propose @leonard84 to pass the invocation instance to the listener in any simpler way?

We could easily pass in the current instance. The question is whether we should.

Definitely that's the good question. What are the design difference between interceptors (which have access to invocation) and listeners (also in work there are intended for)? The first could abort the processing, while listeners should not. Listeners should only perform (external) side effects? What also is different and if invocation in the second case could "complicate" something?

I've also been debating whether I should give access to the current ISpockExecution so that a listener can get an IStore.

Do you see any good cases where it would be necessary for listeners in practice? To read the values set by the other extensions/interceptors?

szpak · 2024-05-31T16:07:40Z

@szpak FWIW, you can probably get away with using a single instance of each and a ThreadLocal to share the data from the interceptor to the listener.

Trying to use ThreadLocal was failing with strange effect at first and in the end, it turned out that my original problem with "wrong iteration" was caused by the fact, I was not removing the block listener added in the iteration interceptor. As a result, in the second iteration 2 listeners were called (and checking the iteration index worked as a workaround)...

In the end, I can use just one block listener without any iteration index checking. Thanks.

https://github.com/szpak/spock-jpa-flush-enforcer/compare/preview2...preview3?expand=1

leonard84 · 2024-05-31T16:16:09Z

@szpak you should just register the blocklister once for the feature where you add the interationInterceptor instead of adding/removing it every time. Otherwise, you'll run into problems in parallel execution mode.

szpak · 2024-05-31T18:34:40Z

@szpak you should just register the blocklister once for the feature where you add the interationInterceptor instead of adding/removing it every time. Otherwise, you'll run into problems in parallel execution mode.

Thanks, fixed:
szpak/spock-jpa-flush-enforcer@bb830e1

szpak · 2024-06-03T20:17:57Z

Regarding GString in labels, I do not have a strong opinion. It might be useful for (more) fancy reporting, so some people might be delighted. On the other hand, I don't know how my (performance) impact it will have.

It would be a potential breaking change, although the impact should be negligible.

Some people might use it as a informal (and broken) placeholders. However, it should be easy to fix.

Btw, could we reuse BlockInfo if no GString is detected, and create a new instance only if an evaluation is needed?

leonard84 · 2024-06-05T21:38:31Z

Btw, could we reuse BlockInfo if no GString is detected, and create a new instance only if an evaluation is needed?

Maybe, with increased complexity.

AndreasTu · 2024-06-06T15:53:52Z

I have no real preference in any way for or against the GString support.

How about we go with the existing behavior, and if someone wants the GString support in the future, we can still make the break?

renatoathaydes · 2024-06-20T15:19:25Z

@renatoathaydes do you have any comments about the new listeners?

I am a bit out of the loop. But this sounds very useful to me.

This feature allows extension authors to register a IBlockListener for a feature to observe the execution of a feature in more detail. This surfaces some of Spock's idiosyncrasies, for example interaction assertions are actually setup right before entering the preceding `when`-block as well as being evaluated on leaving the `when`-block before actually entering the `then`-block. The only valid block description is a constant String, although some users mistakenly try to use a dynamic GString. Using anything other than a String, will be treated as a separate statement and thus ignored.

Prior to this commit, IRunListener.error(ErrorInfo) didn't give any context where the error happened.

leonard84 · 2024-06-23T17:23:08Z

@renatoathaydes, please take a look at the new BlockListener and ErrorInfo interfaces.

I guess you are going to be one of the prime users of this feature, so if you have any comments, now would be the best time to address them.

renatoathaydes · 2024-06-25T08:00:33Z

@leonard84 oh I see. I will try to checkout this code and get spock-reports to use the new feature, I suppose that will let me fix renatoathaydes/spock-reports#242 and maybe even renatoathaydes/spock-reports#89?

leonard84 mentioned this pull request Feb 16, 2023

Adding block listener support #111

Closed

szpak reviewed Feb 16, 2023

View reviewed changes

spock-core/src/main/java/org/spockframework/runtime/extension/IBlockListener.java Outdated Show resolved Hide resolved

leonard84 force-pushed the add-block-listener-support branch from e49b0e3 to f1d1840 Compare March 10, 2023 10:46

leonard84 commented Mar 10, 2023

View reviewed changes

szpak reviewed Mar 13, 2023

View reviewed changes

spock-core/src/main/java/org/spockframework/runtime/model/ErrorInfo.java Outdated Show resolved Hide resolved

spock-core/src/main/java/org/spockframework/runtime/extension/IBlockListener.java Outdated Show resolved Hide resolved

leonard84 force-pushed the add-block-listener-support branch from c3f30eb to 0f3197e Compare May 7, 2023 05:41

leonard84 force-pushed the add-block-listener-support branch from 0f3197e to ae38bda Compare February 25, 2024 16:53

leonard84 force-pushed the add-block-listener-support branch from ae38bda to 353cbf9 Compare March 21, 2024 17:20

leonard84 requested review from Vampire and AndreasTu March 21, 2024 17:24

AndreasTu requested changes Mar 21, 2024

View reviewed changes

szpak reviewed Mar 21, 2024

View reviewed changes

Vampire requested changes Mar 21, 2024

View reviewed changes

Vampire requested changes Apr 20, 2024

View reviewed changes

leonard84 force-pushed the add-block-listener-support branch from 5a2f3f2 to c9ee05f Compare April 28, 2024 11:35

leonard84 added 17 commits June 23, 2024 18:02

Expose IErrorContext in ErrorInfo...

78b4071

Prior to this commit, IRunListener.error(ErrorInfo) didn't give any context where the error happened.

Clear current block on exit from feature method

fc66c53

Update AST tests with block listeners

344cd96

Add blockExited to IBlockListener

eac9130

Add interaction example AstSpec

990394a

Fix block listeners for interactions

0681739

Implement Review Feedback

3ee0203

Fix test in RunListenerSpec

eca6a4d

Implement Review comments

fa40dbe

Move current block clearing to PlatformSpecRunner

4eb7cf8

Move DataProviders expected ast to snapshots

3a74036

Restore failed block when cleanup runs

4c23e6d

Implement Review comments re SpecificationContext

18d73fb

Implement Review comments re ErrorContext/Info

5922d3f

Revert all changes to DeepBlockRewriter

158bafe

Implement Review Comments

6d3c148

leonard84 force-pushed the add-block-listener-support branch 2 times, most recently from dcb9f79 to b819d74 Compare June 23, 2024 17:09

Use BlockInfo from SpecInfo via an index lookup

d6f4c9d

leonard84 force-pushed the add-block-listener-support branch from b819d74 to d6f4c9d Compare June 23, 2024 17:22

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add BlockListener support... #1575

Add BlockListener support... #1575

leonard84 commented Feb 16, 2023

codecov bot commented Feb 16, 2023 •

edited

Loading

kriegaex commented Feb 17, 2023

leonard84 commented Mar 10, 2023

leonard84 Mar 10, 2023

leonard84 Mar 10, 2023

szpak commented Oct 2, 2023 •

edited

Loading

leonard84 commented Oct 6, 2023

leonard84 commented Mar 21, 2024

leonard84 commented Mar 21, 2024

szpak left a comment •

edited

Loading

Vampire left a comment

Vampire commented Mar 21, 2024

leonard84 commented Mar 22, 2024

Vampire commented Mar 22, 2024

leonard84 commented Apr 19, 2024

leonard84 commented Apr 22, 2024

szpak commented Apr 25, 2024 •

edited

Loading

leonard84 commented May 14, 2024

szpak commented May 18, 2024 •

edited

Loading

szpak commented May 31, 2024 •

edited

Loading

leonard84 commented May 31, 2024

szpak commented May 31, 2024 •

edited

Loading

szpak commented Jun 3, 2024

leonard84 commented Jun 5, 2024

AndreasTu commented Jun 6, 2024

renatoathaydes commented Jun 20, 2024

leonard84 commented Jun 23, 2024

renatoathaydes commented Jun 25, 2024

Add BlockListener support... #1575

Are you sure you want to change the base?

Add BlockListener support... #1575

Conversation

leonard84 commented Feb 16, 2023

codecov bot commented Feb 16, 2023 • edited Loading

Codecov Report

kriegaex commented Feb 17, 2023

leonard84 commented Mar 10, 2023

leonard84 Mar 10, 2023

Choose a reason for hiding this comment

leonard84 Mar 10, 2023

Choose a reason for hiding this comment

szpak commented Oct 2, 2023 • edited Loading

leonard84 commented Oct 6, 2023

leonard84 commented Mar 21, 2024

leonard84 commented Mar 21, 2024

szpak left a comment • edited Loading

Choose a reason for hiding this comment

Vampire left a comment

Choose a reason for hiding this comment

Vampire commented Mar 21, 2024

leonard84 commented Mar 22, 2024

Vampire commented Mar 22, 2024

leonard84 commented Apr 19, 2024

leonard84 commented Apr 22, 2024

szpak commented Apr 25, 2024 • edited Loading

leonard84 commented May 14, 2024

szpak commented May 18, 2024 • edited Loading

szpak commented May 31, 2024 • edited Loading

leonard84 commented May 31, 2024

szpak commented May 31, 2024 • edited Loading

szpak commented Jun 3, 2024

leonard84 commented Jun 5, 2024

AndreasTu commented Jun 6, 2024

renatoathaydes commented Jun 20, 2024

leonard84 commented Jun 23, 2024

renatoathaydes commented Jun 25, 2024

codecov bot commented Feb 16, 2023 •

edited

Loading

szpak commented Oct 2, 2023 •

edited

Loading

szpak left a comment •

edited

Loading

szpak commented Apr 25, 2024 •

edited

Loading

szpak commented May 18, 2024 •

edited

Loading

szpak commented May 31, 2024 •

edited

Loading

szpak commented May 31, 2024 •

edited

Loading