hiding "technical" tests in report

[adrian-herscu]

Sometimes there are tests which exist only to initiate some state but they are not the focus.

For example, logging-in is required by all scenarios in an application and there also might be a dedicated LoginTest class, however the login itself is just some technicality for most scenario methods and we would like to keep it hidden in the test report.

Sometimes this can be solved with @before... stuff, but sometimes it requires too much tricks and doing it in a separate test method upon which others depend is much simpler.

Can it be done with current library? Any direction about where I need to look in order to implement such thing myself?

[janschaefer]

One way would be to just use standard mechanism of JUnit for this.

[adrian-herscu]

Hmmm... I do not use JUnit :(
What is the standard mechanism for this in TestNG?

[janschaefer]

What I basically meant was that you can execute test code outside of JGiven. That way it will also not appear in the report. In TestNG this could be done with a method annoted with @BeforeTest.

[adrian-herscu]

Yes I know that the @Before/@after methods are not reported.
Still there are situations, for example when testing long scenarios on Web/mobile applications, due to performance reasons, these have to be done inside same @Before/@after context but spread across multiple test methods. We have many of these:

class JumbleBumbleMobileTests {
  @BeforeClass
  public void beforeClassStartMobileApplication() {...}
  @AfterClass
  public void afterClassQuitMobileApplication() {...}

  @Test
  public void shouldLogin() {...}

  @Test(dependensOnMethod="shouldLogin")
  public void shouldAssociatePerson() {...}

  // This is the important method -- the ones before are just kind of initialization;
  // also these previous initialization methods are repeated in every test class
  @Test(dependensOnMethod="shouldAssociatePerson", dataProvider="details")
  public void shouldHaveDetails() {...}
}

[l-1squared]

@adrian-herscu wouldn't it be possible to just define these technical tests as a JGiven test step and then make use of the @Hidden annotation?

[adrian-herscu]

@adrian-herscu wouldn't it be possible to just define these technical tests as a JGiven test step and then make use of the @Hidden annotation?

For simple cases that is the correct approach.
Now consider having an initialization that comprise of:

1. launching a mobile application
2. login
3. sending a REST request
4. verifying something was logged in Logstash
5. setting up something in the application

These are 3 different interfaces to juggle between including actions and verifications.
So far I managed to keep actions and verifications in separate classes. Also I keep separated sets of classes for mobile, REST, SSH, and other interfaces. Doing all the above in a step means mixing all these.

[adrian-herscu]

I am thinking about changing the current JGiven reporting mechanism to recognize @hidden annotation on test methods to and just ignore these for reporting purposes.

[l-1squared]

@adrian-herscu might not be the worst idea. In a sense. Having @hidden work for tests and stages is more intuitive than having it only work for one of them

[l-1squared]

Hi,
@adrian-herscu I haven't heard from you about this issue in a while. Did you resolve your issues, so that this ticket can be closed?

[adrian-herscu]

Hi,
@adrian-herscu I haven't heard from you about this issue in a while. Did you resolve your issues, so that this ticket can be closed?

I started to look into ScenarioExecutor but found it too complex to override its behavior...
If I am not wrong, that is the place where the reporting is done. So if I will be able to hook into my handler then I will be able to filter out test methods based on some criteria, e.g. having some annotation. Any suggestion about how achieve that without rewriting the entire class or using some intrusive methods like AspectJ?

[l-1squared]

I will take a look into it this week and hopefully come up with a suggestion for you

[l-1squared]

Hi @adrian-herscu,
I finally had got to look into the code. It seems you are right. In hide tests via an annotation, one would have to add changes in ScenarioExecutor , ScenarioModelBuilder and others.

However, I just tried to run a test within a JGiven Scenario that doesn't make use of the given, when, then methods. I found that this test did not appear in the JGiven report, while still being executed.
This actually sounds like the solution you desire, doesn't it?

[adrian-herscu]

Hmmm... Issue is that all our actions and verifications are coded as JGiven steps :( ...or, maybe that I am missing your intention "doesn't make use of the given, when, then methods" ? ‫בתאריך יום ב׳, 13 ביולי 2020 ב-18:25 מאת ‪Kristof‬‏ <‪ notifications@github.com‬‏>:‬

…

Hi, I finally had got to look into the code. It seems you are right. In hide tests via an annotation, one would have to add changes in ScenarioExecutor , ScenarioModelBuilder and others. However, I just tried to run a test within a JGiven Scenario that doesn't make use of the given, when, then methods. I found that this test did not appear in the JGiven report, while still being executed. This actually sounds like the solution you desire, doesn't it? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#471 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAW3SQOKHLDC7LSOZXUV5RLR3MRP7ANCNFSM4OARRTVQ> .

[l-1squared]

That is very unfortunate...
Have you been reusing the test stages in tests that you want reported?

[l-1squared]

So, I have checked whether tests are not reported if you mark all their test steps as hidden. They do.
Given the options that already exist to not report tests, I do not think anything that could be added to JGiven to improve the report selection.
Therefore, I will close the issue