Debug information

[bartoszmajsak]

We have to provide to the end users ability to have diagnostic information. This will not only help them trying to understand what is going on, but also would be very important input for us in case issues will arise.

The goal of this story would be to design a solution with following aspects:

• how this should be enabled
• where and how it should be presented
• what should be part of diagnostic information

One bit which in particular would be useful an effective POM we are using after Maven Extension configures all the required bits. This is implemented now in a very ad-hoc manner as showPom method.

Thoughts and ideas about the topic more than welcome.

[bartoszmajsak]

#63 is related to it.

[MatousJobanek]

• how this should be enabled

By default, it should be quiet in the console log - enable it by system property smart.testing.debug=true
The question is if the "debug log" shouldn't be stored somewhere by default - having a file in target directory where the log with all diagnostic information is stored regardless the test fails or not. Smart testing behavior may change between two test suite executions (based on the previous test suite execution) so it can be hard to reproduce the previous run.

• where and how it should be presented

As I have already mentioned - in a file by default. When smart.testing.debug=true is used, then also in the console log.

• what should be part of diagnostic information

• whole ST log on the debug level.
• link to the resulting pom file - result of showPom method
• original list of test classes to be run and then the optimized one
• in the list of optimized tests have also information which strategies the particular test is falling in

[hemanik]

how this should be enabled

As per discussion with @dipak-pawar, we should enable smart testing debug logs in two conditions:

1. when property smart.testing.debug is set to true.
2. when maven parameter -X is enabled along with smart testing.

where and how it should be presented

So far, we propose to log it on the console with prefix DEBUG: Smart-Testing

what should be part of diagnostic information

• Smart Testing Extension Information such as groupID, artifactID.
• whole ST log on the debug level.
• result of showPom method.
• Information related to test selections from the build artifacts relating to Implemented report test selection as build artifact #63

[bartoszmajsak]

Sounds reasonable. Just two points:

Smart Testing Extension Information such as groupID, artifactID.

What exactly would we show here? If we have modified pom.xml isn't it enough?

result of showPom method.

1. Maybe name can be changed and also moved to separated class e.g. ModifiedPomExporter?
2. Where this file will be stored?

[dipak-pawar]

What exactly would we show here? If we have modified pom.xml isn't it enough?

Totally agree, we have modified pom.xml. But for that user has to look for entire pom.xml where that extension is placed in pom.xml. Instead of that we'll just show at the top when we are starting execution. So with this there is no need for user to dig into pom.xml

[bartoszmajsak]

Instead of that we'll just show at the top when we are starting execution.

On the top of the log? Then he would have to scroll up if it goes to STDIO.

[dipak-pawar]

I am fine with any position.

[MatousJobanek]

So far, we propose to log it on the console with prefix [Smart-Testing-Debug]

Why this prefix. Isn't DEBUG: Smart-Testing - enough? In the same way, as we use it for WARN and INFO Logger.java

Smart Testing Extension Information such as groupID, artifactID.

I'm missing the purpose of this information. Instead of this, show just the activated strategies, the mode and the version of ST tool that is used.

INFO: Smart-Testing - Applied strategies: [new,changed]
INFO: Smart-Testing - Applied mode: [selecting]
DEBUG: Smart-Testing - Version: 0.0.1