New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Debug information #91

Closed
bartoszmajsak opened this Issue Aug 4, 2017 · 8 comments

Comments

Projects
None yet
4 participants
@bartoszmajsak
Member

bartoszmajsak commented Aug 4, 2017

We have to provide to the end users ability to have diagnostic information. This will not only help them trying to understand what is going on, but also would be very important input for us in case issues will arise.

The goal of this story would be to design a solution with following aspects:

  • how this should be enabled
  • where and how it should be presented
  • what should be part of diagnostic information

One bit which in particular would be useful an effective POM we are using after Maven Extension configures all the required bits. This is implemented now in a very ad-hoc manner as showPom method.

Thoughts and ideas about the topic more than welcome.

@bartoszmajsak

This comment has been minimized.

Show comment
Hide comment
@bartoszmajsak

bartoszmajsak Aug 4, 2017

Member

#63 is related to it.

Member

bartoszmajsak commented Aug 4, 2017

#63 is related to it.

@MatousJobanek

This comment has been minimized.

Show comment
Hide comment
@MatousJobanek

MatousJobanek Aug 8, 2017

Contributor
  • how this should be enabled

By default, it should be quiet in the console log - enable it by system property smart.testing.debug=true
The question is if the "debug log" shouldn't be stored somewhere by default - having a file in target directory where the log with all diagnostic information is stored regardless the test fails or not. Smart testing behavior may change between two test suite executions (based on the previous test suite execution) so it can be hard to reproduce the previous run.

  • where and how it should be presented

As I have already mentioned - in a file by default. When smart.testing.debug=true is used, then also in the console log.

  • what should be part of diagnostic information
  • whole ST log on the debug level.
  • link to the resulting pom file - result of showPom method
  • original list of test classes to be run and then the optimized one
  • in the list of optimized tests have also information which strategies the particular test is falling in
Contributor

MatousJobanek commented Aug 8, 2017

  • how this should be enabled

By default, it should be quiet in the console log - enable it by system property smart.testing.debug=true
The question is if the "debug log" shouldn't be stored somewhere by default - having a file in target directory where the log with all diagnostic information is stored regardless the test fails or not. Smart testing behavior may change between two test suite executions (based on the previous test suite execution) so it can be hard to reproduce the previous run.

  • where and how it should be presented

As I have already mentioned - in a file by default. When smart.testing.debug=true is used, then also in the console log.

  • what should be part of diagnostic information
  • whole ST log on the debug level.
  • link to the resulting pom file - result of showPom method
  • original list of test classes to be run and then the optimized one
  • in the list of optimized tests have also information which strategies the particular test is falling in

@dipak-pawar dipak-pawar closed this Sep 1, 2017

@dipak-pawar dipak-pawar reopened this Sep 1, 2017

@hemanik

This comment has been minimized.

Show comment
Hide comment
@hemanik

hemanik Sep 1, 2017

Contributor

how this should be enabled

As per discussion with @dipak-pawar, we should enable smart testing debug logs in two conditions:

  1. when property smart.testing.debug is set to true.
  2. when maven parameter -X is enabled along with smart testing.

where and how it should be presented

So far, we propose to log it on the console with prefix DEBUG: Smart-Testing

what should be part of diagnostic information

  • Smart Testing Extension Information such as groupID, artifactID.
  • whole ST log on the debug level.
  • result of showPom method.
  • Information related to test selections from the build artifacts relating to #63
Contributor

hemanik commented Sep 1, 2017

how this should be enabled

As per discussion with @dipak-pawar, we should enable smart testing debug logs in two conditions:

  1. when property smart.testing.debug is set to true.
  2. when maven parameter -X is enabled along with smart testing.

where and how it should be presented

So far, we propose to log it on the console with prefix DEBUG: Smart-Testing

what should be part of diagnostic information

  • Smart Testing Extension Information such as groupID, artifactID.
  • whole ST log on the debug level.
  • result of showPom method.
  • Information related to test selections from the build artifacts relating to #63
@bartoszmajsak

This comment has been minimized.

Show comment
Hide comment
@bartoszmajsak

bartoszmajsak Sep 1, 2017

Member

Sounds reasonable. Just two points:

Smart Testing Extension Information such as groupID, artifactID.

What exactly would we show here? If we have modified pom.xml isn't it enough?

result of showPom method.

  1. Maybe name can be changed and also moved to separated class e.g. ModifiedPomExporter?
  2. Where this file will be stored?
Member

bartoszmajsak commented Sep 1, 2017

Sounds reasonable. Just two points:

Smart Testing Extension Information such as groupID, artifactID.

What exactly would we show here? If we have modified pom.xml isn't it enough?

result of showPom method.

  1. Maybe name can be changed and also moved to separated class e.g. ModifiedPomExporter?
  2. Where this file will be stored?
@dipak-pawar

This comment has been minimized.

Show comment
Hide comment
@dipak-pawar

dipak-pawar Sep 1, 2017

Contributor

What exactly would we show here? If we have modified pom.xml isn't it enough?

Totally agree, we have modified pom.xml. But for that user has to look for entire pom.xml where that extension is placed in pom.xml. Instead of that we'll just show at the top when we are starting execution. So with this there is no need for user to dig into pom.xml

Contributor

dipak-pawar commented Sep 1, 2017

What exactly would we show here? If we have modified pom.xml isn't it enough?

Totally agree, we have modified pom.xml. But for that user has to look for entire pom.xml where that extension is placed in pom.xml. Instead of that we'll just show at the top when we are starting execution. So with this there is no need for user to dig into pom.xml

@bartoszmajsak

This comment has been minimized.

Show comment
Hide comment
@bartoszmajsak

bartoszmajsak Sep 1, 2017

Member

Instead of that we'll just show at the top when we are starting execution.

On the top of the log? Then he would have to scroll up if it goes to STDIO.

Member

bartoszmajsak commented Sep 1, 2017

Instead of that we'll just show at the top when we are starting execution.

On the top of the log? Then he would have to scroll up if it goes to STDIO.

@dipak-pawar

This comment has been minimized.

Show comment
Hide comment
@dipak-pawar

dipak-pawar Sep 1, 2017

Contributor

I am fine with any position.

Contributor

dipak-pawar commented Sep 1, 2017

I am fine with any position.

@MatousJobanek

This comment has been minimized.

Show comment
Hide comment
@MatousJobanek

MatousJobanek Sep 4, 2017

Contributor

So far, we propose to log it on the console with prefix [Smart-Testing-Debug]

Why this prefix. Isn't DEBUG: Smart-Testing - enough? In the same way, as we use it for WARN and INFO Logger.java

Smart Testing Extension Information such as groupID, artifactID.

I'm missing the purpose of this information. Instead of this, show just the activated strategies, the mode and the version of ST tool that is used.

INFO: Smart-Testing - Applied strategies: [new,changed]
INFO: Smart-Testing - Applied mode: [selecting]
DEBUG: Smart-Testing - Version: 0.0.1
Contributor

MatousJobanek commented Sep 4, 2017

So far, we propose to log it on the console with prefix [Smart-Testing-Debug]

Why this prefix. Isn't DEBUG: Smart-Testing - enough? In the same way, as we use it for WARN and INFO Logger.java

Smart Testing Extension Information such as groupID, artifactID.

I'm missing the purpose of this information. Instead of this, show just the activated strategies, the mode and the version of ST tool that is used.

INFO: Smart-Testing - Applied strategies: [new,changed]
INFO: Smart-Testing - Applied mode: [selecting]
DEBUG: Smart-Testing - Version: 0.0.1
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment