New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

TEAM Engine 5.0 new report #81

Closed
bermud opened this Issue Feb 12, 2015 · 17 comments

Comments

Projects
None yet
6 participants
@bermud
Contributor

bermud commented Feb 12, 2015

Related to #39 and #45

The report should have at least these elements and be very simple to understand.

Main page for report

Summary Results

  • Test Name
  • Test id
  • Test version
  • Test date
  • Basic Conformance Classes
  • Test input
    • $key $value
  • Result
    • Number of conformance classes tested:
    • Passed core (can be certified) ?
    • Number of conformance class passed?
    • Number of conformance class failed?

Conformance Classes

For each conformance class

  • Conformance Class Name
  • Conformance Class Description

Summary box pass fail

The user should be able to click on a row on the table (a test) and get another page with the following information:

Page for a particular test

  • name of the test
  • description about the test (assertion)
  • result
  • inputs (request, file , part of file)
  • outputs (response provided by the implementation being tested or output file, etc)
  • if it failed
    • the reason
    • possible changes

results

Updates required on tests

Changes are required on existing tests to support this new reporting. Details here: https://github.com/opengeospatial/cite/wiki/Encoding-Conformance-Classes

@bermud

This comment has been minimized.

Show comment
Hide comment
@bermud
Contributor

bermud commented Apr 1, 2015

@bermud

This comment has been minimized.

Show comment
Hide comment
@bermud

bermud Oct 9, 2015

Contributor

Ideas for improvement were discussed and coded in this pull request
#78

Contributor

bermud commented Oct 9, 2015

Ideas for improvement were discussed and coded in this pull request
#78

@bermud bermud added waiting and removed in-investigation labels Dec 22, 2015

@bermud bermud added in-investigation and removed waiting labels Apr 7, 2016

@bermud

This comment has been minimized.

Show comment
Hide comment
@bermud

bermud Apr 7, 2016

Contributor

Please continue looking into EARL and conversion from TestNG and CTL output to RDF and then to HTML.

Contributor

bermud commented Apr 7, 2016

Please continue looking into EARL and conversion from TestNG and CTL output to RDF and then to HTML.

@bermud bermud closed this Apr 19, 2016

@bermud bermud reopened this Apr 19, 2016

@bermud

This comment has been minimized.

Show comment
Hide comment
@bermud

bermud Jun 7, 2016

Contributor

An initial implementation of an EARL listener for TestNG should be available by the end of this week; see #152.

We can use the output of this as a basis for considering what info should be included in the results.

Work to be advance:

  • script to convert CTL test results to EARL RDF/XML
  • script to convert EARL RDF/XML results to HTML5 report
Contributor

bermud commented Jun 7, 2016

An initial implementation of an EARL listener for TestNG should be available by the end of this week; see #152.

We can use the output of this as a basis for considering what info should be included in the results.

Work to be advance:

  • script to convert CTL test results to EARL RDF/XML
  • script to convert EARL RDF/XML results to HTML5 report

@bermud bermud added waiting and removed in-investigation labels Jul 22, 2016

@bermud

This comment has been minimized.

Show comment
Hide comment
@bermud

bermud Sep 16, 2016

Contributor

@keshav-nangare EARL Listener was implemented in 4.7.1. We should be able to move this task forward.

Contributor

bermud commented Sep 16, 2016

@keshav-nangare EARL Listener was implemented in 4.7.1. We should be able to move this task forward.

@bermud bermud added in-investigation and removed waiting labels Sep 16, 2016

@keshav-nangare

This comment has been minimized.

Show comment
Hide comment
@keshav-nangare

keshav-nangare Sep 17, 2016

Contributor

The branch "earl_html_report" contains xslt file, which will convert the earl resut into HTML report.

Commit: 191084d

Contributor

keshav-nangare commented Sep 17, 2016

The branch "earl_html_report" contains xslt file, which will convert the earl resut into HTML report.

Commit: 191084d

@bermud bermud changed the title from Propose a solution for improving reporting in TEAM ENGINE to Develop script to convert HTML report from EARL Nov 22, 2016

@keshav-nangare

This comment has been minimized.

Show comment
Hide comment
@keshav-nangare

keshav-nangare Mar 7, 2017

Contributor

@rjmartell

We are trying to add the XML content in the EARL report it is added with the encoded format.

Example:
XML content:

41

    EARL Report:
                <cnt:ContentAsXML>
                    <cnt:rest>&lt;?xml version="1.0" encoding="UTF-8"? &lt;values&gt;&lt;value key="inputs"&gt;41&lt;/value&gt;&lt;/values&gt;</cnt:rest>
                </cnt:ContentAsXML>

Is there any way to add these xml content in earl report as it is?

Contributor

keshav-nangare commented Mar 7, 2017

@rjmartell

We are trying to add the XML content in the EARL report it is added with the encoded format.

Example:
XML content:

41

    EARL Report:
                <cnt:ContentAsXML>
                    <cnt:rest>&lt;?xml version="1.0" encoding="UTF-8"? &lt;values&gt;&lt;value key="inputs"&gt;41&lt;/value&gt;&lt;/values&gt;</cnt:rest>
                </cnt:ContentAsXML>

Is there any way to add these xml content in earl report as it is?

@keshav-nangare

This comment has been minimized.

Show comment
Hide comment
@keshav-nangare

keshav-nangare Mar 8, 2017

Contributor

CTL EARL Report:

The branch "earl_report" contains the work that we have done up till now.

We have updated the teamengine with the following:

  1. The teamengine will generate the EARL report when CTL test executed through the web
  2. Moved the CtlEarlReporter class to teamengine-core module.
  3. Updated the request & response. Added the isBasic attribute in earl report from CTL test context.
  4. Added the isBasic attribute in TestNG test context and updated the testName with good names:
    - While adding the isBasic attribute in the TestNG test context we need to do following changes:
    - We need to add the conformance classes into config.xml of the corresponding test.
                   - Example: 
                               &lt;conformanceClasses&gt;
	                             &lt;conformanceClass&gt;GML Documents&lt;/conformanceClass&gt;
                               &lt;/conformanceClasses&gt;
  1. Fixed the result issue of the viewSessionLog page.
  2. Added the request and response in earl report.
Contributor

keshav-nangare commented Mar 8, 2017

CTL EARL Report:

The branch "earl_report" contains the work that we have done up till now.

We have updated the teamengine with the following:

  1. The teamengine will generate the EARL report when CTL test executed through the web
  2. Moved the CtlEarlReporter class to teamengine-core module.
  3. Updated the request & response. Added the isBasic attribute in earl report from CTL test context.
  4. Added the isBasic attribute in TestNG test context and updated the testName with good names:
    - While adding the isBasic attribute in the TestNG test context we need to do following changes:
    - We need to add the conformance classes into config.xml of the corresponding test.
                   - Example: 
                               &lt;conformanceClasses&gt;
	                             &lt;conformanceClass&gt;GML Documents&lt;/conformanceClass&gt;
                               &lt;/conformanceClasses&gt;
  1. Fixed the result issue of the viewSessionLog page.
  2. Added the request and response in earl report.
@keshav-nangare

This comment has been minimized.

Show comment
Hide comment
@keshav-nangare

keshav-nangare Mar 22, 2017

Contributor

Updated the EARL HTML Report:

Following are screenshots of CTL and TestNG tests results:

CTL sample test result:

Test Result:
ctl_sample_test_result

Test Details:
ctl_sample_testdetails_result

TestNG gml32 test result:

Test Result:
testng_gml_test_result

Test Details:
testng_gml_testdetails_result

@bermud
Please review this report and suggest if any changes required.

Thank you.

Contributor

keshav-nangare commented Mar 22, 2017

Updated the EARL HTML Report:

Following are screenshots of CTL and TestNG tests results:

CTL sample test result:

Test Result:
ctl_sample_test_result

Test Details:
ctl_sample_testdetails_result

TestNG gml32 test result:

Test Result:
testng_gml_test_result

Test Details:
testng_gml_testdetails_result

@bermud
Please review this report and suggest if any changes required.

Thank you.

@bermud

This comment has been minimized.

Show comment
Hide comment
@bermud

bermud Mar 22, 2017

Contributor

@keshav-nangare great progress
Recommendations:

  • Test with CTL test (e.g. WFS 1.1)
  • Test Input should have a value
  • The link to the test should be presented in one page. See the first comment on this issue.
  • Main information of the test should summarize conformance classes. See the first comment on this issue.
Contributor

bermud commented Mar 22, 2017

@keshav-nangare great progress
Recommendations:

  • Test with CTL test (e.g. WFS 1.1)
  • Test Input should have a value
  • The link to the test should be presented in one page. See the first comment on this issue.
  • Main information of the test should summarize conformance classes. See the first comment on this issue.
@bermud

This comment has been minimized.

Show comment
Hide comment
@bermud

bermud Mar 30, 2017

Contributor

Mar 30 - From the individual test report:

  • add a link to “go back to the main report”
  • remove the “java.lang.Assertion>error> and other types of java errors. Just provide the text of the error
  • Remove the" expected true but found false" message.
  • Reason of failure should be "Reason of Failure"
  • Show colorful XML for the requests and responses if they are in XML
  • For future: provide XML snippet and highlight where the code fails
Contributor

bermud commented Mar 30, 2017

Mar 30 - From the individual test report:

  • add a link to “go back to the main report”
  • remove the “java.lang.Assertion>error> and other types of java errors. Just provide the text of the error
  • Remove the" expected true but found false" message.
  • Reason of failure should be "Reason of Failure"
  • Show colorful XML for the requests and responses if they are in XML
  • For future: provide XML snippet and highlight where the code fails
@bermud

This comment has been minimized.

Show comment
Hide comment
@bermud

bermud Apr 24, 2017

Contributor

April 24 Meeting

  • change the test name to be human readable (w.g. GML 3.2)
  • add test version
  • add test id
  • before test id; add “Basic Conformance Classes"
  • improve the key an valued in tabular form
  • update the test to include information about conformance classes
  • specify in the report, which ones are the core and make sure they appear first.
  • create issues on all tests github repositories that require update
Contributor

bermud commented Apr 24, 2017

April 24 Meeting

  • change the test name to be human readable (w.g. GML 3.2)
  • add test version
  • add test id
  • before test id; add “Basic Conformance Classes"
  • improve the key an valued in tabular form
  • update the test to include information about conformance classes
  • specify in the report, which ones are the core and make sure they appear first.
  • create issues on all tests github repositories that require update
@bermud

This comment has been minimized.

Show comment
Hide comment
@bermud

bermud May 23, 2017

Contributor

Provided by Keshav 2017.05.23
test session results 2

Comments:
The new reported addresses the points from previous comment.

We need to improve as follows:

  • Conformance Clases (in the 5th line) should be indented and presented similar to how the items of the Result are presented
  • add statistics (counts) about number of tests in conformance classes. See issue #216
  • test inputs should be indented and presented similar to how the items of the Result are presented
Contributor

bermud commented May 23, 2017

Provided by Keshav 2017.05.23
test session results 2

Comments:
The new reported addresses the points from previous comment.

We need to improve as follows:

  • Conformance Clases (in the 5th line) should be indented and presented similar to how the items of the Result are presented
  • add statistics (counts) about number of tests in conformance classes. See issue #216
  • test inputs should be indented and presented similar to how the items of the Result are presented
@keshav-nangare

This comment has been minimized.

Show comment
Hide comment
@keshav-nangare

keshav-nangare Jun 21, 2017

Contributor

Updated Report with the suggestions:

gml32_result_with_new_earl_report

Contributor

keshav-nangare commented Jun 21, 2017

Updated Report with the suggestions:

gml32_result_with_new_earl_report

@dstenger

This comment has been minimized.

Show comment
Hide comment
@dstenger

dstenger Sep 27, 2017

Contributor

There are still open check boxes in previous comments. Are all tasks solved?

Contributor

dstenger commented Sep 27, 2017

There are still open check boxes in previous comments. Are all tasks solved?

@bermud bermud changed the title from Develop script to convert HTML report from EARL to TEAM Engine 5.0 new report Nov 20, 2017

@dstenger

This comment has been minimized.

Show comment
Hide comment
@dstenger

dstenger Nov 20, 2017

Contributor

TODO: Review this issue and add new ideas/ improvements to wiki page.

Contributor

dstenger commented Nov 20, 2017

TODO: Review this issue and add new ideas/ improvements to wiki page.

@lgoltz

This comment has been minimized.

Show comment
Hide comment
@lgoltz

lgoltz Dec 20, 2017

Contributor

Open issue documented in #285. New ideas should be reported as new issue.

Contributor

lgoltz commented Dec 20, 2017

Open issue documented in #285. New ideas should be reported as new issue.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment