Add Ant build support for code coverage #279

AirQuick · 2018-08-09T17:56:59Z

This pull request derives from #269. So needs to be handled after that.

Closes #147

This pull request allows to generate the code coverage report using Ant build script.
The new property xspec.coverage.enabled controls that. (default false)

How to use

Command line

Clone this pull request.
Just add -Dxspec.coverage.enabled=true to your usual command line. For example,
```
ant -buildfile /path/to/xspec/build.xml -lib /path/to/saxon-ee.jar -Dxspec.xml=/path/to/your.xspec -Dxspec.coverage.enabled=true
```
==> *-coverage.html will be generated in the xspec subdirectory.

Oxygen

Clone this pull request. For example, clone it into /test/xspec/.
In Oxygen, open your .xspec file.
In the Transformation Scenarios pane, duplicate Run XSpec Test.
In the Options tab of Edit Ant Scenario dialog, specify /test/xspec/ in Working directory.
In the Parameters tab, set xspec.project.dir = /test/xspec/
Click New in the Parameters tab and add xspec.coverage.enabled = true.
Click New in the Parameters tab and add xspec.coverage.html = ${cfd}/${cfn}-coverage.html.
In the Output tab, set ${cfd}/${cfn}-coverage.html in Open.
Click OK to close the Edit Ant Scenario dialog.
Apply the new duplicated transformation scenario.
==> XSpec Ant begins collecting code coverage. Oxygen will open the coverage report in your default browser.

Test

Using the files in #197, I manually verified that this pull request generated the same HTML file as the current master.

No automated tests for now. It should be addressed by #195.

Further work

Document xspec.coverage.enabled in Wiki.
(xspec.coverage.html should be kept undocumented unless xspec.result.html is documented...)

…ge_stderr # Conflicts: # test/xspec-bat.cmd # test/xspec.bats

…ge_stderr # Conflicts: # test/xspec.bats

…ge_stderr

…ge_stderr # Conflicts: # test/xspec-bat.cmd # test/xspec.bats

…ge_stderr

…ge_stderr # Conflicts: # build.xml

…ge_stderr # Conflicts: # test/saxon-custom-options/config.xml # test/saxon-custom-options/test.xspec # test/xspec-bat.cmd # test/xspec.bats

…ge_stderr

AirQuick · 2018-11-19T11:03:35Z

Although labelled as enhancement, this PR is actually required for testing. The following tests depend on this PR.

Regression test for code coverage (Regression test for code coverage #326)
Test code coverage report files (as well as XML, HTML and JUnit) (Test code coverage report files (as well as XML, HTML and JUnit) #341)
Fix issue 214 (coverage-report.xsl fails to parse CDATA) (Fix issue 214 (coverage-report.xsl fails to parse CDATA) #342)

So, in short, I hope this PR get merged asap and move forward :)

AirQuick · 2018-11-20T16:56:47Z

@yamahito
This will close #147. Please let us know if you have a concern.

cirulls · 2018-11-21T16:09:41Z

@AirQuick: this is the next pull request to review on my list. I'll try to review by the end of this week, please bear with me for few days.

cirulls · 2018-11-21T17:44:59Z

@AirQuick: I tried to run it on the command line and on oXygen but I got inconsistent coverage reports, I'm not sure if it is an issue in my configuration and want to double check with you first.

Command line
Environment: Linux
Shell: bash
Ant version: 1.9.6
Saxon EE: 9.7
Command:

ant -buildfile build.xml -lib /home/cirulls/Documents/programming/saxon-9ee/saxon9ee.jar -Dxspec.xml=tutorial/escape-for-regex.xspec -Dxspec.coverage.enabled=true

I get this coverage report:

This is what I expected and it's the same coverage report I would get via xspec.sh -c. So far so good.

oXygen
oXygen version: 20.0
Ant version: 1.9.8 (default on oXygen 20.0)
Saxon EE: 9.8.0.8 (default on oXygen 20.0)

I followed your instructions for setting the transformation scenario, here is my setting:

However, I get a different coverage report:

I would expect the same report but instead I get no missing target (i.e. nothing is highlighted in red).

Could you double check if you get the same report at your end?

I initially suspected that Saxon EE was not picked up in my transformation scenario settings but a) I could not see any option in the transformation scenario to specify which Saxon processor to use; b) I would expect the transformation scenario to fail in case Saxon HE is used by default.

Can you see any obvious mistake on my oXygen settings? Could you share your configuration?

AirQuick · 2018-11-21T18:21:41Z

That difference comes from the Saxon version difference.
Using the current master + command line, Saxon 9.6 and 9.8 behave differently in the same way as you found. It's my impression that the 9.8 output (and your oXygen output) is correct.

AirQuick · 2018-11-21T18:31:31Z

And that's why I filed #196, submitted #197 (now #299) and used the files in it for testing this pull request.

cirulls · 2018-11-22T17:52:37Z

Thanks for your comments, it makes sense now. I tried the same with the tutorial files for code coverage you committed in #299 and I got identical code coverage reports on both the command line and oXygen 20.0 so this is good.

I'll finish off my review and keep you posted, I expect to merge this soon into master.

AirQuick · 2018-11-26T03:22:17Z

I'll merge this if no concern arises in the next 72 hours.

cirulls

This looks good to me. Sorry for the wait, I finished the review yesterday but didn't have time to post it. As discussed on the pull request thread, I was able to run this successfully and consistently on both the command line and oXygen 20 using the coverage demo files in #299. I also inspected the files changed and ran the test suite locally on Linux, all tests are green.

After #299 is merged and the demo files for code coverage are added to master, I think it would be useful to update the documentation for both Code Coverage and Ant (maybe adding a paragraph on how to run the code coverage via Ant). If you want, you can add now documentation for the parameter xspec.coverage.enabled in Ant.

I'm merging this into master.

@AlexJitianu and @yamahito: you may be interested in this for the oXygen harmonisation.

AirQuick · 2018-11-26T10:36:06Z

Document xspec.coverage.enabled in Wiki.

Done:

cirulls · 2018-11-26T15:14:14Z

Looks good, thanks for updating the documentation.

AirQuick added 30 commits August 2, 2018 09:59

Check Ant test results more closely

cb85346

Double quote environment variable lines

e4b5629

Enable echo new line in xspec.bats

491a8ac

Copy silently and fail on error

59752e7

Allow XML_RESOLVER_CP to be any name

303cfbb

Use work dir in xspec.bats

3e7f9a4

Use work dir in xspec-bat.cmd

3f09e1b

Add missing XQuery tests

4338907

Add comments

8db0d87

Set ANT_VERSION

159e6cc

Verify line count

473589f

Sort mkdir and rmdir lines in alphabetical order

cabaeca

Remove redundant ../test/

d5685e5

Remove redundant spaces after =~

03c08f8

Rename APOSTROPHE_DIR to SPECIAL_CHARS_DIR

96d6b9b

Adjust BaseX command line syntax to example in Wiki

32bc4f1

Check default compiled-file

db39489

Set compiled-file

0794bed

Add test for xspec.properties

3e377dc

Check the actual result of DEL and RMDIR

039980f

Delete Schematron temp files

37a7e59

Verify and delete Schematron temp files at the same time

71e669a

Replace stat with test

7d46c63

Consolidate -j tests

138c6b4

Consolidate default report file tests

8c3f418

Consolidate default HTML report file tests

9fe575a

Consolidate HTML report file inspection

0e93d9a

Remove redundant echo

67a9e03

Use demo-03 instead of demo-01

251e845

Consolidate -s tests

d8b39b6

AirQuick added 2 commits August 30, 2018 08:49

Merge branch 'master' of git://github.com/xspec/xspec into ant-covera…

0728cc1

…ge_stderr # Conflicts: # test/xspec-bat.cmd # test/xspec.bats

Merge branch 'master' of git://github.com/xspec/xspec into ant-covera…

044eca2

…ge_stderr # Conflicts: # test/xspec.bats

AirQuick mentioned this pull request Sep 5, 2018

Add tests for code coverage #195

Closed

AirQuick added 2 commits September 24, 2018 10:25

Merge branch 'master' of git://github.com/xspec/xspec into ant-covera…

0711074

…ge_stderr

Merge branch 'master' of git://github.com/xspec/xspec into ant-covera…

25aa7b1

…ge_stderr # Conflicts: # test/xspec-bat.cmd # test/xspec.bats

AirQuick mentioned this pull request Oct 2, 2018

Make inline-css the default for Ant as well #391

Closed

AirQuick added 3 commits October 7, 2018 14:58

Merge branch 'master' of git://github.com/xspec/xspec into ant-covera…

85c5218

…ge_stderr # Conflicts: # test/xspec-bat.cmd # test/xspec.bats

Merge branch 'master' of git://github.com/xspec/xspec into ant-covera…

67cdb1e

…ge_stderr

Merge branch 'master' of git://github.com/xspec/xspec into ant-covera…

0348d80

…ge_stderr

AirQuick mentioned this pull request Oct 15, 2018

Add Ant build support for XQuery #266

Merged

cirulls added this to the v1.2.0 milestone Oct 18, 2018

AirQuick added 4 commits October 22, 2018 10:22

Merge branch 'master' of git://github.com/xspec/xspec into ant-covera…

62a6a41

…ge_stderr # Conflicts: # build.xml

Merge branch 'master' of git://github.com/xspec/xspec into ant-covera…

d277023

…ge_stderr # Conflicts: # test/saxon-custom-options/config.xml # test/saxon-custom-options/test.xspec # test/xspec-bat.cmd # test/xspec.bats

Merge branch 'master' of git://github.com/xspec/xspec into ant-covera…

8bce723

…ge_stderr

Merge branch 'master' of git://github.com/xspec/xspec into ant-covera…

af09226

…ge_stderr

AirQuick requested a review from cirulls November 19, 2018 10:51

cirulls approved these changes Nov 26, 2018

View reviewed changes

cirulls merged commit 6f016ec into xspec:master Nov 26, 2018

AirQuick deleted the ant-coverage_stderr branch November 26, 2018 09:32

cirulls mentioned this pull request Sep 5, 2019

Automatically generate release notes #601

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add Ant build support for code coverage #279

Add Ant build support for code coverage #279

AirQuick commented Aug 9, 2018 •

edited

Loading

AirQuick commented Nov 19, 2018

AirQuick commented Nov 20, 2018

cirulls commented Nov 21, 2018

cirulls commented Nov 21, 2018

AirQuick commented Nov 21, 2018

AirQuick commented Nov 21, 2018

cirulls commented Nov 22, 2018

AirQuick commented Nov 26, 2018

cirulls left a comment

AirQuick commented Nov 26, 2018

cirulls commented Nov 26, 2018

Add Ant build support for code coverage #279

Add Ant build support for code coverage #279

Conversation

AirQuick commented Aug 9, 2018 • edited Loading

How to use

Command line

Oxygen

Test

Further work

AirQuick commented Nov 19, 2018

AirQuick commented Nov 20, 2018

cirulls commented Nov 21, 2018

cirulls commented Nov 21, 2018

AirQuick commented Nov 21, 2018

AirQuick commented Nov 21, 2018

cirulls commented Nov 22, 2018

AirQuick commented Nov 26, 2018

cirulls left a comment

Choose a reason for hiding this comment

AirQuick commented Nov 26, 2018

cirulls commented Nov 26, 2018

AirQuick commented Aug 9, 2018 •

edited

Loading