[WIP] RFC8 - Move to pytest testing tool [Issue#428] #492

ricardogsilva · 2016-12-14T18:34:19Z

Overview

This pull request implements the changes proposed in RFC8 - Move to pytest testing tool

The work is not finished yet. I am submitting the PR already in order to have a more public place for early review for the RFC and also to allow testing the incoming changes to travis.

A rough draft of the completed/todo list and current status:

I plan to complete the missing items on the list shortly and then call for a vote on the RFC. Currently I'm facing some troubles with the harvesting suite as I'm getting some errors that seem to be external to pycsw. The other suites are all passing.

Related Issue / Discussion

RFC8 - Move to pytest testing tool
Issue #428

Additional Information

I'm contemplating other tasks that could be done and that are somehow related to this PR. I've decided to keep them out of scope for this:

Running tests in parallel with the pytest-xdist plugin - May require further changes and I did not feel like delaying this PR even more;
All tests pass with postgresql as backend - There are some tests that are currently not working with postgresql. This has nothing to do with this PR, the tests are broken in the current master as well. We can tackle this more effectively once we get an automated way to run postgresql tests;
Allow running tests with a mysql database - Requires further development. Since this feature is not present in the current master I guess it is not a priority;
Subscribing to a code coverage service like codecov - This would give some visibility to our code coverage and help improving it. This particular service is free for open source projects. This seems like something to discuss with other devs (or maybe the PSC?) after merging this PR.

Contributions and Licensing

(as per https://github.com/geopython/pycsw/blob/master/CONTRIBUTING.rst#contributions-and-licensing)

I'd like to contribute [feature X|bugfix Y|docs|something else] to pycsw. I confirm that my contributions to pycsw will be compatible with the pycsw license guidelines at the time of contribution.
I have already previously agreed to the pycsw Contributions and Licensing Guidelines

These are not needed anymore on GET requests

Removed a lot of boiler plate code, simplifying things

Reorganized the location of test data files. This keeps things a bit cleaner

These are not needed anymore on GET requests

Removed a lot of boiler plate code, simplifying things

Reorganized the location of test data files. This keeps things a bit cleaner

…to issue#428

…ites

Removed a lot of boiler plate code, simplifying things

These changes have to be manual, since this branch changed the location of the expected test files

This simplifies installation for develoment and testing purposes by requiring a single pip install -r requirements-dev.txt

…to issue#428

tomkralidis · 2017-01-10T03:09:41Z

cc @kalxas

@ricardogsilva I've done a first pass review and have provide specific comments/questions as part of the review -- very impressive work here!

Additional comments:

what is the difference between tests/conftest.py and tests/functionaltests/conftest.py?
this PR should cleanup git commits. I don't mind (and encourage) the numerous commits that are specific to the PR/RFC, but let's make sure the other changes made upstream are not re-committed here
what/where is the actual test runner workflow/code? A high level explanation of the workflow would be valuable so other devs wishing to update the test code have a clearer picture

tomkralidis · 2017-01-10T02:31:03Z

.coveragerc

+source =
+    pycsw
+    .tox/*/lib/python*/site-packages/pycsw
+    /usr/local/lib/python*/dist-packages/pycsw


Is this hardcoded path safe enough or should it be abstracted somehow?

I believe so.

The second line refers to paths inside the virtualenvs generated by tox, so they are always the same, and they do not depend on the base system.

The third line is the standard path where python packages are installed whenever they are pip installed in a debian-like system. I'm not so sure this is the same on other distros. However, I don't think it is relevant, as this configuration's purpose is for allowing the coverage output's to be treated as the same by coverage.py, specifically for the continuous integration service. Since we are using Travis, which is based on ubuntu, the paths are the expected ones.

tomkralidis · 2017-01-10T02:31:40Z

.coveragerc

@@ -0,0 +1,12 @@
+[run]


is this file for coverage.py? Is coverage.py enabled by the tox --cov flag?

Yes, .coveragerc is the configuration for running coverage.py. More details here. In our case I am using it mostly to:

enable branch covering, which is useful to see whether tests cover multiple options in if.. elif.. else blocks

allow for collecting results inside tox virtualenvs

Yes, coverage is enabled in tox too. This is configured in the tox.ini. I've chosen to enable coverage only on python3.4 for now. In the future we can report coverage on all python versions and then combine them all, but this will require additional work

tomkralidis · 2017-01-10T02:32:21Z

.github/ISSUE_TEMPLATE.md

@@ -6,15 +6,15 @@
 - Python version:
 - pycsw version:
 - source/distribution
- - [ ] git clone


These changes in this file are already committed upstream.

yeah, I really have to clean up the commit history...

tomkralidis · 2017-01-10T02:34:01Z

.travis.yml

-  - pip install -r requirements-dev.txt
-  - pip install -r requirements-standalone.txt
-  - python setup.py -q install
+  - pip install tox


Shouldn't we just include tox in requirements-dev.txt and pip install -r requirements-dev.txt?

That is not possible. The way in which tox works is that it is responsible for creating separate virtualenvs for each python version (and other combinations of dependencies, like one virtualenv for python3.4 with sqlite, another virtualenv for python3.4 with postgresql, etc). As such, the tox process is the one which is going to be calling pip to install the dependencies inside each virtualenv after it has been created. This happens when we run tox on the command-line. This means that tox must be installed system-wide and a priori of installing pycsw rquirements.

In short, its tox's job to install the requirements, so tox has to be already installed and there is no benefit in including it in requirements.txt.

tomkralidis · 2017-01-10T02:34:32Z

docs/conf.py

@@ -48,6 +48,9 @@

 # -- General configuration -----------------------------------------------------

+# locale 


These changes in this file are already committed upstream.

tomkralidis · 2017-01-10T02:53:57Z

tests/functionaltests/suites/harvesting/data/.gitignore

@@ -0,0 +1,30 @@
+# This file exists to force git to include this directory in the code


Should this file be called .gitkeep to be consistent with other .gitkeep files in the repo?

I'd actually prefer if the other (what other .gitkeep files are there elsewhere in the codebase if not in the tests directories) files would be renamed to .gitignore and would have a textual description of their purpose, as I've done here.

It took me a while to understand what this .gitkeep thing was. It is not a git feature and is therefore not documented in the git docs. It is merely a convention that these files are called .gitkeep. IMHO using .gitignore with a brief explanation of the file's purpose inside is more explicit.

OK, Can you make this consistency change across the codebase?

yes, I can do it :)
Maybe after this has been merged?

sounds good, thanks!

tomkralidis · 2017-01-10T02:54:10Z

tests/functionaltests/suites/harvesting/get/requests.txt

@@ -0,0 +1,5 @@
+Exception-Harvest-missing-resourcetype,service=CSW&version=2.0.2&request=Harvest


I think this file should be git mv'd as opposed to added/deleted? This will help preserve history.

tomkralidis · 2017-01-10T02:54:22Z

tests/functionaltests/suites/manager/data/.gitignore

@@ -0,0 +1,30 @@
+# This file exists to force git to include this directory in the code


I think this file should be git mv'd as opposed to added/deleted? This will help preserve history.

tomkralidis · 2017-01-10T02:54:35Z

tests/functionaltests/suites/oaipmh/get/requests.txt

@@ -0,0 +1,20 @@
+empty,mode=oaipmh


I think this file should be git mv'd as opposed to added/deleted? This will help preserve history.

tomkralidis · 2017-01-10T02:54:44Z

tests/functionaltests/suites/sru/get/requests.txt

@@ -0,0 +1,5 @@
+explain,mode=sru


I think this file should be git mv'd as opposed to added/deleted? This will help preserve history.

ricardogsilva · 2017-01-11T16:44:28Z

@tomkralidis

The difference between tests/conftest.py and tests/functionaltests/conftest.py is that the first file houses general configuration for all tests while the second file has configuration specific for the tests under the functionaltests/ directory. The rationale for this is that I'd like to have a tests/unittests directory too in the near future (i.e. after this PR is merged) where some smaller unit tests will reside. These will not need the configuration used in tests/functionaltests/conftest.py.
You're right I really have to clean up the commit history. I have to spend some time improving my git skills for this. I agree that stuff that has already been commited does not belong here. I'll also squash all this stuff into a smaller number of commits too.
The test runner workflow for these tests is in tests/functionaltests/conftest.py. It is where the suites are discovered and set up for testing. I've tried to provide an explanation in the docstring of the pytest_generate_tests() function here. Perhaps it is not the most visible place. I'll add something to the docs/testing.rst file too in order to make it more explicit then.

ricardogsilva · 2017-01-13T11:24:40Z

@tomkralidis

I'm having a difficult time figuring out how to properly clean up the git commit history and I'm kind of scared to mess thing up. Can you help me out with this? Thanks a lot in advance

ricardogsilva · 2017-01-16T00:23:10Z

@tomkralidis @kalxas

It seems I've found a way to clean up the git history and commits by checking out the master and then diff'ing my changes onto it (with some more tweaking on top). It's still WIP, but I-m confident it will work. I'll report when I have it working. Afterwards I'll be closing this PR and opening another with the cleaned stuff.

ricardogsilva · 2017-01-23T21:24:39Z

I am closing this PR for now. The same functionality is available in a newer PR, with a cleaner commit history

ricardogsilva added 30 commits May 25, 2016 02:15

moving tests to pytest

84acf66

working on POST tests

95cee97

adding ports for each suite

4aca3bd

refactoring tests to use py.test

07d644a

Removed pycsw placeholder text and unnecessary config KVP param

0ecc653

These are not needed anymore on GET requests

Adding back test logic

5f5c272

adding database stuff to test configuration

6d78dbb

refactoring tests to use py.test

820d390

Merge branch 'master' into issue#428

625db7f

added pycharm's project files to gitignore

28b8127

refactored the implementation

e481836

Removed a lot of boiler plate code, simplifying things

made new copies of the test data files under tests/functionaltests

3d640da

Reorganized the location of test data files. This keeps things a bit cleaner

moving tests to pytest

3e4ae83

working on POST tests

207b9e4

adding ports for each suite

a96c146

refactoring tests to use py.test

9c93bbd

Removed pycsw placeholder text and unnecessary config KVP param

fb02cb2

These are not needed anymore on GET requests

Adding back test logic

d3e8c73

adding database stuff to test configuration

45740e0

refactoring tests to use py.test

825c417

added pycharm's project files to gitignore

963c91b

refactored the implementation

1f7dbd5

Removed a lot of boiler plate code, simplifying things

made new copies of the test data files under tests/functionaltests

daa52ba

Reorganized the location of test data files. This keeps things a bit cleaner

changed test parameters

852347d

Merge branch 'issue#428' of https://github.com/ricardogsilva/pycsw in…

5909cc5

…to issue#428

Merge branch 'master' into issue#428

22d20d8

updated failing harvesting tests like in master

6ba89af

Added docstrings to all functions

094f602

refactored gen_html to account for the change in location for test su…

8fa84a1

…ites

Added support for postgresql in tests

ed8b384

ricardogsilva added 22 commits December 23, 2016 18:13

Changed some tests in order to have uniform sorting

cbcafb0

Added support for tox

052b5ec

Using unicode strings instead of bytes in test function

a102698

fixed version for pytest

ca8f4bf

improved docs on testing

a16a907

Removed old testing tasks from pavement file

306ba20

moving tests to pytest

a1c0f9c

working on POST tests

0de23c0

refactoring tests to use py.test

dc8853d

Adding back test logic

151cb47

refactoring tests to use py.test

cc5a433

refactored the implementation

76d29e2

Removed a lot of boiler plate code, simplifying things

Updated several expected test results

bc30518

These changes have to be manual, since this branch changed the location of the expected test files

deleted old suites and expected diectories

7eb3367

deleted older run_tests.py test runner

cda6ab6

Improving docstrings

25c708e

Added python2.6 to tox

686d2a4

Requirements-dev now includes the other requirements files

393f162

This simplifies installation for develoment and testing purposes by requiring a single pip install -r requirements-dev.txt

added gitignore files to relevant test dirs

4acbcdd

changed travis settings in order to use tox

32299b3

Merge branch 'issue#428' of https://github.com/ricardogsilva/pycsw in…

c904a06

…to issue#428

re-added travis notifications

d2bdf23

tomkralidis reviewed Jan 10, 2017

View reviewed changes

ricardogsilva mentioned this pull request Jan 23, 2017

RFC8 - Move to pytest testing tool #495

Merged

21 tasks

ricardogsilva closed this Jan 23, 2017

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[WIP] RFC8 - Move to pytest testing tool [Issue#428] #492

[WIP] RFC8 - Move to pytest testing tool [Issue#428] #492

ricardogsilva commented Dec 14, 2016 •

edited

Loading

tomkralidis commented Jan 10, 2017

tomkralidis Jan 10, 2017

ricardogsilva Jan 11, 2017

tomkralidis Jan 10, 2017

ricardogsilva Jan 11, 2017

tomkralidis Jan 10, 2017

ricardogsilva Jan 11, 2017

tomkralidis Jan 10, 2017

ricardogsilva Jan 11, 2017

tomkralidis Jan 10, 2017

tomkralidis Jan 10, 2017

ricardogsilva Jan 11, 2017 •

edited

Loading

tomkralidis Jan 11, 2017

ricardogsilva Jan 12, 2017

tomkralidis Jan 13, 2017

tomkralidis Jan 10, 2017

tomkralidis Jan 10, 2017

tomkralidis Jan 10, 2017

tomkralidis Jan 10, 2017

ricardogsilva commented Jan 11, 2017

ricardogsilva commented Jan 13, 2017

ricardogsilva commented Jan 16, 2017

ricardogsilva commented Jan 23, 2017

		@@ -48,6 +48,9 @@

		# -- General configuration -----------------------------------------------------

		# locale

		@@ -0,0 +1,30 @@
		# This file exists to force git to include this directory in the code

		@@ -0,0 +1,5 @@
		Exception-Harvest-missing-resourcetype,service=CSW&version=2.0.2&request=Harvest

[WIP] RFC8 - Move to pytest testing tool [Issue#428] #492

[WIP] RFC8 - Move to pytest testing tool [Issue#428] #492

Conversation

ricardogsilva commented Dec 14, 2016 • edited Loading

Overview

Related Issue / Discussion

Additional Information

Contributions and Licensing

tomkralidis commented Jan 10, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ricardogsilva Jan 11, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ricardogsilva commented Jan 11, 2017

ricardogsilva commented Jan 13, 2017

ricardogsilva commented Jan 16, 2017

ricardogsilva commented Jan 23, 2017

ricardogsilva commented Dec 14, 2016 •

edited

Loading

ricardogsilva Jan 11, 2017 •

edited

Loading