Separate unit tests from integration tests #130

r24mille · 2017-08-19T19:31:06Z

Currently, there are many tests in the test directory that call external resources, rather than mocking requests/responses. Some also require environment variables (e.g. setting EIA_KEY in test_eia.py). This is causing issues in Travis and we're losing the value from existing continuous integration work that has been done.

I propose the following changes:

The codebase seems to follow a tests outside application code directory structure, which is fine. In line with this Stack Overflow thread I'd further organize directories into:

pyiso
  + --- tests
  |       + --- unit
  |       |         | test_base.py
  |       |         | test_client_one.py
  |       |         | test_client_two.py
  |       |
  |       + --- integration 
  |                 | integration_test_client_one.py
  |                 | integration_test_client_two.py
  |       
  + --- pyiso
          | base.py
          | client_one.py
          | client_two.py

Fix travis.yml so that the build server only runs unit tests when pull requests are opened. I believe this is just changing the [nosetests](https://nose.readthedocs.io/en/latest/usage.html) script on line 26 to something like:

script: nosetests -w ./tests/unit

This only leaves one question: What automated process should run the integration tests? Should the maintainers just be responsible for running integration tests before formalizing a release? Should the integration tests just be there for developers to run out-of-band so they can be confident in their code before opening pull requests?

The text was updated successfully, but these errors were encountered:

This is the minimal change necessary to fix the BPA unit tests.

…ory. This keeps the tests cleaner and more readable.

It was not used anywhere in the unit tests and it is not Excel content as the file extension suggests.

…directory.

It makes a network request for the latest content, so it should go in the /pyiso/tests/integration directory.

Moved and renamed files to follow the proposed structure of issue WattTime#130.

These changes are inline with the issue WattTime#130 proposal.

…n tests. Unit and integration tests are now segregated into separate locations. With the new location for integration tests, I've added the NBPower integration tests which I'd previously held back from PR WattTime#125.

…ests. Unit and integration tests are now segregated into separate locations. With the new location for integration tests, I've added the SaskClient integration tests which I'd previously held back from PR WattTime#126.

r24mille · 2017-08-30T05:01:51Z

Hmm... it seems #131 auto-closed this issue when it got merged in (?) Anyway, it may be useful to split the work up into multiple issues or re-open this issue. I don't have permissions, since I'm not a collaborator.

ajdonnison · 2017-08-30T05:03:06Z

Reopening as it needs more work.

@skiptest

Now that there's a place for intergration tests, I've added several for the `IESOClient`. Several tests are annotated with @skiptest. I'm trying to find a better way to address the gap in the data (which is valid) but still check something meaningful with the test.

I would like to upgrade further, but don't think it's prudent until we get unit tests and integration tests sorted in WattTime#130. Then we could probably bump the version up further. For now, I think upgrading the dependency from 3.6.1 to 3.6.4 is fairly safe (unit tests pass) and the version change bring in stabilization fixes only. See http://lxml.de/3.6/changes-3.6.4.html for details.

This should create a coverage report which can be sent via the coveralls build steps in travis.yml

This commit splits test_ercot.py into unit tests, which have no external dependencies, and integration tests, which test pyiso's integration with external systems (e.g. ERCOT's web servers).

This makes the unit tests easier to read and work with.

#130 Cleaned up test_nvenergy.py

…ent.

…nit tests.

This allows the class to be initialized and unit tests which don't make web requests to pass.

Having the fixtures_base_path outside the function wasn't necessary and was less clear.

Also I renamed some of the fixtures that were being used for expected responses to be clearer.

Historical data from 2015 no longer exists. I changed the date to 2017 to kick the problem down the road for two years ;)

I also tweaked test_auth(...) so that it mocks the response from the ENSOe server similar to the other mocked auth calls.

* #130 Fixed test_dt_index(...) in test_caiso.py The DatetimeIndex class was moved to from pandas.tseries.index to pandas.core.indexes.datetimes with the release of 0.20.1: http://pandas.pydata.org/pandas-docs/stable/whatsnew.html#reorganization-of-the-library-privacy-changes This means that our requirement of pandas >= 0.18 could cause test_dt_index(...) to fail in test_caiso.py sporadically if users fulfill the requirement with pandas >= 0.20.1. It felt foolish to increment one of pyiso's core requirements for the sake of a unit test. So I changed the test to check for the type(...) class name rather than the type(...) exactly. * #130 Increased default request timeout to 30 seconds. I've increased the default request timeout to 30 seconds because test_get_lmp_dataframe_fifteen(...) from test_caiso.py was sporadically failing for me, not because the request was bad but because processing time was right at the threshold of the request processing timeout. Increasing the default timeout from 20 seconds to 30 seconds seemed reasonable, since 20 seconds is rather short. However, any further increases to the timeout due to large .zip requests from CAISO should possibly be isolated to the CAISOClient. * Removed TIMEOUT_SECONDS constant from BaseClient. The field was not being referenced anywhere. It only caused confusion given that __init__ has a timeout_seconds parameter with a default value. * #130 Split up integration tests and unit tests out of test_caiso.py They now follow the directory conventions being established with #130. * #130 Increased LMP request timeout for CAISO to 45 seconds. * #130 Pulled static content out of test_caiso.py into fixtures directory. * #130 Rely on setUp(...) to call client_factory('CAISO') before each test. Unit and integration tests were each creating their own CAISOClient at that start of each test. That type of setup can be left to the setUp(...) method. * #130 Fixed BaseClient test timeout. All we should be checking is that the request timeout is assignable. Checking its default value with a unit test is just annoying. * #130 Fixed byte encoding issues in mocked response. This was causing errors in Python 3.

) * #130 Split up unit tests and integration tests up from EIAClient. * #130 Moved EIA docstring from unit test to integration test. * #130 Added static 'EIA_KEY' environment variable for mocked unit tests. * Edited EIAClient's set_url(...) method for clarity.

) * #130 Added fake ISONE environment variables for unit tests. This allows the class to be initialized and unit tests which don't make web requests to pass. * #130 Minor cleanup for read_fixtures(...) function. Having the fixtures_base_path outside the function wasn't necessary and was less clear. * #130 Moved test fixture files into isone subdirectory. * #130 Fixed mocked requests in ISONE unit tests. Also I renamed some of the fixtures that were being used for expected responses to be clearer. * #130 Split ISONE integration tests out of unit tests. * #130 Fixed up the LMP integration tests for ISONE. Historical data from 2015 no longer exists. I changed the date to 2017 to kick the problem down the road for two years ;) * #130 Fixed up mocked response encoding issue for Python 3.

* #130 Moved test_eu.py into unit test directory. * #130 Added test ENTSOe username/password environment variables. I also tweaked test_auth(...) so that it mocks the response from the ENSOe server similar to the other mocked auth calls.

The TestNYISOTrade class in test_trade.py was mocking static responses in an integration test. With improvements made to mocked responses in WattTime#183, testing static responses are handled by unit tests. This commit changes the integration tests in TestNYISOTrade to use current times, so that the tests dynamically test our ability to integration with the external NYISO server (in line with WattTime#130).

…NYISOGenMix The TestNYISOGenMix class in test_genmix.py had comment-out references to static responses. They're now properly dynamic (furthered by WattTime#130) so the old comments are no longer relevant. This commit removes the comments, deletes the now-unused files the comments were about, and corrected some minor errors in the TestNYISOGenMix integration tests.

…CAISOLoad The TestCAISOLoad class in test_load.py had comment-out references to static responses. The tests are now now dynamic (furthered by WattTime#130) so the old comments are no longer relevant. This commit removes the comments and deletes the now unused files the comments were about.

…tNEVPLoad The TestNEVPLoad class in test_load.py had commented-out references to static responses. The tests are now now dynamic (furthered by WattTime#130) so the old comments are no longer relevant. This commit removes the comments and deletes the now unused file the comments were about.

…tSPPCLoad The TestSPPCLoad class in test_load.py had commented-out references to static responses. The tests are now now dynamic (furthered by WattTime#130) so the old comments are no longer relevant. This commit removes the comments and deletes the now unused file the comments were about.

…tGenerationTask The TestGenerationTask class in test_tasks.py had commented-out references to static responses. The tests are now now dynamic (furthered by WattTime#130) so the old comments are no longer relevant. This commit removes the comments and deletes the now unused file the comments were about.

…tSVERIGenMix The TestSVERIGenMix class in test_genmix.py had commented-out references to static responses. The tests are now now dynamic (furthered by WattTime#130) so the old comments are no longer relevant. This commit removes the comments and deletes the now unused files the comments were about.

r24mille added a commit to r24mille/pyiso that referenced this issue Aug 20, 2017

WattTime#130 Fixing broken BPA unit test and moved static response.

58c4887

This is the minimal change necessary to fix the BPA unit tests.

r24mille added a commit to r24mille/pyiso that referenced this issue Aug 20, 2017

WattTime#130 Moved tab-separated wind data to file in fixtures direct…

398c55e

…ory. This keeps the tests cleaner and more readable.

r24mille added a commit to r24mille/pyiso that referenced this issue Aug 20, 2017

WattTime#130 Removed wind_xls variable from BPA unit test.

09e460c

It was not used anywhere in the unit tests and it is not Excel content as the file extension suggests.

r24mille added a commit to r24mille/pyiso that referenced this issue Aug 20, 2017

WattTime#130 PEP 8 line formatting cleanup in test_bpa.py

9b82528

r24mille mentioned this issue Aug 20, 2017

Fixes to BPA unit tests (i.e. test_bpa.py) #131

Merged

r24mille added a commit to r24mille/pyiso that referenced this issue Aug 22, 2017

WattTime#130 Load WindGenTotalLoadYTD_2014.xls regardless of working …

645583c

…directory.

r24mille added a commit to r24mille/pyiso that referenced this issue Aug 22, 2017

WattTime#130 Split out one integration test from test_bpa.py

dae9da0

It makes a network request for the latest content, so it should go in the /pyiso/tests/integration directory.

r24mille added a commit to r24mille/pyiso that referenced this issue Aug 23, 2017

WattTime#130 Moved test_ieso.py and renamed fixtures.

33889b7

Moved and renamed files to follow the proposed structure of issue WattTime#130.

r24mille added a commit to r24mille/pyiso that referenced this issue Aug 23, 2017

WattTime#130 Missed a file with my previous commit.

5ea0c15

r24mille added a commit to r24mille/pyiso that referenced this issue Aug 24, 2017

WattTime#130 Restructured AESO unit tests and added an integration test.

8cf630d

These changes are inline with the issue WattTime#130 proposal.

ajdonnison closed this as completed in #131 Aug 30, 2017

ajdonnison reopened this Aug 30, 2017

r24mille mentioned this issue Sep 3, 2017

#108 Upgraded lxml dependency from 3.6.1 to 3.6.4. #135

Merged

r24mille added a commit to r24mille/pyiso that referenced this issue Sep 4, 2017

WattTime#130 Add usage of coverage plugin to nosetests call from Travis.

310ece7

This should create a coverage report which can be sent via the coveralls build steps in travis.yml

r24mille mentioned this issue Sep 4, 2017

#130 Add usage of coverage plugin to nosetests call from Travis. #136

Merged

r24mille added a commit to r24mille/pyiso that referenced this issue Sep 4, 2017

WattTime#130 Fixed LMP node count in integration_test_ercot.py

cb111fb

r24mille added a commit to r24mille/pyiso that referenced this issue Sep 5, 2017

WattTime#130 Changed coveralls badge from PNG to SVG.

85d5fe0

r24mille added a commit to r24mille/pyiso that referenced this issue Sep 5, 2017

WattTime#130 Moved test_miso.py to the /pyiso/tests/unit folder.

57567f1

r24mille added a commit to r24mille/pyiso that referenced this issue Sep 6, 2017

WattTime#130 Moved test_nvenergy.py to the /pyiso/tests/unit folder.

c082e19

r24mille added a commit to r24mille/pyiso that referenced this issue Sep 6, 2017

WattTime#130 Extracted response string variables as static files

eea1bf6

This makes the unit tests easier to read and work with.

r24mille added a commit to r24mille/pyiso that referenced this issue Sep 6, 2017

WattTime#130 Clean up imports in test_nvenergy.py

9cddafd

r24mille added a commit to r24mille/pyiso that referenced this issue Sep 6, 2017

WattTime#130 Fixed unicode prefix compatibility issue with Python 3.

f819821

ajdonnison added a commit that referenced this issue Sep 7, 2017

Merge pull request #140 from r24mille/130_clean_up_nvenergy_tests

82a589d

#130 Cleaned up test_nvenergy.py

r24mille added a commit to r24mille/pyiso that referenced this issue Sep 20, 2017

WattTime#130 Split up unit tests and integration tests up from EIACli…

a15a090

…ent.

r24mille added a commit to r24mille/pyiso that referenced this issue Sep 21, 2017

WattTime#130 Moved EIA docstring from unit test to integration test.

481ab92

r24mille added a commit to r24mille/pyiso that referenced this issue Sep 21, 2017

WattTime#130 Added static 'EIA_KEY' environment variable for mocked u…

86cb205

…nit tests.

r24mille added a commit to r24mille/pyiso that referenced this issue Sep 22, 2017

WattTime#130 Added fake ISONE environment variables for unit tests.

a1ce1fa

This allows the class to be initialized and unit tests which don't make web requests to pass.

r24mille added a commit to r24mille/pyiso that referenced this issue Sep 22, 2017

WattTime#130 Minor cleanup for read_fixtures(...) function.

8fb36b2

Having the fixtures_base_path outside the function wasn't necessary and was less clear.

r24mille added a commit to r24mille/pyiso that referenced this issue Sep 22, 2017

WattTime#130 Moved test fixture files into isone subdirectory.

160d4fa

r24mille added a commit to r24mille/pyiso that referenced this issue Sep 22, 2017

WattTime#130 Fixed mocked requests in ISONE unit tests.

eed6238

Also I renamed some of the fixtures that were being used for expected responses to be clearer.

r24mille added a commit to r24mille/pyiso that referenced this issue Sep 22, 2017

WattTime#130 Split ISONE integration tests out of unit tests.

343d0f3

r24mille added a commit to r24mille/pyiso that referenced this issue Sep 22, 2017

WattTime#130 Fixed up the LMP integration tests for ISONE.

99dc842

Historical data from 2015 no longer exists. I changed the date to 2017 to kick the problem down the road for two years ;)

r24mille added a commit to r24mille/pyiso that referenced this issue Sep 22, 2017

WattTime#130 Fixed up mocked response encoding issue for Python 3.

ac4a570

r24mille added a commit to r24mille/pyiso that referenced this issue Sep 23, 2017

WattTime#130 Moved test_eu.py into unit test directory.

bda1217

r24mille added a commit to r24mille/pyiso that referenced this issue Sep 23, 2017

WattTime#130 Added test ENTSOe username/password environment variables.

c73e752

I also tweaked test_auth(...) so that it mocks the response from the ENSOe server similar to the other mocked auth calls.

r24mille added a commit to r24mille/pyiso that referenced this issue Sep 23, 2017

WattTime#130 Moved test_tasks.py into integratin test directory.

11ffa77

This was referenced Sep 23, 2017

Moved test_tasks.py into integratin test directory (#130) #153

Merged

Port functionality from a number of integration tests back to their unit tests using mocked responses #154

Closed

ajdonnison closed this as completed in #153 Sep 25, 2017

ajdonnison pushed a commit that referenced this issue Sep 25, 2017

#130 Moved test_tasks.py into integratin test directory. (#153)

756e304

This was referenced Dec 7, 2017

Remove responses directory #185

Merged

Loosen version restrictions of packages required for development #187

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Separate unit tests from integration tests #130

Separate unit tests from integration tests #130

r24mille commented Aug 19, 2017

r24mille commented Aug 30, 2017

ajdonnison commented Aug 30, 2017

Separate unit tests from integration tests #130

Separate unit tests from integration tests #130

Comments

r24mille commented Aug 19, 2017

r24mille commented Aug 30, 2017

ajdonnison commented Aug 30, 2017