Drop unused unittest components from test helpers and replace tests with pytest-based ones #67

webknjaz · 2017-12-02T22:04:48Z

What kind of change does this PR introduce? (Bug fix, feature, docs update, ...)
Cleanup
What is the related issue number (starting with #)
[TODO] Get rid of unittest.TestCase #42
What is the current behavior? (You can also link to an open issue here)
TerseTestResult, TerseTestRunner and ReloadingTestLoader exist.
What is the new behavior (if this is a feature change)?
Drop above.
Other information:

codecov · 2017-12-14T19:36:18Z

Codecov Report

Merging #67 into master will decrease coverage by 43.99%.
The diff coverage is 12.3%.

@@            Coverage Diff            @@
##           master      #67     +/-   ##
=========================================
- Coverage   62.33%   18.34%    -44%     
=========================================
  Files          15       15             
  Lines        2963     2748    -215     
=========================================
- Hits         1847      504   -1343     
- Misses       1116     2244   +1128

jaraco · 2017-12-16T15:23:26Z

Although it's found in the 'tests' package, the webtest module in particular is part of the public interface of cheroot and cherrypy.

I did a quick search for ReloadingTestLoader, which turns up quite a few results. If there are users relying on this functionality, this change will need to describe what the recommendation is for those users. Maybe the recommendation is they adapt their tests to a new model--a fairly harsh ask. Or maybe we release this functionality in a separate package. I can't decide whether it's worth an intermediate release deprecating this functionality first.

In any case, I'm largely in favor of this change as long as we can make it without causing undue disruption.

jaraco · 2017-12-16T15:25:36Z

cheroot/test/conftest.py

+
+
+config = {
+    'bind_addr': ('127.0.0.1', 54583),


Here I'd like to see support for IPv6 and I'd like not to see hard-coded port numbers... which could conflict with other applications or other instances of the tests running. I suggest using portend.find_available_local_port or binding to the ephemeral port 0 and then resolving the bind port for the client.

Yeah, I was thinking about it and prefer ephemeral port 0 approach as it's safer to defer port allocation to the OS and portend dependency is not even needed in this case.

I was trying to decouple existing unittest.TestCase based class into pieces, thus I didn't really take any steps to optimize or add new behavior at this stage. I didn't even replace the test suite module, but added new one because of this experimenting process.

webknjaz · 2017-12-17T22:29:37Z

I did a quick search for ReloadingTestLoader, which turns up quite a few results.

I followed the search and found out that all (or nearly all) of the results are just copies of cherrypy repo.

webknjaz · 2017-12-17T22:39:09Z

@jaraco

Do you think anyone uses/needs WebCase._handlewebError?

If yes, do you have any ideas on how to reimplement this using pytest?

jaraco · 2017-12-17T23:42:50Z

I don't know about _handlewebError, but I would recommend that in a pytest scenario, one shouldn't use _handlewebError, and the tests should just make assertions and let the test runner handle interactivity (such as through --pdb). Given that it's got the underscore prefix, I'd say it's sane to deprecate/remove it.

webknjaz · 2017-12-22T12:46:53Z

@jaraco well, it's kinda public, because depending on the environment variable (which I would consider to be a public interface) it intercepts assertion failures and provides an interactive interface for accessing/printing HTTP request/response values and suppressing/raising a real failures during test runs.

On the other hand I don't really see any need for this interface.

Also, I'd like to provide some new public pytest-oriented fixtures/helpers in cheroot.testing module rather that mixing them into cheroot.test.webtest

Ref #42

It's not used in cheroot and is available in cherrypy copy of helper

cherrypy impl has it's own anyway

Eliminated code duplication

While this workaround would work, I'd prefer keeping both requested and actual bind addrs within the server object, which might need bits of redesign.

webknjaz · 2018-01-02T11:37:22Z

@jaraco could you plz share your opinion regarding public helpers for end-users (aka cheroot.testing)?

jaraco · 2018-01-03T16:34:38Z

I agree, creating an officially-supported testing module and separating it from a private tests package makes a lot of sense. Of course, doing so would be a backward-incompatible change, but we can manage that with deprecations and compatibility shims and iterative releases.

Using ``--testmon`` breaks tests under Windows. Removing it temporarily. Ref: tarpas/pytest-testmon#81 Ref: https://ci.appveyor.com/project/CherryPy/cheroot/build/166/job/nlioynp1ls4xf3em#L215

This reverts commit 72ab1ea. Ref tarpas/pytest-testmon#84 Ref tarpas/pytest-testmon#14

jaraco

Overall, this is great. I see lots to love in this change. I could have added a few nitpicks, but I decided to refrain so we can get this out and iterate on it. I did add a couple of comments I think are worthy of addressing in this PR, but nothing glaring. And tests work for me.

I haven't tested it against CherryPy. If you can confirm CherryPy tests pass with these changes, I'm prepared to rubberstamp!

jaraco · 2018-01-06T16:04:04Z

cheroot/server.py

@@ -1600,6 +1603,7 @@ def bind(self, family, type, proto=0):
                pass

        self.socket.bind(self.bind_addr)
+        self.bind_addr = self.socket.getsockname()[:2]  # TODO: keep separate


This change seems dangerous to me. You're overwriting the state on the object... so where one could previously access self.bind_addr to determine what address was attempted to be bound, now one gets instead the addr than was bound. Plus, there's a TODO - not sure why not just do:

@property def bound_addr(self): return self.socket.getsockname()[:2]

or even better, just let the caller call server.socket.getsockname() directly.

Yeah, it's just a dirty hack I did while refactoring tests. I wanted to postpone its resolution and to not dig into it while focusing on tests rewrite.
I don't want to mix this into the current PR, so I've logged this in #70

jaraco · 2018-01-06T16:06:30Z

cheroot/server.py

-    def _get_interrupt(self):
+    @property
+    def interrupt(self):
+        """Return interrupt Exception instance."""


This docstring is the one that will be present on the property, whereas previously the docstring would have provided indication about setting the exception. The docstring should be copied from the setter and the setter should probably have no meaningful docstring, maybe something like "setter for interrupt".

Fair enough

But still the docstring should describe the property/getter

I'll try to rephrase that

jaraco · 2018-01-06T16:21:58Z

cheroot/test/webtest.py

+                'We are moving to pytest-based testing and thus deprecating '
+                'WEBTEST_INTERACTIVE environment variable support. '
+                'Interactive test failure interceptor is going to be removed '
+                'in Cheroot v7.0.0',  # TODO: decide whether it's v7 or v8


I've learned that it's difficult to predict when a feature will be dropped unless you have a project with long iterations and scheduled releases, neither of which apply to cheroot. I would simply say "WEBTEST_INTERACTIVE is deprecated." ...and in a future, backward-incompatible release, which we will determine when we get to it, we drop support.

@jaraco

As advised by @jaraco

@jaraco

Loosely addresses comment in PR by @jaraco. To be improved.

webknjaz · 2018-01-06T19:38:03Z

@jaraco thanks for the review! 💫 🚀 🤖

webknjaz · 2018-01-06T21:24:58Z

Confirmed that it's compatible with CherryPy

webknjaz self-assigned this Dec 2, 2017

webknjaz requested a review from jaraco December 2, 2017 22:04

webknjaz mentioned this pull request Dec 2, 2017

[TODO] Get rid of unittest.TestCase #42

Closed

webknjaz force-pushed the feature/drop-unused-unittest-components branch from 4eba97d to 5ff6a72 Compare December 14, 2017 19:36

webknjaz force-pushed the feature/drop-unused-unittest-components branch from 5ff6a72 to 5081f9c Compare December 14, 2017 19:41

webknjaz changed the title ~~Drop unused unittest components from test helpers~~ WIP: Drop unused unittest components from test helpers Dec 14, 2017

jaraco reviewed Dec 16, 2017

View reviewed changes

webknjaz added 8 commits December 22, 2017 15:06

Drop unused unittest components from test helpers

6bfef14

Ref #42

Drop cheroot.test.helper.CherootWebCase.base()

9bc3b0e

It's not used in cheroot and is available in cherrypy copy of helper

Drop exit method from CherootWebCase and WebCase

a8d74bf

cherrypy impl has it's own anyway

Cleanup CherootWebCase.skip() from test dir

97f39d7

Drop cheroot.test.helper.CherootProcess

0d971bd

Add pure pytest implementation of test_core suite

ef46c57

Drop _get_http_connection helper function

e79e9ef

Drop yield from to keep compat w/ 2.7

c7825a2

webknjaz force-pushed the feature/drop-unused-unittest-components branch from 436a16a to c7825a2 Compare December 22, 2017 13:07

webknjaz added 8 commits December 23, 2017 12:17

Deprecate interactive test interceptor in WebCase

09723c2

Refactor socket info lookup in server.HTTPServer

82cb5f6

Eliminated code duplication

Hotfix saving actual bound addr for ephemeral port

81abec0

While this workaround would work, I'd prefer keeping both requested and actual bind addrs within the server object, which might need bits of redesign.

Make HTTP server fixture bind to an ephemeral port

14cfc89

Listen to IPv6 interface during tests if available

342fee8

Catch proper error when unable to bind to IPv6

d84550e

Print test box interfaces info in Travis on fail

631be3f

Print sysctl on fail in Travis

54e5a6c

webknjaz added 10 commits December 29, 2017 16:21

Drop bogus comment delimiter

1ae2fed

Refactor loops in test_conn test suite

62abfbb

Refactor test client, provided by fixture

3845561

Factor test client definition out of server_client

63ab280

Refactor server_client fixture

f4a1083

Turn server_client into two fixtures

81a6fec

Simplify test client creation

378fdef

Drop unrelated CherryPy tests

d551c2e

Redistribute fixtures btw testing & test.conftest

a8d51a4

Unignore modules in pep257 check

d9cd1c5

webknjaz changed the title ~~WIP: Drop unused unittest components from test helpers~~ WIP: Drop unused unittest components from test helpers and replace tests with pytest-based ones Dec 29, 2017

Refer issue related to test_Chunked_Encoding

c44fa71

webknjaz requested review from jaraco and a team December 29, 2017 20:00

Integrate pytest-testmon and pytest-watch

a7f3cd7

webknjaz added 3 commits January 3, 2018 20:05

Use testmon in watch toxenv only

72ab1ea

Using ``--testmon`` breaks tests under Windows. Removing it temporarily. Ref: tarpas/pytest-testmon#81 Ref: https://ci.appveyor.com/project/CherryPy/cheroot/build/166/job/nlioynp1ls4xf3em#L215

Revert "Use testmon in watch toxenv only"

de0de59

This reverts commit 72ab1ea. Ref tarpas/pytest-testmon#84 Ref tarpas/pytest-testmon#14

Pin pytest-testmon to v0.9.7 or higher

b612b50

jaraco reviewed Jan 6, 2018

View reviewed changes

jaraco approved these changes Jan 6, 2018

View reviewed changes

webknjaz added 2 commits January 6, 2018 21:05

Rephrase deprecation msg for WEBTEST_INTERACTIVE

bc3060e

As advised by @jaraco

Rephrase server.HTTPServer.interrupt docstring

0d0f18e

Loosely addresses comment in PR by @jaraco. To be improved.

webknjaz merged commit 0d0f18e into master Jan 6, 2018

webknjaz changed the title ~~WIP: Drop unused unittest components from test helpers and replace tests with pytest-based ones~~ Drop unused unittest components from test helpers and replace tests with pytest-based ones Jan 7, 2018

webknjaz mentioned this pull request Feb 3, 2018

Require client certificates #52

Closed

webknjaz deleted the feature/drop-unused-unittest-components branch December 7, 2020 17:54

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Drop unused unittest components from test helpers and replace tests with pytest-based ones #67

Drop unused unittest components from test helpers and replace tests with pytest-based ones #67

webknjaz commented Dec 2, 2017

codecov bot commented Dec 14, 2017 •

edited

Loading

jaraco commented Dec 16, 2017

jaraco Dec 16, 2017

webknjaz Dec 17, 2017

webknjaz commented Dec 17, 2017

webknjaz commented Dec 17, 2017

jaraco commented Dec 17, 2017

webknjaz commented Dec 22, 2017

webknjaz commented Jan 2, 2018

jaraco commented Jan 3, 2018

jaraco left a comment

jaraco Jan 6, 2018 •

edited by webknjaz

Loading

webknjaz Jan 6, 2018

jaraco Jan 6, 2018

webknjaz Jan 6, 2018

webknjaz Jan 6, 2018

webknjaz Jan 6, 2018

jaraco Jan 6, 2018

webknjaz commented Jan 6, 2018

webknjaz commented Jan 6, 2018

Drop unused unittest components from test helpers and replace tests with pytest-based ones #67

Drop unused unittest components from test helpers and replace tests with pytest-based ones #67

Conversation

webknjaz commented Dec 2, 2017

codecov bot commented Dec 14, 2017 • edited Loading

Codecov Report

jaraco commented Dec 16, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

webknjaz commented Dec 17, 2017

webknjaz commented Dec 17, 2017

jaraco commented Dec 17, 2017

webknjaz commented Dec 22, 2017

webknjaz commented Jan 2, 2018

jaraco commented Jan 3, 2018

jaraco left a comment

Choose a reason for hiding this comment

jaraco Jan 6, 2018 • edited by webknjaz Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

webknjaz commented Jan 6, 2018

webknjaz commented Jan 6, 2018

codecov bot commented Dec 14, 2017 •

edited

Loading

jaraco Jan 6, 2018 •

edited by webknjaz

Loading