Multiple directories support, norecursedirs, passing args to pytest #11

aldanor · 2015-05-27T16:45:57Z

I believe this PR closes all outstanding issues (#5, #6, #7 and #8). If merged in, this probably calls for a version bump.

The README will probably need to be updated as well..

Options added / things modified:

All positional arguments after -- are now passed as is to py.test executable.
Added support for multiple watched folders (passed as positional arguments).
Allow to ignore sub folders (currently via an exact match only at a first level of each watched folder, but this could be improved quite easily to support arbitrary patterns). Note that this is done at the observer level and not in the event handler because if you have tens of thousands of files in a subfolder like .tox for example, the handler would get completely swamped by that (so a simple pattern matching inside on_any_event is not going to cut it).
py.test subprocess no longer changes directory to a watched folder -- this no longer makes any sense given that any args can be passed to pytest and you can watch multiple folders at once.
py.test subprocess no longer runs with shell=True, to handle quoted arguments properly.
Catch KeyboardInterrupt even if it was caught after the first test run and exit gracefully.
Instead of catching all file events, only react to FileModifiedEvent and FileCreatedEvent. This fixed the problem where duplicate watchdog events would be fired upon a single file modification if using remote NFS mounts on Linux.
Added -p alias to --poll.
The exact py.test command is now printed to stdout upon the first test run.
Added event spooling with 200ms cooldown to work around atomic saves in different editors.
Added --verbose / --quiet options.
Added --no-spool option to disable spooling.
--ext option will prepend extensions with periods if needed.

An example using the new functionality:

ptw -v -p --no-spool --ignore=.tox,.git folder1 folder2 -- tests/*.py -svv

- This fixes the problem with catching duplicate watchdog events related to a single file modification if using remote NFS mounts.

…acefully

…ge cwd. - Positional arguments are now passed directly to py.test executable and can be separated from options via the optional "--" - py.test subprocess no longer changes the current directory - Directories are now specified via --watch option (comma-separated), default is the current directory. - Added "-p" alias for "--poll"

…ames

This fixes the edge case when py.test arguments contained quoted strings with spaces, e.g.: ptw -- -m 'not slow'

This fixes the issue with some editors that do atomic saves via a temporary file and a subsequent move.

joeyespo · 2015-05-28T05:02:37Z

This all looks great! Thank you for putting this all together, @aldanor!

Also, really clear PR message. I really appreciate the writeup. It set good expectations. And the example usage was helpful to envision these changes.

I'm all ready to pull it in. Just address those few comments first. For convenience:

Use [directories] instead of --watch to keep it simple and backwards compatible?
Can you think of any cases where shell=True is an advantage here?
If not, go ahead and remove the redundant shell=False
Are file renames picked up by the current events?
--ignore instead of --nonrecursedirs?

I can also bump the version and update the README after.

joeyespo · 2015-05-28T05:46:05Z

Nevermind about --norecursedirs. I just remembered that came from py.test.

Do you think we should also pass that argument to py.test? Otherwise, py.test might still run tests even though we're not listening for changes in those deeper directories.

aldanor · 2015-05-28T09:27:26Z

Renaming --norecursedirs to --ignore seems reasonable. Passing it to py.test automatically is probably not, as you may want to listen to a restricted set of subfolders while still running the full test suite, I think -- these are two different notions. You could do it the other way around though and use norecursedirs from pytest config file (which is most always present anyway) as a default value for --ignore.

aldanor · 2015-05-28T16:19:45Z

@joeyespo The --watch vs [directory] ([directories]) question is still open -- if you can suggest an unambiguous docopt signature for this I'd be happy to fix it.

…d by --) Also, make --ext a bit smarter by prepending the '.' automatically if need be.

aldanor · 2015-06-10T12:13:52Z

@joeyespo CLI args fixed, you can try testing it now. What's remaining is event spooling, will hack on it now.

aldanor · 2015-06-10T14:20:39Z

@joeyespo Event spooling done:

Changes detected:
    deleted:  pytest_watch/watcher.py
    moved:    pytest_watch/.sublc49.tmp -> pytest_watch/watcher.py

Rerunning pytest command: py.test --import .

I guess I'll have to add -v / -q options for verbose/quiet output, and also an option to opt out of event spooling (should it be enabled by default? the default cooldown is 200ms).

blueyed · 2015-06-10T18:47:20Z

@aldanor
I've noticed that the text of the modified filenames and the command in "Rerunning pytest command" is not visible in my terminal, but can be copy'n'pasted.

blueyed · 2015-06-10T18:49:43Z

Instead of using highlight = lambda arg: Fore.LIGHTWHITE_EX + arg + Fore.CYAN it might be better to just make it bold?

diff --git i/pytest_watch/watcher.py w/pytest_watch/watcher.py
index abe97f7..ebdaa2f 100644
--- i/pytest_watch/watcher.py
+++ w/pytest_watch/watcher.py
@@ -6,7 +6,7 @@

 from .spooler import EventSpooler

-from colorama import Fore
+from colorama import Fore, Style
 from watchdog.observers import Observer
 from watchdog.observers.polling import PollingObserver
 from watchdog.events import (FileSystemEventHandler, FileModifiedEvent,
@@ -60,7 +60,7 @@ def run(self, summary=None):
         if self.auto_clear:
             subprocess.call(CLEAR_COMMAND)
         command = ' '.join(['py.test'] + self.args)
-        highlight = lambda arg: Fore.LIGHTWHITE_EX + arg + Fore.CYAN
+        highlight = lambda arg: Style.BRIGHT + arg + Style.NORMAL
         msg = 'Running pytest command: {}'.format(highlight(command))
         if summary:
             msg = 'Changes detected in files:\n{}\n\nRerunning pytest command: {}'.format(

tarpas · 2015-06-10T19:19:47Z

On Mac OS the spooling seems to cause problems.
tibor$ ptw

Running pytest command: py.test
=============================================== test session starts ================================================
platform darwin -- Python 2.7.8 -- py-1.4.28 -- pytest-2.7.1
rootdir: /Users/tibor/tmonworkspace/testmon/exampleproject, inifile: pytest.ini
plugins: testmon, cache, html
collected 13 items 
  tests/test_a.py sx..  tests/test_ab.py .  tests/test_b.py ..  tests/single/test_s.py ...  tests/single/test_t.py ..  tests_2/test_e.py s
================================= 10 passed, 2 skipped, 1 xfailed in 0.08 seconds ==================================
Process Timer-1:
Traceback (most recent call last):
  File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/multiprocessing/process.py", line 258, in _bootstrap
    self.run()
  File "/Users/tibor/tmonworkspace/pytest-watch/pytest_watch/spooler.py", line 21, in run
    self.function(*self.args, **self.kwargs)
  File "/Users/tibor/tmonworkspace/pytest-watch/pytest_watch/spooler.py", line 39, in process
    self.callback([self.outbox.get() for _ in range(self.outbox.qsize())])
  File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/multiprocessing/queues.py", line 143, in qsize
    return self._maxsize - self._sem._semlock._get_value()
NotImplementedError

joeyespo · 2015-06-10T21:02:48Z

@aldanor There's a lot of divergent subtle cases being handled in this PR. If it's getting too unwieldy, perhaps open a few different ones to address specific issues. I'm very ready to pull in most things in here, but until it's completely ready, the whole thing is held up.

aldanor · 2015-06-10T21:07:18Z

I got to my mac, will run the whole thing on it tonight.

I'm actually thinking this whole thing needs a test suite...

aldanor · 2015-06-10T21:09:36Z

@joeyespo Yep I know.... it will be real hard for me to start going backwards and split it up because I have to use it on daily basis, which means I'll have to keep track of more than three versions now -- remote, local and multiple PR branches... :(

I'd rather make this whole thing work. (the norecursedirs changes I can roll back -- that's the only truly questionable part which will need further effort and thought).

aldanor · 2015-06-10T21:18:16Z

@tarpas Fixed. OSX weirdness...

aldanor · 2015-06-10T21:34:41Z

@blueyed Fixed

aldanor · 2015-06-10T21:36:08Z

Anything else?

tarpas · 2015-06-11T06:35:55Z

Works without problems for me also with pytest testmon. Thanks for the
work! @joeyespo it would be great if you merge it.

On Wed, Jun 10, 2015 at 11:02 PM, Joe Esposito notifications@github.com
wrote:

@aldanor https://github.com/aldanor There's a lot of divergent subtle
cases being handled in this PR. If it's getting too unwieldy, perhaps open
a few different ones to address specific issues. I'm very ready to pull in
most things in here, but until it's completely ready, the whole thing is
held up.

—
Reply to this email directly or view it on GitHub
#11 (comment).

joeyespo · 2015-06-13T19:28:57Z

Yep I know.... it will be real hard for me to start going backwards and split it up because I have to use it on daily basis, which means I'll have to keep track of more than three versions now -- remote, local and multiple PR branches

Yeah, it'd be kind of tough now. Oh well.

Pulling this is now so we can move forward. We can fix up --ignore / --norecursedirs stuff after. Lots of good work in this PR.

Many thanks, @aldanor 😃

joeyespo · 2015-06-13T19:30:19Z

@blueyed Did 0a24376 do what you need? If not, could you open a new issue?

Multiple directories support, norecursedirs, passing args to pytest

joeyespo · 2015-06-13T21:18:18Z

Thanks again, @aldanor for your fantastic work here. And for your patience :-)

aldanor · 2015-06-13T21:20:59Z

@joeyespo Np :) I think all the outstanding issues have been covered, everyone's happy? (--ignore is a separate issue but it actually does somewhat work already)

Btw, maybe it would make sense to move this thing over to http://github.com/pytest-dev? (seeing as pytest itself will move there soon)

aldanor · 2015-06-13T21:31:37Z

This one was one hell of a PR, more Python LOC than the original code :)

joeyespo · 2015-06-13T21:38:46Z

Btw, maybe it would make sense to move this thing over to http://github.com/pytest-dev?

I'm up for it if they are.

This one was one hell of a PR, more Python LOC than the original code :)

😃

aldanor · 2015-06-13T21:43:34Z

What exactly is Linux-only? I'm mostly on OSX myself (occasionally on Linux)

joeyespo · 2015-06-14T00:24:13Z

@aldanor Oops, meant Unix, not Linux. os.path.samefile is Unix-only.

blueyed · 2015-06-14T00:43:38Z

@joeyespo

@blueyed Did 0a24376 do what you need? If not, could you open a new issue?

Yes, thanks!

joeyespo · 2015-06-14T21:02:58Z

FYI released these changes today. Be sure to pip install --upgrade pytest-watch 😃

aldanor added 8 commits May 27, 2015 15:17

Only react to FileModifiedEvent and FileCreatedEvent watchdog events.

8b1eb9e

- This fixes the problem with catching duplicate watchdog events related to a single file modification if using remote NFS mounts.

If KeyboardInterrupt occurred after the first test run, still exit gr…

b2b23d9

…acefully

Remove redundant docstring

f5b14af

Add --norecursedirs (only a first-level exact match for now)

b4135b7

Report the exact pytest command executed, highlight command and filen…

8c4fdf9

…ames

Remove duplicated line

d1d1910

Launch py.test subprocess with shell=False.

dc066e2

This fixes the edge case when py.test arguments contained quoted strings with spaces, e.g.: ptw -- -m 'not slow'

This was referenced May 27, 2015

Cannot specify excluded directory patterns #6

Closed

multiple directories support #5

Closed

Add a way to pass parameters to pytest #8

Closed

Subscribe to FileMovedEvent as well and match its dest_path.

f0b97bb

This fixes the issue with some editors that do atomic saves via a temporary file and a subsequent move.

aldanor added 2 commits May 28, 2015 10:41

Remove shell=* keyword argument from all subprocess.call uses

a1a7002

Rename --norecursedirs to --ignore

2181ce2

CLI now accepts both directories and pytest positional args (separate…

17b22b1

…d by --) Also, make --ext a bit smarter by prepending the '.' automatically if need be.

aldanor added 4 commits June 10, 2015 13:15

Update CLI help string

59676fa

Remove temporary print

0835433

Add event spooler module

7012a90

Implement event spooling with 200ms cooldown

0d8dcc0

aldanor mentioned this pull request Jun 10, 2015

Suite reruns few times #12

Closed

Fix EventSpooler on OSX (sem_getvalue is not implemented)

3ec1363

Don't use LIGHT*_EX colors

0a24376

aldanor added 2 commits June 10, 2015 22:40

Add --no-spool option (spooling is enabled by default)

3a30c1e

Add --verbose/-v and --quiet/-q args

0e6e14a

joeyespo added a commit that referenced this pull request Jun 13, 2015

Merge pull request #11 from aldanor/better-pytest-watch

de5d34d

Multiple directories support, norecursedirs, passing args to pytest

joeyespo merged commit de5d34d into joeyespo:master Jun 13, 2015

joeyespo mentioned this pull request Jun 13, 2015

Proposal for allowing nowatchdirs #7

Closed

joeyespo added the enhancement label Jun 28, 2015

pyup-bot mentioned this pull request Aug 17, 2016

pin poncki/pydd#1

Closed

pyup-bot mentioned this pull request Jul 24, 2017

Pin pytest-watch to latest version 4.1.0 michaeljoseph/mould#12

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Multiple directories support, norecursedirs, passing args to pytest #11

Multiple directories support, norecursedirs, passing args to pytest #11

aldanor commented May 27, 2015

joeyespo commented May 28, 2015

joeyespo commented May 28, 2015

aldanor commented May 28, 2015

aldanor commented May 28, 2015

aldanor commented Jun 10, 2015

aldanor commented Jun 10, 2015

blueyed commented Jun 10, 2015

blueyed commented Jun 10, 2015

tarpas commented Jun 10, 2015

joeyespo commented Jun 10, 2015

aldanor commented Jun 10, 2015

aldanor commented Jun 10, 2015

aldanor commented Jun 10, 2015

aldanor commented Jun 10, 2015

aldanor commented Jun 10, 2015

tarpas commented Jun 11, 2015

joeyespo commented Jun 13, 2015

joeyespo commented Jun 13, 2015

joeyespo commented Jun 13, 2015

aldanor commented Jun 13, 2015

aldanor commented Jun 13, 2015

joeyespo commented Jun 13, 2015

aldanor commented Jun 13, 2015

joeyespo commented Jun 14, 2015

blueyed commented Jun 14, 2015

joeyespo commented Jun 14, 2015

Multiple directories support, norecursedirs, passing args to pytest #11

Multiple directories support, norecursedirs, passing args to pytest #11

Conversation

aldanor commented May 27, 2015

joeyespo commented May 28, 2015

joeyespo commented May 28, 2015

aldanor commented May 28, 2015

aldanor commented May 28, 2015

aldanor commented Jun 10, 2015

aldanor commented Jun 10, 2015

blueyed commented Jun 10, 2015

blueyed commented Jun 10, 2015

tarpas commented Jun 10, 2015

joeyespo commented Jun 10, 2015

aldanor commented Jun 10, 2015

aldanor commented Jun 10, 2015

aldanor commented Jun 10, 2015

aldanor commented Jun 10, 2015

aldanor commented Jun 10, 2015

tarpas commented Jun 11, 2015

joeyespo commented Jun 13, 2015

joeyespo commented Jun 13, 2015

joeyespo commented Jun 13, 2015

aldanor commented Jun 13, 2015

aldanor commented Jun 13, 2015

joeyespo commented Jun 13, 2015

aldanor commented Jun 13, 2015

joeyespo commented Jun 14, 2015

blueyed commented Jun 14, 2015

joeyespo commented Jun 14, 2015