Alpha Windows support

[mitchellh]

@wez edited this issue to place this status at the top

Current status: Alpha for 64-bit Windows.
It builds and passes the test suite on Windows Server 2012 R2 (x64) and should work on Windows 7 and later. PCRE expression terms are not currently supported.

At this time we're looking for alpha testers. Watchman on Windows may hang, crash or otherwise be unreliable. If you encounter problems we'll need you to provide detailed information about how to trigger the problem, and we may ask you to use visual studio to attach and perform some rudimentary debugging.

The latest build is available here:
https://ci.appveyor.com/api/projects/wez/watchman/artifacts/watchman.zip?branch=master&job=Environment:+WATCHMAN_WIN7_COMPAT%3D

Extract the zip file and make sure that watchman.exe is located in your PATH.
The watchman.pdb file is provided to facilitate debugging.

If you are consuming watchman from python, you will need to build pywatchman for yourself.
The released package on https://pypi.python.org/pypi/pywatchman should work for this purpose.

If you are consuming watchman from nodejs, the existing fb-watchman package available via npm should work, but the version in master has slightly better diagnostics in case you experience hangs.

[wez]

Windows is not currently supported. We have no immediate plans to add support for it. We would welcome a contribution to add this support if someone has the time and inclination to produce it.

[wez]

To support Windows, we first need to tidy up our handling of case-insensitive filesystems; that work is in-progress, but we still have no immediate plans for Windows at this time.

[wez]

We're tracking the list of things that would need to be tackled for Windows support in this wiki page:
https://github.com/facebook/watchman/wiki/Changes-required-for-Windows-support

[pclouds]

This is something you guys may want to watch out when using ReadDirectoryChangesW [1]

I dropped ReadDirectoryChangesW because maintaining a 'live' file system cache became more and more
complicated. For example, according to MSDN docs, ReadDirectoryChangesW may report short DOS 8.3
names (i.e. "PROGRA~1" instead of "Program Files"), so a correct and fast cache implementation would
have to be indexed by long and short names...

[1] http://article.gmane.org/gmane.comp.version-control.git/216195

[ziriax]

You might want to consider using the NTFS Change Journal. It is rock solid. I used it for Evolver, a closed source version control system that I developed for a game company in 2002.

It seems a Python wrapper is already made

[wez]

cc: @dsp

I just pushed https://github.com/wez/watchman/tree/win with my hacking in progress; it's been sitting on my laptop for a while and I just rebased on master as of today and make sure that it still builds on both osx and windows.

You will need the microsoft visual C++ compiler to build it; you can use the community edition for this.
The compiler tools (cl, nmake and so on) must be in your path.
To run the integration tests you will need arcanist and php installed and in your path.

To build and test:

cd watchman
nmake -f winbuild\Makefile check
arc test

The integration tests are a bit rough and ready largely because of the filesystem semantics on windows are not POSIX; there are a lot of errors around attempting to delete dirs while handles are still open on them. There are also some errors with / instead of \ in path names. These are for the most part cosmetic.

The current status is 26 of 74 tests failed.

If you're interested in helping with hacking and testing, you are most welcome! We're at the cleaning things up stage.

If you're interested in making a client connect to it, you can connect to a named pipe for the current user; for example, my watchman instance listens at \\.\pipe\watchman-wez. The code to select the default name of the pipe can be found here: https://github.com/wez/watchman/blob/win/main.c#L456

Once connected, you use the standard JSON or BSER watchman protocol to communicate with the server.

[ziriax]

Great! I just tried to compile and run watchman.exe, but the code seems to use Windows 8 specific APIs, while we're still at Windows 7. For example, the API function GetOverlappedResultEx is used. Is this by design?

[wez]

Windows 8 has a number of API calls that make it easier to port posix style applications and we're currently using those. In addition, I didn't have easy access to Windows 7 while I was prototyping this. I'm not opposed to making changes to have this run on Windows 7.

[ziriax]

It seems the only Win8 specific API function call is the GetOverlappedResultEx. At first sight it seems you can just replace this with the old GetOverlappedResult function, since your timeout is either 0 or INFINITE, so the timeout maps to the BOOL bWait argument of the old function.

I was able to compile and run the watchman fine on my Win7 machine, and the log shows that watching indeed works.

I was not yet able to get the hgwatchman Mercurial extension running on Windows, but I suspect I have a conflict between 32-bit and 64-bit version. I'll dig deeper.

[wez]

@ziriax great! I just (force) pushed some fixes to my branch to pick up the python build and test bits. You'll also see how to establish the named pipe connection you'll need for hgwatchman--you have to use os.open because the higher level file abstraction in python doesn't work properly with named pipes

[wez]

I remembered that I added make.bat in my branch as a short cut, so you can just run:

make

to have it run nmake with the right makefile for you. This script will also source the visual studio build environment. We're moving towards having the tests use python, so it is best if you have python in your path when you build; I'm using python 2.7.

[wez]

Current status is 18 of 73 failing, and a quick scan of the output looks like they're all / vs \ path related in the test expectations, or because we don't yet have pcre in the build and aren't smart enough to skip the pcre tests if support is missing.

I fixed a couple of race conditions and issues with relative roots and window filesystem convention handling.

I'll take a look at the GetOverlappedResult thing as well a bit later tonight.

[wez]

OK, the most recent run is down to just 6 failing tests.
I've also pushed a commit that tries to do the right thing on Windows 7; it's not as simple as just calling GetOverlappedResult as that will impact the ability of the service to handle concurrent clients (more details can be found in that commit)

[stefanpenner]

@wez if I recall some modes of windows FS watching (at-least the flags that are used in libuv) trigger change events on a file when attaching a hard link to it.

If possible, could watchman on windows not share this same behavior?

[stefanpenner]

If memory serves (it was quite some time ago), attaching a hard link mutates an attribute on the file: https://github.com/libuv/libuv/blob/188e0e94ac8374a5e177e1407052ed1b26c528a6/src/win/fs-event.c#L49

[stefanpenner]

It's also worth noting, my above statement might just be describing a symptom of an entirely different problem. So feel free to disregard as you feel appropriate.

[wez]

[FYI: am currently traveling and may not be super responsive]

As of the most recent commit on wez/watchman:win, watchman is buildable on windows and basically functions (as in, I have only performed the most basic test as shown below) with hgwatchman.

To get hgwatchman to work, you'll need to copy the python/pywatchman dir from the windows watchman tree to hgwatchman/hgwatchman/pywatchman and then apply this patch to hgwatchman:

diff --git a/hgwatchman/__init__.py b/hgwatchman/__init__.py
--- a/hgwatchman/__init__.py
+++ b/hgwatchman/__init__.py
@@ -170,6 +170,8 @@
     if normalize:
         foldmap = dict((normcase(k), k) for k in results)

+    hg_slash = '.hg' + os.sep
+
     for entry in result['files']:
         fname = entry['name']
         if normalize:
@@ -183,7 +185,7 @@
         fmode = entry['mode']
         fexists = entry['exists']
         kind = getkind(fmode)
-        if fname == '.hg' or fname.startswith('.hg/'):
+        if fname == '.hg' or fname.startswith(hg_slash):
             continue

         if fname not in results:

Then run this in the root of the hgwatchman repo:

python setup.py build_py -c -d .  build_ext -i

Then:

hg --config extensions.hgwatchman=z:\fb\hgwatchman\hgwatchman --config watchman.mode=on st

[wez]

@stefanpenner https://github.com/wez/watchman/blob/win/watcher/win32.c#L142 is what we're using for our win32 watcher.

We're really at the mercy of what windows reports here. The nature of these things is that neither the kernel nor watchman itself can reliably maintain enough state to guarantee that we can quash that kind of a notification without risking making the wrong choice and not notifying you of a similar but otherwise notify-able change.

Our philosophy on these situations in watchman is that we are conservative and err on the side of over notifying rather than missing a notification.

[stefanpenner]

Our philosophy on these situations in watchman is that we are conservative and err on the side of over notifying rather than missing a notification.

this sounds very reasonable :) If we run into issues, i will explore it in more details and report back. Thank you for the link.

[ziriax]

For a similar project I used the NTFS change journal. As far as I recall, this is very reliable. But since it logs all changes to a partition, we used to have a dedicated partition for our source files, and another one for the generated files.

[wez]

Pull request for hgwatchman to support windows:
https://bitbucket.org/facebook/hgwatchman/pull-requests/3

[wez]

@ziriax yeah, we've often talked about how it would be great if the filesystems themselves gave us an API for these kinds of operations; it would help to eliminate some of the TOCTOU issues that we have to deal with in the watchman service.

I haven't looked at the NTFS change journal API, but watchman does have a reasonably modular way to plug in an alternative watcher implementation (watcher/win32.c is currently the only windows implementation); it might be interesting to try building a change journal watcher to contrast/compare. It does sound a bit more painful to configure and deploy, so it probably would not be the default mode of operation. In addition, the ReadDirectoryChangeW API should work with more filesystems.

I don't think we're actively going to try the change journal ourselves, but I'm definitely open to reviewing and evaluating that contribution :-)

[ziriax]

Well, that's the nice thing about the NTFS change journal, since the file system writes entries in the journal, you do not need to "watch" anything. You just ask the file system about what changes happened between two "timestamps". No software has to run to constantly monitor the changes.

We found the directory notification system to be rather unreliable. But we used Windows XP in those days, so a lot could be improved by now.

-----Original Message-----
From: "Wez Furlong" notifications@github.com
Sent: ‎30/‎07/‎2015 19:22
To: "facebook/watchman" watchman@noreply.github.com
Cc: "Peter Verswyvelen" bugfact@gmail.com
Subject: Re: [watchman] Windows support? (#19)

@ziriax yeah, we've often talked about how it would be great if the filesystems themselves gave us an API for these kinds of operations; it would help to eliminate some of the TOCTOU issues that we have to deal with in the watchman service.
I haven't looked at the NTFS change journal API, but watchman does have a reasonably modular way to plug in an alternative watcher implementation (watcher/win32.c is currently the only windows implementation); it might be interesting to try building a change journal watcher to contrast/compare. It does sound a bit more painful to configure and deploy, so it probably would not be the default mode of operation. In addition, the ReadDirectoryChangeW API should work with more filesystems.
I don't think we're actively going to try the change journal ourselves, but I'm definitely open to reviewing and evaluating that contribution :-)
—
Reply to this email directly or view it on GitHub.

[upsuper]

Hmm, I thought the bitly link in the main description above was auto-updating to be current, I'm not sure if that is the case.

It doesn't seem to me that's the case.

• http://bit.ly/watchmanwinalpha redirects to
• https://ci.appveyor.com/api/buildjobs/bd0fppc1sh70emkq/artifacts/watchman.zip redirect to
• https://appveyorcidata.blob.core.windows.net/wez-25370/watchman/b11/bd0fppc1sh70emkq/watchman.zip?sv=2014-02-14&sr=c&sig=91vkxVcvwaWe05R2AmTni1NRu3%2F1xvhPcd0j%2BvdBR8k%3D&st=2016-09-23T11%3A32%3A13Z&se=2016-09-23T11%3A38%3A13Z&sp=r

The Last-Modified field of the last url shows Sat, 08 Aug 2015 19:22:02 GMT.

Instead, the link in your comment shows Tue, 23 Aug 2016 17:20:41 GMT for Last-Modified.

[igorsvee]

Can confirm that hot reloading works on Win 8.1 + genymotion + react-native 0.34.1 using https://ci.appveyor.com/api/buildjobs/kravw77nw7fqhvio/artifacts/watchman.zip. Thanks guys.

[waynebloss]

Are you guys officially supporting Windows now or not?

The first comment still reads "Windows is not currently supported" and this issue is still open, so I am assuming the answer is No?

Is this project even necessary for Windows, since Windows had a file-change notifications API way before the Mac OS or Linux had them?

Thanks.

[wez]

@waynebloss at the moment the folks working on Watchman don't run it on Windows for any real purpose. We're anticipating that this will change in the future, which is why we've added support and have CI configured and so on. Until that changes and we've seen what really happens to the system outside of our integration test suite, we feel that it would be disingenuous to label it more than alpha or perhaps beta.

Watchman provides both a real-time file change subscription mechanism (echoing what the native OS facilities provide, but in a cross-platform way) and functionality to query the view of the filesystem without talking to the filesystem. The latter part of this may sound strange, but when you are dealing with a very large directory tree, the act of looking at a "cold" portion of it can block for a non-trivial amount of time.

Whether watchman is necessary depends on your needs.

[muloka]

@wez out of curiosity, got any recommendations of alternative tools for Windows?

[kumarharsh]

@wez: I'm writing a plugin for vscode: graphql-for-vscode which relies on watchman. Users have been using it for about a month now, and till date there's not been a single issue. Pretty good work by your team 👍

[wez]

@muloka do you mean alternatives to watchman, or just tools for windows?

[muloka]

@wez Alternatives to watchman that are suitable for Windows

Anyway since writing my comment I've dabbled in both Powershell and syncthing

Using the two I've managed to arrive at a solution that works.

[lednhatkhanh]

Hi, where can I find the latest windows version?

[kumarharsh]

@lednhatkhanh The link is in the original post above: http://bit.ly/watchmanwinalpha

[wez]

Figured out that we can use this link to always reference the latest build:

https://ci.appveyor.com/api/projects/wez/watchman/artifacts/watchman.zip?branch=master&job=Environment:+WATCHMAN_WIN7_COMPAT%3D

(updated the description to reflect this)

[masaeedu]

@wez The clockspec on Windows looks a bit weird; it doesn't seem to match the two segment clockspec shown in the docs. Is there some special meaning for each

I get something with four segments when I run watchman since D:\depot\git\agentautodeditor\sampledata n:foo '':

{
    "version": "4.9.0",
    "is_fresh_instance": false,
    "clock": "c:1496775649:11508:1:63",
    "files": []
}

Additionally, this clockspec seems to increment every time I run that command, whereas from my understanding of the docs logical clock IDs should not increment unless files have actually changed in the directory. I've been using watchman clock D:\depot\git\agentautodeditor\sampledata instead, which seems to maintain a stable clock when I haven't changed anything.

Could you point me to some detailed docs on the anatomy of a clock id, and whether this is how things should be working?

[wez]

@masaeedu would you mind opening a separate issue to ask those questions? This one is already pretty overloaded and long

[wez]

OK folks, thanks for following along; I'm going to promote us from alpha to beta on Windows, so I'm closing this issue.

The install section of the docs has more information about downloading and installing the beta, as well as encouraging folks to file issues if/when they encounter them.

https://facebook.github.io/watchman/docs/install.html#download-for-windows-beta

[chidionuekwusi]

The link is broken

[AdrianoCahete]

The link is broken

This is working: https://ci.appveyor.com/api/buildjobs/41p1lupk97crwvm0/artifacts/watchman.zip

[wez]

This link works correctly:

https://ci.appveyor.com/api/projects/wez/watchman/artifacts/watchman.zip?branch=master&job=Environment:+WATCHMAN_WIN7_COMPAT%3D&pr=false

The issue is that the default for pr is true so it's trying to download artifacts from the not-yet-ready cmake changes I'm hacking on in another pr :-/