Watchman doesn't work with react-native anymore

[mlabrum]

Hey,

I've grabbed the latest watchman from https://ci.appveyor.com/api/projects/wez/watchman/artifacts/watchman.zip?branch=master&job=Environment:+WATCHMAN_WIN7_COMPAT%3D

React-native version: 0.44.2 (latest)
Windows 10

Running react-native with react-native start

Looking for JS files in
   C:\tmp\test\test


React packager ready.

Loading dependency graph, done.
error: bundling: Error
    at DependencyGraph._getAbsolutePath (C:/tmp/test/test/node_modules/react-native/packager/src/node-haste/index.js:272:11)
    at DependencyGraph.getDependencies (C:/tmp/test/test/node_modules/react-native/packager/src/node-haste/index.js:216:26)
    at Resolver.getDependencies (C:/tmp/test/test/node_modules/react-native/packager/src/Resolver/index.js:106:27)
    at _resolverPromise.then.resolver (C:/tmp/test/test/node_modules/react-native/packager/src/Bundler/index.js:561:62)
    at process._tickCallback (internal/process/next_tick.js:109:7)
Bundling `index.android.js`  0.0% (0/1), failed.

React native on device gives the following error
Cannot find entry file index.android.js in any of the roots ["c:/tmp/test/test/"]

Watchman logs

2017-06-05T09:27:39,625: [listener] Watchman 4.9.0 <no build info set> starting up on NTCSRF206
2017-06-05T09:27:39,632: [listener] failed to parse json from C:\Users\Matt\AppData\Local\Temp/Matt-state/state: unable to open C:\Users\Matt\AppData\Local\Temp/Matt-state/state: No such file or directory
2017-06-05T09:27:39,632: [listener] waiting for pipe clients on \\.\pipe\watchman-Matt
2017-06-05T09:27:39,656: [client=000001D8D9454810:stm=000001D8D9452770:pid=0] path C:/tmp/test/test is on filesystem type NTFS
2017-06-05T09:27:39,658: [client=000001D8D9454810:stm=000001D8D9452770:pid=0] root C:/tmp/test/test using watcher mechanism win32 (auto was requested)
2017-06-05T09:27:41,398: [client=000001D8D9454810:stm=000001D8D9452770:pid=0] PERF: {"start_time": 1496620659.6559999, "version": "4.9.0", "description": "dispatch_command:watch-project", "meta": {"root": {"path": "C:/tmp/test/test", "watcher": "win32", "ticks": 1, "recrawl_count": 0, "case_sensitive": false, "number": 1}, "args": ["watch-project", "C:/tmp/test/test"], "client": {"pid": 0}}, "pid": 12892, "elapsed_time": 1.742}
2017-06-05T09:27:41,398: [io 000001D8D946DC10 C:/tmp/test/test] crawl complete
2017-06-05T09:27:41,398: [io 000001D8D946DC10 C:/tmp/test/test] PERF: {"start_time": 1496620659.6689999, "version": "4.9.0", "description": "full-crawl", "meta": {"root": {"path": "C:/tmp/test/test", "watcher": "win32", "ticks": 2, "recrawl_count": 0, "case_sensitive": false, "number": 1}}, "pid": 12892, "elapsed_time": 1.7290000000000001}

I can confirm watchman 4.7.0 worked (though it had the symlink issue on windows)

Thanks,
I'm currently digging through RN code trying to see if I can find anything...

[wez]

Can you try running:

watchman --server-encoding=json --persistent log-level debug > debug.log

in another window while you try running react native, and share the debug.log file with me? That should show us more detail about what watchman is doing.

[mlabrum]

Hey,
I've done some hacking and I think I've found the issue and it seems to be due to path separators

I've modified node_modules\jest-haste-map\build\crawlers\watchman.js:110 and added
console.log(name)

Watchman 4.9.0
Gives

C:/tmp/test/test\node_modules/qs/bower.json
C:/tmp/test/test\node_modules/random-bytes/package.json
C:/tmp/test/test\node_modules/randomatic/package.json
C:/tmp/test/test\node_modules/range-parser/package.json
C:/tmp/test/test\node_modules/raw-body/package.json
C:/tmp/test/test\node_modules/react/package.json
C:/tmp/test/test\node_modules/react-clone-referenced-element/package.json
C:/tmp/test/test\node_modules/react-deep-force-update/package.json
C:/tmp/test/test\node_modules/react-devtools-core/package.json
C:/tmp/test/test\node_modules/react-native/package.json
C:/tmp/test/test\node_modules/react-native/jest-preset.json
C:/tmp/test/test\node_modules/react-proxy/package.json

Watchman 4.7.0
Gives

C:\tmp\test\test\node_modules\sprintf-js\demo\angular.html
C:\tmp\test\test\node_modules\serve-index\public\directory.html
C:\tmp\test\test\node_modules\rebound\browser_test\index.html
C:\tmp\test\test\node_modules\rebound\docs\rebound.html
C:\tmp\test\test\node_modules\rebound\examples\index.html
C:\tmp\test\test\node_modules\errorhandler\public\error.html
C:\tmp\test\test\node_modules\clone\test.html
C:\tmp\test\test\node_modules\clone\test-apart-ctx.html
C:\tmp\test\test\node_modules\jsbn\example.html
C:\tmp\test\test\node_modules\rebound\index.html
C:\tmp\test\test\node_modules\sax\LICENSE-W3C.html
C:\tmp\test\test\node_modules\through2\LICENSE.html

So it seems newer versions of watchman are returning different file paths?

(I can still attach that debug log if required)

[wez]

if you hack that javascript file to replace / with \\ does it make react native happy again?

[wez]

... I'm not really a javascript guy; presumably something like name = name.replace('/', '\\'); would do the job

[mlabrum]

Yes it does work

[wez]

For the sake of clarifying what the problem is, can you also try normalizing it the other way?

name = name.replace('\\', '/');

[wez]

also: how did you track it down to this? Can you share the error you found and where you found it?

[wez]

FWIW, f257233 is where this behavior was changed

[wez]

The reason I'm asking the above is to assess whether we need to go back to the old behavior to minimize headaches, or whether it is just that react-native's code needs to be fixed up.

[mlabrum]

replacing it all with / doesn't work.

I tracked this down by doing the following....

Running: react-native start

Looking for JS files in
   C:\tmp\test\test


React packager ready.

Loading dependency graph, done.
C:\tmp\test\test\index.android.js
(node:3152) UnhandledPromiseRejectionWarning: Unhandled promise rejection (rejection id: 2): NotFoundError: Cannot find entry file index.android.js in any of the roots: ["C:\\tmp\\test\\test"]
(node:3152) DeprecationWarning: Unhandled promise rejections are deprecated. In the future, promise rejections that are not handled will terminate the Node.js process with a non-zero exit code.
Bundling `index.android.js`  ░░░░░░░░░░░░░░░░  0.0% (0/1)
error: bundling: Error
    at DependencyGraph._getAbsolutePath (C:/tmp/test/test/node_modules/react-native/packager/src/node-haste/index.js:276:11)
    at DependencyGraph.getDependencies (C:/tmp/test/test/node_modules/react-native/packager/src/node-haste/index.js:216:26)
    at Resolver.getDependencies (C:/tmp/test/test/node_modules/react-native/packager/src/Resolver/index.js:106:27)
    at _resolverPromise.then.resolver (C:/tmp/test/test/node_modules/react-native/packager/src/Bundler/index.js:561:62)
    at process._tickCallback (internal/process/next_tick.js:109:7)
Bundling `index.android.js`  0.0% (0/1), failed.

So I jumped to the first file in the exception
C:/tmp/test/test/node_modules/react-native/packager/src/node-haste/index.js:276

Found it was trying to build an absolute path for the file it was trying to find, so I added a console.log to to figure out what potentialAbsPath was
and it gave me "C:\tmp\test\test\index.android.js"
then I worked out what this._hasteFS.exists(potentialAbsPath) was calling and hasteFS seems to do a lookup against the file cache
I found a method on hasteFS called getAllFiles so I added a console.log(this._hasteFS.getAllFiles() and found that it listed files with the wrong path separator

then it was just a hunt to figure out how react-native received the files from watchman node_modules\jest-haste-map\build\crawlers\watchman.js and testing replacing the file separator.

I guess this could just be fixed in react-native, but I do wonder what else in the watchman ecosystem depends on windows file paths matching....

[wez]

@cpojer (picking you because you touched that portion of RN most recently!) can you help me (or find me someone that can help me!) understand where the sensitivity to the slashes is coming from within the hasteFS code?

Forward slashes and mixtures of forward slashes are fine on windows (you'll notices that node itself is using them in the stack trace in the comment above), so I'd prefer to make RN do the right thing here.

[cpojer]

This is the code that is shared between Jest and react-native-packager: https://github.com/facebook/jest/blob/master/packages/jest-haste-map/src/index.js

[wez]

We're chatting about this internally; at the moment we're learning towards making a change in the javascript side of haste/react-native

[iremlopsum]

facebook/react-native#14736

Having the same issue

[desfero]

@wez @mlabrum is there a possibility to downgrade watchman version to 4.7?

[mlabrum]

@iremlopsum @desfero did you try patching react native?

[cpojer]

@jeanlauliac is working on deploying this fix to metro-bundler and react-native with the latest alpha of jest-haste-map.

[iremlopsum]

@mlabrum nor name = name.replace('/', '\\'); or name = name.replace('\\', '/'); did anything. getting the same result.

[iremlopsum]

@mlabrum normalizing paths made it work.

[jeanlauliac]

@mlabrum nor name = name.replace('/', '\'); or name = name.replace('\', '/'); did anything.

You need to use a regex with g flag, otherwise only the first occurrence is replaced. That is maybe why it did not work? Ex. name.replace(/\//g, '\\'). I've landed that tweak in jest-haste-map, updating it in the bundler.

[patroza]

For me it works if I apply following workarounds:
L62 add: root = root.replace(/\//g, '\\');
L66 add: subRoot = subRoot.replace(/\//g, '\\');
L110 replace: const name = (root + path.sep + fileData.name).replace(/\//g, '\\');

One thing to note is that if you have Jest installed globally, you most likely have to edit the global node_modules/jest/node_modules/jest-haste-map, not the one in your app.

Lastly you might have to close Watchman so it starts fresh again next run to see results (working turning to failing, or failing turning to working) due to watchman caching etc.

[JCMais]

Adding to @patroza comment, you need to add the g flag to the path regex, like so:
const name = (root + path.sep + fileData.name).replace(/\//g, '\\');

[patroza]

@JCMais thanks, overlooked that one, made a new PR @ https://github.com/facebook/jest/pull/4018/files
that seems to address it for me.

[Fonze]

Are there any out-of-the-box fixes for this yet?

[JCMais]

@Fonze besides the ones mentioned on @patroza comment, some fixes were made to jest already, however a new release was not made yet.

[wez]

facebook/metro#2 was closed today; please follow up there if you're continuing to experience these issues!

[timothyf]

I have react 0.50.4 and watchman 4.9 and still encountering this same problem.

[chidionuekwusi]

Same here.

[lcr0815]

can't install watchman anymore, npm errors out but the android build of a new react-native build gives the the error of accessibility info and directs the user to run watchman to clear the haste and clear the cache. this is a real PITA issue as it takes an entire day to create a simple app... :(:

[mustafakameldev]

I getting the same problem when i tried watchman --server-encoding=json --persistent log-level debug > debug.log
i got these

{
    "version": "4.9.0",
    "log_level": "debug"
}
{
    "log": "2021-03-02T14:01:46,915: [client=0x55a45da1a9f0:stm=0x55a45da17640:pid=4299] dispatch_command: log-level (completed)\n",
    "level": "debug",
    "unilateral": true
}
{
    "log": "2021-03-02T13:58:06,544: [io 0x55a45da1ef10 /home/mospro/flipper] crawl complete\n",
    "level": "error",
    "unilateral": true
}
{
    "log": "2021-03-02T13:58:09,388: [client=0x55a45dc208e0:stm=0x55a45da1b340:pid=3437] PERF: {\"ru_nvcsw\": 160799, \"ru_oublock\": 0, \"ru_inblock\": 980024, \"ru_msgsnd\": 0, \"ru_nswap\": 0, \"ru_majflt\": 0, \"ru_nivcsw\": 64234, \"ru_minflt\": 25607, \"elapsed_time\": 12.691421, \"ru_msgrcv\": 0, \"version\": \"4.9.0\", \"start_time\": 1614686276.693538, \"meta\": {\"client\": {\"pid\": 3437}, \"root\": {\"watcher\": \"inotify\", \"ticks\": 2, \"number\": 1, \"path\": \"/home/mospro/Documents/warshitna\", \"recrawl_count\": 0, \"case_sensitive\": true}, \"args\": [\"query\", \"/home/mospro/Documents/warshitna\", {\"glob\": [\"**/*.js\", \"**/*.json\", \"**/*.ts\", \"**/*.tsx\", \"**/*.bmp\", \"**/*.gif\", \"**/*.jpg\", \"**/*.jpeg\", \"**/*.png\", \"**/*.psd\", \"**/*.svg\", \"**/*.webp\", \"**/*.m4v\", \"**/*.mov\", \"**/*.mp4\", \"**/*.mpeg\", \"**/*.mpg\", \"**/*.webm\", \"**/*.aac\", \"**/*.aiff\", \"**/*.caf\", \"**/*.m4a\", \"**/*.mp3\", \"**/*.wav\", \"**/*.html\", \"**/*.json\", \"**/*.pdf\", \"**/*.yaml\", \"**/*.yml\", \"**/*.otf\", \"**/*.ttf\", \"**/*.zip\"], \"expression\": [\"allof\", [\"type\", \"f\"], [\"anyof\", [\"suffix\", \"js\"], [\"suffix\", \"json\"], [\"suffix\", \"ts\"], [\"suffix\", \"tsx\"], [\"suffix\", \"bmp\"], [\"suffix\", \"gif\"], [\"suffix\", \"jpg\"], [\"suffix\", \"jpeg\"], [\"suffix\", \"png\"], [\"suffix\", \"psd\"], [\"suffix\", \"svg\"], [\"suffix\", \"webp\"], [\"suffix\", \"m4v\"], [\"suffix\", \"mov\"], [\"suffix\", \"mp4\"], [\"suffix\", \"mpeg\"], [\"suffix\", \"mpg\"], [\"suffix\", \"webm\"], [\"suffix\", \"aac\"], [\"suffix\", \"aiff\"], [\"suffix\", \"caf\"], [\"suffix\", \"m4a\"], [\"suffix\", \"mp3\"], [\"suffix\", \"wav\"], [\"suffix\", \"html\"], [\"suffix\", \"json\"], [\"suffix\", \"pdf\"], [\"suffix\", \"yaml\"], [\"suffix\", \"yml\"], [\"suffix\", \"otf\"], [\"suffix\", \"ttf\"], [\"suffix\", \"zip\"]]], \"fields\": [\"name\", \"exists\", \"mtime_ms\", \"size\", \"content.sha1hex\"]}]}, \"description\": \"dispatch_command:query\", \"user_time\": 6.6904279999999998, \"ru_maxrss\": 101816, \"system_time\": 7.0393920000000003, \"ru_nsignals\": 0, \"pid\": 3455, \"ru_ixrss\": 0, \"ru_idrss\": 0}\n",
    "level": "error",
    "unilateral": true
}
{
    "log": "2021-03-02T14:02:46,977: [listener] accepted client:stm=0x55a45da1e4c0\n",
    "level": "debug",
    "unilateral": true
}
{
    "log": "2021-03-02T14:02:46,977: [client=0x55a45da1a6b0:stm=0x55a45da1e4c0:pid=3455] dispatch_command: get-pid\n",
    "level": "debug",
    "unilateral": true
}
{
    "log": "2021-03-02T14:02:46,977: [client=0x55a45da1a6b0:stm=0x55a45da1e4c0:pid=3455] dispatch_command: get-pid (completed)\n",
    "level": "debug",
    "unilateral": true
}
{
    "log": "2021-03-02T14:02:46,977: [NOT_CONN:client=0x55a45da1a6b0:stm=0x55a45da1e4c0:pid=3455] client_delete 0x55a45da1a6b0\n",
    "level": "debug",
    "unilateral": true
}