WARNING failed to contact turbod.

[AddictArts]

What version of Turborepo are you using?

1.5.1

What package manager are you using / does the bug impact?

npm

What operating system are you using?

Windows

Describe the Bug

After upgrading from 1.4.7 I now get a warning. Following the warning text leads to not a fix. The log is fine and I've removed the temp files, no fix.

Expected Behavior

No warning or turbod properly launching. If a warning must be shown, it should reflect failure to launch turbod, not contact it.

To Reproduce

Upgrade a 1.4.7 to 1.5.1

[nathanhammond]

In 1.5 we have toggled on using the daemon by default. On your machine the connection between the main turbo process and the daemon is failing.

This does not prevent your build from running to completion but we do need to identify why that connection is failing.

• Can you tell me more about your OS?
• Can you identify if you have write permissions to the path generated here? https://github.com/vercel/turborepo/blob/0dfa8e681c64a2bba9e8227d349b87f2f4bf7d04/cli/internal/daemon/daemon.go#L47 (we need better logging here to make this easier, you'd need to identify where Golang tempdir is creating a folder)
• what permissions does your user have?

[AddictArts]

@nathanhammond Thanks for the reply. How do you even start the daemon? I searched the docs and only the cache config mentions a daemon. I understand the principle, but I just upgraded, didn't make a change to turbo.json to run a daemon. Is this something like Gradle where it spins it up on first launch? Just wanted to mention that some confusion for me is the lack of docs.

I also found the option --no-daemon to turn the warning off. The issue though is that the process identified in turbo.pid doesn't exist. Not running. So, maybe it tries to launch a daemon and can't?

* Can you tell me more about your OS?

Edition Windows 10 Pro for Workstations
Version 21H2
Installed on ‎11/‎24/‎2021
OS build 19044.2006
Experience Windows Feature Experience Pack 120.2212.4180.0

* Can you identify if you have write permissions to the path generated here? https://github.com/vercel/turborepo/blob/0dfa8e681c64a2bba9e8227d349b87f2f4bf7d04/cli/internal/daemon/daemon.go#L47
   (we need better logging here to make this easier, you'd need to identify where Golang tempdir is creating a folder)

%HOME%\AppData\Local\Temp full permission. Also turbo creates a turbod folder there and turbod.pid and turbod.sock.

* what permissions does your user have?

For me that hexHash join is Temp\turbod\5ad724522893520a

I am fairly certain I have full permission. I run other go executables and go tools without issue. I can also open and edit the pid file. The sock file is 0 bytes and the pid file has a pid, but when I run that pid is not running, exluding --no-daemon. This makes the warning inaccurate. The log shows no error.

It should be WARNING: Failed to start turbod, because that seems to be what is happening. Something needs to go into the log to say why, which is also not happening.

[ringoz]

the same warning here on Windows 11

[gsoltis]

turbo daemon status should give a location for logs from the daemon. If you could post those, it would be helpful to figure out why the daemon is not running for is unable to be contacted.

[jayg-hive]

Hi there, also getting the same issue. On trying turbo daemon status I get this:

❯ pnpm exec turbo daemon status
Failed to contact daemon: connection to turbo daemon process failed. Please ensure the following:
        - the process identified by the pid in the file at /tmp/turbod/0f771c1e12c2b3b3/turbod.pid is not running, and remove /tmp/turbod/0f771c1e12c2b3b3/turbod.pid
        - check the logs at /home/jayg-hive/.local/share/turborepo/logs/0f771c1e12c2b3b3-Prosper-Ultimate.log
        - the unix domain socket at /tmp/turbod/0f771c1e12c2b3b3/turbod.sock has been removed
        You can also run without the daemon process by passing --no-daemon

I'm on WSL2 with the following details:

WSL version: 0.67.6.0
Kernel version: 5.15.62.1
WSLg version: 1.0.44
MSRDC version: 1.2.3401
Direct3D version: 1.606.4
DXCore version: 10.0.25131.1002-220531-1700.rs-onecore-base2-hyp
Windows version: 10.0.22621.601

Nothing out of the ordinary on the logs, they just say something about fsnotify events on my monorepo files:

2022-09-23T09:27:16.339+0800 [DEBUG] turbod.rpc server.GlobWatcher: Got fsnotify event {/home/jayg-hive/hive-gaming/**redacted**/packages/utils/.turbo/turbo-test.log 3}

[MariuzMM]

Same issue for me on Mac, today installed turbo for first time

 WARNING  failed to contact turbod. Continuing in standalone mode: connection to turbo daemon process failed. Please ensure the following:
        - the process identified by the pid in the file at /var/folders/b2/7xflcg550111jtksj2sb32n00000gn/T/turbod/5c8a39790d3101dc/turbod.pid is not running, and remove /var/folders/b2/7xflcg550111jtksj2sb32n00000gn/T/turbod/5c8a39790d3101dc/turbod.pid
        - check the logs at /Users/marius/Library/Application Support/turborepo/logs/5c8a39790d3101dc-test__turborepo.log
        - the unix domain socket at /var/folders/b2/7xflcg550111jtksj2sb32n00000gn/T/turbod/5c8a39790d3101dc/turbod.sock has been removed
        You can also run without the daemon process by passing --no-daemon

Error from log is

2022-09-23T10:48:48.558+0700 [ERROR] turbod: error: EXTRA_VALUE_AT_END="watching /Volumes/Media/Dev Projects/Projects/test__turborepo: timed out waiting for initial fsevents cookie: filewatching failed to start"

[gkn]

Gets the same warning message as @MariuzMM and @jayg-hive

Been using turbo for a while now. This warning started in version 1.5.2

On Windows10:

2022-09-23T14:16:10.471+0200 [ERROR] turbod: error: EXTRA_VALUE_AT_END="failed to lock the pid file at . Is another turbo daemon running?: Locked by other process"

No error on WSL2/Ubuntu :)

[himerus]

I'm seeing the same after a sweeping upgrade of dependencies today. Log output is a repeat of:

2022-09-23T09:42:53.645-0400 [ERROR] turbod: error: EXTRA_VALUE_AT_END="watching /Volumes/Work/code/MYPROJECT: timed out waiting for initial fsevents cookie: filewatching failed to start"

Console is outputting:

yarn turbo daemon status
yarn run v1.22.19
$ /Volumes/Work/code/MYPROJECT/node_modules/.bin/turbo daemon status
Failed to contact daemon: the daemon is not running
✨  Done in 0.32s.

Reverting and hard-coding to "turbo": "1.5.1" in package.json "solves" this issue.

[gsoltis]

Couple things going on:

• Minor, display-only bug related to EXTRA_VALUE_AT_END. I'll fix it, but that part is safe to ignore
• re: cookie timeouts, I'm going to bump it a little, but if it takes too long, it's likely better to for us skip filewatching.
• on that topic, the warning should be downgraded to informational. Filewatching is an optional optimization, and is not necessary for the proper functioning of turbo.

For those of you hitting the warning, can you confirm that turbo is still producing the correct results, just with the extra warning? Or is this actively breaking anyone? If so, I'd like to prioritize addressing breakage.

[gsoltis]

@gkn if you run turbo daemon status, it should give you the path to the .pid file? Can you check if there is in fact a process with that PID? Anything in the daemon logs (also from turbo daemon status) ?

[richardtorres314]

Running MacOS 12.6 here. When running the command turbo daemon status, I get "Failed to contact daemon: the daemon is not running." It's not preventing building for me, just giving the warning.

[gsoltis]

Addressing EXTRA_VALUE_AT_END: #2068

[gsoltis]

Additional context and debugging thoughts:

• turbo daemon status is not intended to start the daemon. If it's not running, that's the status and that's fine. The daemon also does not stay alive forever, it has an idle timeout. The turbo cli will attempt to restart it next time it needs it.
• running turbo daemon in your monorepo root will run the daemon in the foreground and log to the terminal.
• There are some additional daemon-management commands: restart, start, and stop. See turbo daemon --help

Also, to everyone, thanks for your patience while we iron out the edge cases on this. There may be platforms and scenarios where the daemon does not work or does not provide benefit, and we'll work to detect those and behave accordingly as we identify them.

[AddictArts]

Hopefully this will help. I run the daemon and no errors to the console etc. Looks good using turbo daemon.

When I run turbo run dev the turbod dies. Last log message is
2022-09-23T14:04:32.896-0700 [DEBUG] turbod.rpc server.GlobWatcher: Got fsnotify event {oranges\apps\web\.turbo\turbo-lint.log451477266 4}

[jayg-hive]

Couple things going on:

• Minor, display-only bug related to EXTRA_VALUE_AT_END. I'll fix it, but that part is safe to ignore
• re: cookie timeouts, I'm going to bump it a little, but if it takes too long, it's likely better to for us skip filewatching.
• on that topic, the warning should be downgraded to informational. Filewatching is an optional optimization, and is not necessary for the proper functioning of turbo.

For those of you hitting the warning, can you confirm that turbo is still producing the correct results, just with the extra warning? Or is this actively breaking anyone? If so, I'd like to prioritize addressing breakage.

Yup, no breakage whatsoever. It still runs the intended task. 👍

[gkn]

On Windows 10:

1. I cleared the logs and pid + cookie file and folders in the user folder.

I still get the same warning message but now get a log file without error messages, just a lot of fsnotify messages. It even includes folders that is not specified as part of npm workspaces. This seems odd and wasteful, especially since it will watch node_modules in that folder too:

2022-09-24T08:41:40.051+0200 [DEBUG] turbod.rpc server.fsnotify: watching directory C:\xxxx\tools\storybook\node_modules@mrmlnc

Initial startup seems slow.

Monorepo is relatively large, a few hundred folders with roughly 5000 source files.

2. Tried manual start of turbod:
   $ npx turbo daemon start
   connection to turbo daemon process failed. Please ensure the following:
   ...
   ...

3. Size of log file

I primarily use WSL2/Ubuntu for this project. I noticed that the size of the turbod log file was large. (roughly 1.5 million lines, 260 MB) in just a few turbo runs since yesterday. Seems odd.

[gsoltis]

Turning down daemon logging: #2085

@gkn Thanks for the details. Better scoping filewatching sounds like an area for improvement.

[mikecousins]

Everything still seems to work fine, but it throws warnings on every run.

[gsoltis]

The logging and EXTRA_VALUE_AT_END fixes are now available in turbo@1.5.4-canary.0

[klapec]

Same here, I'm running turbo@1.5.4 on arm macos 12.5.1.

 WARNING  failed to contact turbod. Continuing in standalone mode: connection to turbo daemon process failed. Please ensure the following:
        - the process identified by the pid in the file at /var/folders/ls/sq7g081s1g3cvtlz41qp8z2m0000gn/T/turbod/fe25f7293c19b4d8/turbod.pid is not running, and remove /var/folders/ls/sq7g081s1g3cvtlz41qp8z2m0000gn/T/turbod/fe25f7293c19b4d8/turbod.pid
        - check the logs at /Users/Andrzej_Klapec/Library/Application Support/turborepo/logs/fe25f7293c19b4d8-tui-frontend-monorepo.log
        - the unix domain socket at /var/folders/ls/sq7g081s1g3cvtlz41qp8z2m0000gn/T/turbod/fe25f7293c19b4d8/turbod.sock has been removed
        You can also run without the daemon process by passing --no-daemon

I ran yarn upgrade-interactive --latest to update turbo (previously I had it at 1.4.3) and ran the codemod npx @turbo/codemod migrate-env-var-dependencies which updated my turbo.json file and created a new .turbo-cookie file. Is the .turbo-cookie file necessary and should it be committed into the repository?

[unional]

Gets the same warning, and the log file is empty.

On Windows 11, turbo 1.5.4

[gsoltis]

@klapec the .turbo-cookie is not necessary, it is intended to be in the directory that turbo uses for config data, not your repository. Did it appear at the root of your repo? And was it created by the codemod?

[klapec]

@gsoltis Thanks for the response. It did appear at the root of my monorepo, I assume it was either created by upgrading turbo to 1.5.4 (postinstall?) or by the codemod. I haven't seen exactly when was it created, I noticed it when I was about to commit my changes to the repository.

[gsoltis]

@klapec got it. we're looking into how it could've ended up in that spot, but it's safe to delete. Sorry about that!

[gsoltis]

For the windows users in this thread: it looks like we have an issue somewhere between Go, windows support for AF_UNIX, and grpc. I'm tracking down the exact fix, but I expect that's the cause of being unable to use the daemon thus far

[gsoltis]

We've published turbo@1.5.5 which disables the daemon on windows while we sort out the AF_UNIX socket issue. I expect it will be re-enabled soon, as we've identified the issue and are working on a mitigation.

For those curious about the nitty-gritty details:

1. The Go grpc dialer uses url.Parse to determine how to connect to a server
2. Go, like many other platforms, requires that the Path portion of a url start with a leading /
3. GRPC uses the Path portion of a url for the location to Dial when the network is unix / url scheme is unix://
4. This doesn't work well on windows, where file paths do not start with a leading /
5. This means that we are either sending in a url that fails to parse (no leading '/'), or a url that parses, but has an extra leading '/'.

Not to worry though, we can patch it to do special handling on windows. Coming soon.

[gkn]

@gsoltis Tried out turbo@1.5.5 on both WSL2/Ubuntu and on the (disabled) Windows variant now. No noticeable issues - good work :)

If you have the time, could you please explain where we actually benefit from utilizing the daemon? I cannot say I notice any differences and could not find any documentation addressing that btw.

[ph55]

MacOS Ventura 13.2.1
Looks like permissions issue. I can start the daemon with sudo. Without the sudo - no success.

Another weird thing. Running turbo from nested project directory (no sudo) throws ERROR timeout while watchin directory: deadline has elapsed error:

But running it from "home" directory gives no error:

Demon pid is diffrrent from the first example (tmp folder)

[philippmossier]

I was able to fix it by deleting ~/.local/share/turborepo

[prodkt]

Just started happening here on MacOS and Windows machines. Seems this is actively being worked through though just confirming this is still happening as my time posting here.

[Muhammed-Alaa-Kanzari]

same thing is happening to me for the latest version 1.9.3
I can't build my project because of this

[zoffyzhang]

I was able to fix it by deleting ~/.local/share/turborepo

works for win10 by deleting C:\Users\zoffy\AppData\Local\Temp\turbod, because some files were locked

[ivancuric]

Issue still present on MacOS 13.3.1 (a), running under pnpm 8.5.1.

Temporarily fixed by removing the turbod directory inside /var/folders/

[iamhectorsosa]

Issue still present: MacOS 13.3.1

 WARNING  failed to contact turbod. Continuing in standalone mode: connection to turbo daemon process failed. Please ensure the following:
	- the process identified by the pid in the file at /var/folders/k2/44m5w6m54p5bw93yvqn4flwh0000gn/T/turbod/25b12ed5a16fdb56/turbod.pid is not running, and remove /var/folders/k2/44m5w6m54p5bw93yvqn4flwh0000gn/T/turbod/25b12ed5a16fdb56/turbod.pid
	- check the logs at /Users/hectorsosa/Library/Application Support/turborepo/logs/25b12ed5a16fdb56-screenshot.log
	- the unix domain socket at /var/folders/k2/44m5w6m54p5bw93yvqn4flwh0000gn/T/turbod/25b12ed5a16fdb56/turbod.sock has been removed
	You can also run without the daemon process by passing --no-daemon

[jhavej]

Issue still present: MacOS 13.3.1

 WARNING  failed to contact turbod. Continuing in standalone mode: connection to turbo daemon process failed. Please ensure the following:
	- the process identified by the pid in the file at /var/folders/k2/44m5w6m54p5bw93yvqn4flwh0000gn/T/turbod/25b12ed5a16fdb56/turbod.pid is not running, and remove /var/folders/k2/44m5w6m54p5bw93yvqn4flwh0000gn/T/turbod/25b12ed5a16fdb56/turbod.pid
	- check the logs at /Users/hectorsosa/Library/Application Support/turborepo/logs/25b12ed5a16fdb56-screenshot.log
	- the unix domain socket at /var/folders/k2/44m5w6m54p5bw93yvqn4flwh0000gn/T/turbod/25b12ed5a16fdb56/turbod.sock has been removed
	You can also run without the daemon process by passing --no-daemon

@ekqt I just resolved it by running turbo daemon status and turbo daemon restart (or turbo daemon run if the daemon is not running) -- macOS 13.3.1 (a)

[ivancuric]

I believe this is triggered when upgrading the turbo local dependency. I haven't had any success using the npx codemod, and upgrading it via pnpm locally still shows the codemod upgrade banner notification.

[Icoder9699]

Hello everyone! I just deleted the file that showed in the warn
example: C:\Users\AKARIM~1\AppData\Local\Temp\turbod\4800a0aaaf98de35\turbod.pid
and it helped

[Tarrowren]

If you run npx turbo daemon start, everything is fine.

If you run turbo run lint let it start the daemon on its own, the daemon exits within a few seconds of finishing and the pid file is not cleaned up.

[ivancuric]

@ekqt I just resolved it by running turbo daemon status and turbo daemon restart (or turbo daemon run if the daemon is not running) -- macOS 13.3.1 (a)

Getting the error ERROR unable to connect: unable to make handshake: bad grpc status code: Operation is not implemented or not supported for first 2 commands, and unrecognized subcommand 'run' for third.

[homer0]

I updated to 1.9.9 a couple of hours ago and started to see this issue. When trying to run the status command, I got the same message as @ivancuric: ERROR unable to connect: unable to make handshake: bad grpc status code: Operation is not implemented or not supported.

I first tried getting the process from the turbod.pid and killing it, but that didn't work, so I just deleted the turbod directory, called pnpm exec turbo daemon start (not restart, which throws because it's not running) and now is working 😀. Not sure if it was the right move tho :P

[aeswibon]

Hello everyone! I just deleted the file that showed in the warn example: C:\Users\AKARIM~1\AppData\Local\Temp\turbod\4800a0aaaf98de35\turbod.pid and it helped

I was also getting the error. By following the above instructions, I deleted the temp files in turbod directory. Now there is no error while using turbo

[ZeldOcarina]

If I try to clean and start the daemon I get:

cat: /var/folders/6x/px3zsgnn2rnbwmdmn3czfnym0000gn/T/turbod/027bccfcf592b126/turbod.pid: No such file or directory
% turbo daemon clean
Done
% turbo daemon clean
Done
% turbo daemon start
% turbo daemon status
ERROR unable to connect: daemon is not running

[the-baaron]

Same here, on MacOS (M2)

If I try to clean and start the daemon I get:

cat: /var/folders/6x/px3zsgnn2rnbwmdmn3czfnym0000gn/T/turbod/027bccfcf592b126/turbod.pid: No such file or directory
% turbo daemon clean
Done
% turbo daemon clean
Done
% turbo daemon start
% turbo daemon status
ERROR unable to connect: daemon is not running

[hellovai]

Same here, on MacOS (M2)

If I try to clean and start the daemon I get:

cat: /var/folders/6x/px3zsgnn2rnbwmdmn3czfnym0000gn/T/turbod/027bccfcf592b126/turbod.pid: No such file or directory
% turbo daemon clean
Done
% turbo daemon clean
Done
% turbo daemon start
% turbo daemon status
ERROR unable to connect: daemon is not running

I'm getting the exact same response

[jfrconley]

Same here, on MacOS (M2)

If I try to clean and start the daemon I get:

cat: /var/folders/6x/px3zsgnn2rnbwmdmn3czfnym0000gn/T/turbod/027bccfcf592b126/turbod.pid: No such file or directory
% turbo daemon clean
Done
% turbo daemon clean
Done
% turbo daemon start
% turbo daemon status
ERROR unable to connect: daemon is not running

I'm getting the exact same response

Also same

[aidenberzins]

Same here, on MacOS (M2)

If I try to clean and start the daemon I get:

cat: /var/folders/6x/px3zsgnn2rnbwmdmn3czfnym0000gn/T/turbod/027bccfcf592b126/turbod.pid: No such file or directory
% turbo daemon clean
Done
% turbo daemon clean
Done
% turbo daemon start
% turbo daemon status
ERROR unable to connect: daemon is not running

I'm getting the exact same response

Also same

+1 as well

[bengrunfeld]

One of the guys on my team solved this by deleting his package-lock.json and running the install again. That was what worked for him.

[chungweileong94]

One of the guys on my team solved this by deleting his package-lock.json and running the install again. That was what worked for him.

Delete lock file and reinstall could potentially break your application tho.

[GerroDen]

Since today i get that error regularly with v1.10.7
There has been no issue before today.
Deleting the file or running clean works, but the error will eventually come back.

[arlyon]

Hello folks, we have been shipping a host of daemon improvements (including a total rewrite) over the last year or so, so while I am not promising there are no issues, the set of issues that you can hit are completely different and I think that warrants opening a new ticket for the new daemon, since it is hard to filter in this thread between current problems and ones affecting the old implementation. Thank you all for chiming in and helping make turbo better, and don't hesitate to open a new issue if you are still having issues post 1.11.