pm2 cluster mode fails to start because Error: ENOENT, no such file or directory for process.cwd()

[yinrong]

same as #1244
This always happen when stop the old server, replace the source code, and start the new server.

One way to fix the problem is to kill PM2 and restart again. However, I can't do this, because I have to keep some processes running while stop/start the others.

[yinrong]

v0.9.1 v0.9.4 all have the same problem

[yinrong]

If I change to a complete new directory, "pm2 start" works just fine.

Once I remove and recreate one of the old directory, "pm2 start" hangs and logging this error. 😞

[yinrong]

I'm creating the directory with a new name every time to overcome this bug in PM2.

I want to know why, so could someone familiar with PM2 tell me where the problem might be?

[iam4x]

@yinrong Are you using NODE4.0.0 I'm seeing these issues also. I need to kill pm2 daemon...

[Unitech]

which hosting solution are you using?

[md2k]

Hi Guys, have 100% same problem with Capistrano deployment and PM2 Cluster mode.
i keep history of deployment , 5 releases +1 current total 6 folders, as soon as i do 7 deployment i hit this problem. i'm using json declaration, and all paths there full (not relative to avoid some issues).
I use:
node -v: v0.12.7
pm2 -v: 0.14.7
below my declaration:

 apps : [
    // First application
    {
      name                      : "MyApp",
      script                    : "app.js",
      cwd                       : "/home/deployment/APP/MyApp/production/current",
      log_date_format           : "YYYY-MM-DD HH:mm Z",
      error_file                : "/home/deployment/APP/MyApp/production/shared/logs/app_error.log",
      out_file                  : "/home/deployment/APP/MyApp/production/shared/logs/app_output.log",
      instances                 : 3,
      max_memory_restart        : "1024M",
      // cron_restart              : "1 0 * * *",
      watch                     : false,
      ignore_watch              : ["[\\/\\\\]\\./", "node_modules"],
      merge_logs                : false,
      exec_interpreter          : "node",
      exec_mode                 : "cluster",
      autorestart               : false,
      vizion                    : false,
      max_restarts              : 2,
      min_uptime                : "15s",
      next_gen_js               : false,  
      restart_delay             : 4000,
      env: {
        COMMON_VARIABLE: "true",
        NODE_ENV: "production",
        APP_PORT: 8443,
        APP_IP: "127.0.0.1"
      },
      env_production : {
        NODE_ENV: "production"
      }
    }
  ]

How looks pm2 show ID

Describing process with id 7 - name MyApp
┌───────────────────┬──────────────────────────────────────────────────────────────────────┐
│ status            │ stopped                                                              │
│ name              │ MyApp                                                              │
│ id                │ 7                                                                    │
│ path              │ /home/deployment/APP/MyApp/production/current/app.js           │
│ args              │                                                                      │
│ exec cwd          │ /home/deployment/APP/MyApp/production/current                      │
│ error log path    │ /home/deployment/APP/MyApp/production/shared/logs/app_error-7.log  │
│ out log path      │ /home/deployment/APP/MyApp/production/shared/logs/app_output-7.log │
│ pid path          │ /home/deployment/.pm2/pids/MyApp-7.pid                             │
│ mode              │ cluster_mode                                                         │
│ node v8 arguments │                                                                      │
│ watch & reload    │ ✘                                                                    │
│ interpreter       │ node                                                                 │
│ restarts          │ 0                                                                    │
│ unstable restarts │ 1                                                                    │
│ uptime            │ 0                                                                    │
│ created at        │ 2015-09-19T19:45:03.739Z                                             │
└───────────────────┴──────────────────────────────────────────────────────────────────────┘

Log in pm2.log:

[PM2] Tailing last 20 lines for [all] processes

PM2:     var cwd = process.cwd();
PM2:                       ^
PM2: Error: ENOENT, no such file or directory
PM2:     at Error (native)
PM2:     at Function.startup.resolveArgv0 (node.js:720:23)
PM2:     at startup (node.js:63:13)
PM2:     at node.js:814:3
PM2: 2015-09-19 21:40:54: App name:MyApp id:7 disconnected
PM2: 2015-09-19 21:40:54: App name:MyApp id:7 exited with code 1
PM2: 2015-09-19 21:45:03: Starting execution sequence in -cluster mode- for app name:MyApp id:7
PM2: node.js:720
PM2:     var cwd = process.cwd();
PM2:                       ^
PM2: Error: ENOENT, no such file or directory
PM2:     at Error (native)
PM2:     at Function.startup.resolveArgv0 (node.js:720:23)
PM2:     at startup (node.js:63:13)
PM2:     at node.js:814:3
PM2: 2015-09-19 21:45:03: App name:MyApp id:7 disconnected
PM2: 2015-09-19 21:45:03: App name:MyApp id:7 exited with code 1

[PM2] Streaming realtime logs for [all] processes

This happens only in cluster mode, fork work pretty well. This mean PM2 itself caching some folders locations from 1st deployment and when 7 deployment triggered 1st folder (which was initial) removed, then i can see this problem with ENOENT, frankly stack trace totally not informative :) i was not able locate those lines in any files (guess this is normal behaviour for JS in general).
@Unitech it will be awesome if we can locate and fix this issue.

As conclusion , i think this issue easy to duplicate if do next:
Use capistrano3 (i can give you my Capistrano project i use) or
create folder releases inside create folder 00001, drop code into it, symlink it to any location for example as ./current which pointing to 00001, npm install (actually tried to cached folder which have persistent location wont help), pm2 start ecosystem.json5, then create under releases new folder 00002
drop code there, change symlink current to this new folder, npm install to install dependencies, restart cluster ( but i use delete), and repeat this situation until you will have 6 releases, then create release 00007, point current symlink to this new release, remove 00001 folder, and try do pm2 delete ecosystem.json5, pm2 start ecosystem.json5 (didnt tested it with pm2 start path/to/app.js -i 3, because i should use ecosystem due project setup and multi env)

Those steps above always lead to error with ENOENT

[md2k]

Below folder structure for project which is under capistrano3 deployment tool

.
└── production
    ├── current -> /home/deployment/APP/MyApp/production/releases/20150919193256
    ├── releases
    │   ├── 20150919193205
    │   ├── 20150919193215
    │   ├── 20150919193232
    │   ├── 20150919193239
    │   ├── 20150919193249
    │   └── 20150919193256
    ├── repo
    │   ├── branches
    │   ├── hooks
    │   ├── info
    │   ├── objects
    │   └── refs
    └── shared
        ├── app_configs
        ├── logs
        └── node_modules

checked with lower history value and it is same, 2 releases + 1 current, 4th deployment hang with ENOENT error. This looks like problem exactly in PM2 and how it handling cluster mode :(
If release folder is same (like it with Pm2 deployment plugin) all is fine, but unfortunately pm2 deployment plugin very basic and not so flexible as capistrano :(

[iam4x]

@md2k Exact same set up, but it's weird because I'm having this issue with digitalocean, but not with my EC2 instances.

What system are you running?

[Jokero]

@md2k I use https://github.com/shipitjs/shipit (like Capistrano, written in JS) and have same issue but It is unpredictable

[md2k]

@iam4x it is dedicated servers, with Ubuntu 14.04 freshly installed, all 6 servers where i use PM2 have same issue. so it is 100% in PM2 side for cluster mode, it simply cached some path, and when folder removed simply unable to locate what it looking for, no other explanation. only for me really hard debug nodejs projects.
Hope @Unitech will check this issue soon.

@Jokero tried it before, but when i tried it, it was pretty buggy :) and as you said you also have issue, so we need to do something with PM2 and not jump over different deployment tools :)

[yinrong]

I'm using node v0.12.4. @iam4x

[Unitech]

Thanks for such detailed report,

It should come from here || here || here

[md2k]

Thanks for links @Unitech , i will check over weekends and post here my findings.

[md2k]

my first finding that i unable duplicate problem when pm2 started with --no-daemon option :(
what makes harder to debug it.

[md2k]

@Unitech seems issue not from those 3 locations you mentioned above.
Looking more, but if you have more ideas from where it can come, it will be good to know :)
By the way, issue still exist with PM2 0.15.6

[md2k]

unfortunately didn't found anything till now :(

[md2k]

Seems i found something.
By strace i found next:

wait4(29587, 0x7ffe9f7cec50, WNOHANG, NULL) = 0
wait4(30216, [{WIFEXITED(s) && WEXITSTATUS(s) == 1}], WNOHANG, NULL) = 30216
write(1, "2015-09-26 01:43:46: App name:St"..., 61) = 61
unlink("/home/deployment/.pm2/pids/Stream-1.pid") = -1 ENOENT (No such file or directory)
epoll_wait(5, {{EPOLLIN, {u32=21, u64=21}}}, 1024, 24156) = 1

It is failing trying unlink pid file for application ID 1. only not clear why it trying to do so if application already stopped ? o_0

@Unitech can you point me to part of code where PM2 cleaning pid files and other similar staff?

[md2k]

shoot me... when strace connected to pid i unable to duplicate issue.
i assume something wrong with some delay in events/loop or so, because on foreground mode and with strace all work perfect. maybe some events fired faster then they should be , and this broke functionality at some point?

P.S. by strace i still can see No such file or directory with unlink command, but at least it not fail.
so maybe issue not in unlink, will play more with strace

[md2k]

Not very informative...

[pid   797] getcwd(0x7ffe5b99d330, 4096) = -1 ENOENT (No such file or directory)
[pid   797] write(2, "node.js:720\n    var cwd = process.cwd();\n                      ^\nError: ENOENT, no such file or directory\n    at Error (native)\n    at Function.startup.resolveArgv0 (node.js:720:23)\n    at startup (node.js:63:13)\n    at node.js:814:3\n", 234) = 234

[md2k]

hm. one more finding:
when i kill PM2 and start it inside directory current (which is symlink) , then PM2 will have this weird issue, most probably PM2 some where convert symlink to realpath (or maybe not PM2 but NodeJs itself).
if i kill pm2 and run it outside of current folder (just from users home folder), then it looks like issue not exists anymore.
In my case, i always did tests like:

1. kill pm2
2. let capistrano start pm2 with application
   as result PM2 always been started from current App folder.

@Unitech any idea how it possible to avoid (except pre start PM2 outside of App folder)?
like allow PM2 start itself from specific location and then proceed with Application start ?

Long story short, it fail on getcwd(), duplicate issue possible even without nodejs.
create folder in terminal, cd into it, do cd . it should cd into same folder, open another window in terminal and remove folder, or replace it with same name folder and in 1st terminal do cd .
it will return cd: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory

I assume this what happens with PM2 when it started from application folder which is symlink and PM2 formally started from real folder. But this only assumption at this moment, need do more tests.

[md2k]

seems my assumption is correct:

started pm from current

production/current$ pm2 list
[PM2] Spawning PM2 daemon

after several deployments PM2 start throw error ENOENT

trying get list of apps while i in same directory:

production/current$ pm2 list
node.js:720
    var cwd = process.cwd();
                      ^
Error: ENOENT, no such file or directory
    at Error (native)
    at Function.startup.resolveArgv0 (node.js:720:23)
    at startup (node.js:63:13)
    at node.js:814:3

As you can see, ENOENT occurred even for pm2 list command (actually for any command from CLI)

Then i do cd ..; cd current/ in same directory (to refresh folder in terminal)
and now i can see list of failed apps:

production/current$ pm2 list
┌──────────┬────┬─────────┬─────┬─────────┬─────────┬────────┬────────┬──────────┐
│ App name │ id │ mode    │ pid │ status  │ restart │ uptime │ memory │ watching │
├──────────┼────┼─────────┼─────┼─────────┼─────────┼────────┼────────┼──────────┤
│ Stream   │ 1  │ cluster │ 0   │ stopped │ 0       │ 0      │ 0 B    │ disabled │

but unable restart them, most probably because PM2 Home dir still points to not-exists folder which was removed due history limit of releases by capistrano (or whatever deployment tool which similar to capistrano's folder layout)

Only it is not explain why with attached strace all was good, but frankly not remember from where i started PM2 yesterday during tracing process or no-damon option. But tested many times and if i pre-start PM2 outside symlink folder current - i not have this problem, if i let capistrano initial start of PM2 - i have this problem.

[md2k]

100% what i said in my last message is true for this kind of issue, question if we can some how programmatically lock PM2 during start to some specific folder without pre-start it manually outside of current release folder , PM2_HOME not affecting any how to this problem. i think issue in process.env.PWD for PM2 main process, question how to override it before start if it possible.
As other solution never do pm2 kill, and full initial PM2 start by deployment tool, because it starts PM2 in application folder and this lead to our issue, or instruct deployment tool to cd outside of application folder, start pm2 and only then cd back to app and run application.

@Unitech it is possible to add to PM2 one more command like 'pm2 up' which just starts PM2 manager process? currently do pm2 list to start manager :)

[yinrong]

@Unitech Hi, I think this is a major bug. Users have to kill PM2 to avoid this problem. In such way, most PM2 promising functionalities do not need to exist any more.

[iam4x]

@yinrong You can use the workaround from @md2k, spawn pm2 into the parent directory and reload from the parent directory as well. It works correctly for me.

[yinrong]

@iam4x thanks for noticing. It's really a good workaround.

[gansbrest]

Great finding @md2k we were just hit by the same issue. Using symlinks to current and after old build clean up pm2 starts to complain with:

 var path = (i >= 0) ? arguments[i] : process.cwd();
                                                 ^

Error: ENOENT: no such file or directory, uv_cwd
    at Error (native)
    at Object.posix.resolve (path.js:424:50)
    at Function.Module._resolveLookupPaths (module.js:250:17)
    at Function.Module._resolveFilename (module.js:317:31)
    at Function.Module._load (module.js:277:25)
    at Module.require (module.js:354:17)
    at require (internal/module.js:12:17)
    at Object.<anonymous> (/usr/local/lib/node_modules/pm2/lib/ProcessContainer.js:7:15)
    at Module._compile (module.js:398:26)
    at Object.Module._extensions..js (module.js:405:10)

Any progress on this issue? Seem pretty critical and dangerous, because you would think it works, roll to production boxes, start releasing and then suddenly whole thing breaks after 7th release and old builds cleanup..

[hguillermo]

We need a fix ASAP!!! Can we help? @soyuka?

[soyuka]

I won't say that it's a bug, it's just how cwd works. Proof:

mkdir testdir
cd testdir
node
# now in another terminal do rm -r testdir
> process.cwd()
Error: ENOENT: no such file or directory, uv_cwd
    at Error (native)
    at repl:1:9
    at REPLServer.defaultEval (repl.js:248:27)
    at bound (domain.js:280:14)
    at REPLServer.runBound [as eval] (domain.js:293:12)
    at REPLServer.<anonymous> (repl.js:412:12)
    at emitOne (events.js:82:20)
    at REPLServer.emit (events.js:169:7)
    at REPLServer.Interface._onLine (readline.js:210:10)
    at REPLServer.Interface._line (readline.js:549:8)

I'm willing to fix this but I don't see any solution, I can wrap a try/catch around process.cwd but then what?

[turneliusz]

Anybody solved the issue?

[gansbrest]

We ended up using upstart with chef to start intitial pm2 process on boot, before we run pm2 in the build folder. This approach seem to solve the problem, becaue pm2 is not tied to the build folder any more.

But as more genearl solution (@soyuka) I was thinking maybe we should change current dir to / if it's anything else but / ?

[soyuka]

You (users) are the only ones who're able to change the pm2 cwd because it's the directory from where pm2 is launched. So, doing pkill -f pm2; cd / && pm2 start might be a workaround for this.

[vjpr]

Just ran into this too. I am using shipit to deploy which symlinks the app dir to a release for rollback purposes. By default only the latest 2 releases stick around. So after 2 deploys the directory that pm2 was started in will be killed - causing the error.

So @soyuka's solution is the way to go. I think the return value is a failure though because it requires the user to specify an app.json file to start.

However, if you cd / && pm2 start /path/to/app.json, your app.json file will use relative paths from /. There is not a cwd CLI flag that I can find either.

This should be fixed in PM2 though.

[gansbrest]

Yes, but it's not really viable solution as most users will be unaware of this problem.. Still thinking for better and reliable approach/solution. At the moment we use upstart to start pm2 daemon on new box creation and then run pm2 within the apps, but it's all seems like walking on the thin ice..

[vjpr]

So it appears to happen anytime the directory of the configured app is deleted :( I think I have to move to forever.

EDIT

Just realized PM2 follows symlinks when recording the app dir, so instead of pm2 restart, I use delete then start config.json when working in a Capistrano-style environment.

[hguillermo]

@vjpr: In our case delete and start didn't work. After pm2 gets the error (the one in this thread) there is no way to make it work unless you kill the pm2 daemon. Also, delete and start doesn't take the advantage of a nice graceful reload.
@ganbrest solution seems the best way to solve this issue.

[hguillermo]

@gansbrest @vjpr I think our solution didn't work. It was okay for a few releases then back to the issue.

[gansbrest]

Yes, once the folder is gone - some issue. When we kill pm2, remove .pm2 folder and then issue /etc/init.d/pm2 start, it displays start message and pm2 process launches (with no processes), but later when we go into the app and issue reload command, pm2 seem to memorize cwd at that very moment. Is there a way to use main process ( the one started with upstart with empty process list) pwd? Any thoughts @soyuka?

[gansbrest]

Or maybe when we issue reload (and other commands) use fetch new cwd for started processes? Is it possible?

[soyuka]

I'm running a capistrano-like deployment too, and I'm not facing this issue. To avoid this cwd issue, as I said before, pm2 must not be started from the current repository if the linked directory might get removed.

This is why it could break:

1 - First deployment:

.
└── production
    ├── current -> /home/deployment/APP/MyApp/production/releases/20150919193256
    ├── releases
    │   ├── 20150919193256

$- cd  /home/deployment/APP/MyApp/production/current && pm2 start app.js

2 - A bit later, fifth deployment, keeping only 4 releases, the first deployment release will be destroyed and so, the pm2 cwd will not exist anymore and it'll throw the cwd error.

---

The fix, or at least what I'm using:

1 - First deployment:

.
└── production
    ├── current -> /home/deployment/APP/MyApp/production/releases/20150919193256
    ├── releases
    │   ├── 20150919193256

# I know this path will always exist through deployments:
$- cd  /home/deployment/APP/MyApp/production && pm2 start current/pm2.json 

Where app.json specifies the cwd (which is the current release symlink):

{
  "apps": [{
    "name": "my-app",
    "script": "./index.js",
    "cwd": "/home/deployment/APP/MyApp/production/current",
    "error_file": "./logs/app.err.log",
    "out_file": "./logs/out.err.log",
    "exec_mode": "fork_mode",
    "treekill": false,
    "vizion": false
  }]
}

FIY, I'm symlinking logs to the previous directory too, because I want to keep them accross different releases.

---

@gansbrest
Now about reload. It's not possible to change the cwd of the script with reload. It doesn't really act on the main process, in other terms it doesn't remove the fork from memory. The script cwd is set during the fork instanciation and we then only restart cluster child processes. On the opposite, doing a restart (with fork or cluster), will instanciate a new child process after deleting the old one, and there the cwd can change.

Are you facing two different issues? One related to pm2 loosing it's cwd the other one it's the process that's lost in the meanders of your hard drive?
If there's a case where your app has an unexisting cwd, you can simply restart the process (not reload).

[huangyingjie]

pm2 update solves.

[ThiagoMiranda]

@huangyingjie I had the same problem ( using a webpack env it was giving me a path is not a absolute path error ). Update solved it. It was a version issue?

[Magnie]

I had this same error pop up. I hadn't run the code for about a month (nor had I rebooted my laptop). I ran rm -rf ~/.pm2/ and that fixed it for me. I didn't do any in depth debugging and was just trying to get it to work. Here is the error I was getting:

Error: ENOENT: no such file or directory, uv_chdir
    at /usr/local/lib/node_modules/pm2/lib/ProcessContainer.js:299:13
    at /usr/local/lib/node_modules/pm2/node_modules/async/lib/async.js:52:16
    at /usr/local/lib/node_modules/pm2/node_modules/async/lib/async.js:1209:30
    at WriteStream.<anonymous> (/usr/local/lib/node_modules/pm2/lib/Utility.js:147:13)
    at emitOne (events.js:96:13)
    at WriteStream.emit (events.js:191:7)
    at WriteStream.<anonymous> (fs.js:2042:10)
    at FSReqWrap.oncomplete (fs.js:114:15)

Mac OS Sierra
In memory PM2 version: 2.4.4
Local PM2 version: 2.4.6
node -v: v7.7.3

Hopefully that helps someone.

[huangyingjie]

@ThiagoMiranda Not update the pm2 version, but the pm2 instance in memory, because some app was deleted.

node node_modules/.bin/pm2 delete someApp || true;
node node_modules/.bin/pm2 update || true;
node node_modules/.bin/pm2 start pm2.json