New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

pm2 cluster mode fails to start because `Error: ENOENT, no such file or directory` for process.cwd() #1623

Closed
yinrong opened this Issue Sep 17, 2015 · 43 comments

Comments

@yinrong

yinrong commented Sep 17, 2015

same as #1244
This always happen when stop the old server, replace the source code, and start the new server.

One way to fix the problem is to kill PM2 and restart again. However, I can't do this, because I have to keep some processes running while stop/start the others.

@yinrong

This comment has been minimized.

Show comment
Hide comment
@yinrong

yinrong Sep 17, 2015

v0.9.1 v0.9.4 all have the same problem

yinrong commented Sep 17, 2015

v0.9.1 v0.9.4 all have the same problem

@yinrong

This comment has been minimized.

Show comment
Hide comment
@yinrong

yinrong Sep 17, 2015

If I change to a complete new directory, "pm2 start" works just fine.

Once I remove and recreate one of the old directory, "pm2 start" hangs and logging this error. 😞

yinrong commented Sep 17, 2015

If I change to a complete new directory, "pm2 start" works just fine.

Once I remove and recreate one of the old directory, "pm2 start" hangs and logging this error. 😞

@yinrong

This comment has been minimized.

Show comment
Hide comment
@yinrong

yinrong Sep 18, 2015

I'm creating the directory with a new name every time to overcome this bug in PM2.

I want to know why, so could someone familiar with PM2 tell me where the problem might be?

yinrong commented Sep 18, 2015

I'm creating the directory with a new name every time to overcome this bug in PM2.

I want to know why, so could someone familiar with PM2 tell me where the problem might be?

@iam4x

This comment has been minimized.

Show comment
Hide comment
@iam4x

iam4x Sep 18, 2015

@yinrong Are you using NODE4.0.0 I'm seeing these issues also. I need to kill pm2 daemon...

iam4x commented Sep 18, 2015

@yinrong Are you using NODE4.0.0 I'm seeing these issues also. I need to kill pm2 daemon...

@Unitech

This comment has been minimized.

Show comment
Hide comment
@Unitech

Unitech Sep 18, 2015

Owner

which hosting solution are you using?

Owner

Unitech commented Sep 18, 2015

which hosting solution are you using?

@md2k

This comment has been minimized.

Show comment
Hide comment
@md2k

md2k Sep 19, 2015

Hi Guys, have 100% same problem with Capistrano deployment and PM2 Cluster mode.
i keep history of deployment , 5 releases +1 current total 6 folders, as soon as i do 7 deployment i hit this problem. i'm using json declaration, and all paths there full (not relative to avoid some issues).
I use:
node -v: v0.12.7
pm2 -v: 0.14.7
below my declaration:

 apps : [
    // First application
    {
      name                      : "MyApp",
      script                    : "app.js",
      cwd                       : "/home/deployment/APP/MyApp/production/current",
      log_date_format           : "YYYY-MM-DD HH:mm Z",
      error_file                : "/home/deployment/APP/MyApp/production/shared/logs/app_error.log",
      out_file                  : "/home/deployment/APP/MyApp/production/shared/logs/app_output.log",
      instances                 : 3,
      max_memory_restart        : "1024M",
      // cron_restart              : "1 0 * * *",
      watch                     : false,
      ignore_watch              : ["[\\/\\\\]\\./", "node_modules"],
      merge_logs                : false,
      exec_interpreter          : "node",
      exec_mode                 : "cluster",
      autorestart               : false,
      vizion                    : false,
      max_restarts              : 2,
      min_uptime                : "15s",
      next_gen_js               : false,  
      restart_delay             : 4000,
      env: {
        COMMON_VARIABLE: "true",
        NODE_ENV: "production",
        APP_PORT: 8443,
        APP_IP: "127.0.0.1"
      },
      env_production : {
        NODE_ENV: "production"
      }
    }
  ]

How looks pm2 show ID

Describing process with id 7 - name MyApp
┌───────────────────┬──────────────────────────────────────────────────────────────────────┐
│ status            │ stopped                                                              │
│ name              │ MyApp                                                              │
│ id                │ 7                                                                    │
│ path              │ /home/deployment/APP/MyApp/production/current/app.js           │
│ args              │                                                                      │
│ exec cwd          │ /home/deployment/APP/MyApp/production/current                      │
│ error log path    │ /home/deployment/APP/MyApp/production/shared/logs/app_error-7.log  │
│ out log path      │ /home/deployment/APP/MyApp/production/shared/logs/app_output-7.log │
│ pid path          │ /home/deployment/.pm2/pids/MyApp-7.pid                             │
│ mode              │ cluster_mode                                                         │
│ node v8 arguments │                                                                      │
│ watch & reload    │ ✘                                                                    │
│ interpreter       │ node                                                                 │
│ restarts          │ 0                                                                    │
│ unstable restarts │ 1                                                                    │
│ uptime            │ 0                                                                    │
│ created at        │ 2015-09-19T19:45:03.739Z                                             │
└───────────────────┴──────────────────────────────────────────────────────────────────────┘

Log in pm2.log:

[PM2] Tailing last 20 lines for [all] processes

PM2:     var cwd = process.cwd();
PM2:                       ^
PM2: Error: ENOENT, no such file or directory
PM2:     at Error (native)
PM2:     at Function.startup.resolveArgv0 (node.js:720:23)
PM2:     at startup (node.js:63:13)
PM2:     at node.js:814:3
PM2: 2015-09-19 21:40:54: App name:MyApp id:7 disconnected
PM2: 2015-09-19 21:40:54: App name:MyApp id:7 exited with code 1
PM2: 2015-09-19 21:45:03: Starting execution sequence in -cluster mode- for app name:MyApp id:7
PM2: node.js:720
PM2:     var cwd = process.cwd();
PM2:                       ^
PM2: Error: ENOENT, no such file or directory
PM2:     at Error (native)
PM2:     at Function.startup.resolveArgv0 (node.js:720:23)
PM2:     at startup (node.js:63:13)
PM2:     at node.js:814:3
PM2: 2015-09-19 21:45:03: App name:MyApp id:7 disconnected
PM2: 2015-09-19 21:45:03: App name:MyApp id:7 exited with code 1

[PM2] Streaming realtime logs for [all] processes

This happens only in cluster mode, fork work pretty well. This mean PM2 itself caching some folders locations from 1st deployment and when 7 deployment triggered 1st folder (which was initial) removed, then i can see this problem with ENOENT, frankly stack trace totally not informative :) i was not able locate those lines in any files (guess this is normal behaviour for JS in general).
@Unitech it will be awesome if we can locate and fix this issue.

As conclusion , i think this issue easy to duplicate if do next:
Use capistrano3 (i can give you my Capistrano project i use) or
create folder releases inside create folder 00001, drop code into it, symlink it to any location for example as ./current which pointing to 00001, npm install (actually tried to cached folder which have persistent location wont help), pm2 start ecosystem.json5, then create under releases new folder 00002
drop code there, change symlink current to this new folder, npm install to install dependencies, restart cluster ( but i use delete), and repeat this situation until you will have 6 releases, then create release 00007, point current symlink to this new release, remove 00001 folder, and try do pm2 delete ecosystem.json5, pm2 start ecosystem.json5 (didnt tested it with pm2 start path/to/app.js -i 3, because i should use ecosystem due project setup and multi env)

Those steps above always lead to error with ENOENT

md2k commented Sep 19, 2015

Hi Guys, have 100% same problem with Capistrano deployment and PM2 Cluster mode.
i keep history of deployment , 5 releases +1 current total 6 folders, as soon as i do 7 deployment i hit this problem. i'm using json declaration, and all paths there full (not relative to avoid some issues).
I use:
node -v: v0.12.7
pm2 -v: 0.14.7
below my declaration:

 apps : [
    // First application
    {
      name                      : "MyApp",
      script                    : "app.js",
      cwd                       : "/home/deployment/APP/MyApp/production/current",
      log_date_format           : "YYYY-MM-DD HH:mm Z",
      error_file                : "/home/deployment/APP/MyApp/production/shared/logs/app_error.log",
      out_file                  : "/home/deployment/APP/MyApp/production/shared/logs/app_output.log",
      instances                 : 3,
      max_memory_restart        : "1024M",
      // cron_restart              : "1 0 * * *",
      watch                     : false,
      ignore_watch              : ["[\\/\\\\]\\./", "node_modules"],
      merge_logs                : false,
      exec_interpreter          : "node",
      exec_mode                 : "cluster",
      autorestart               : false,
      vizion                    : false,
      max_restarts              : 2,
      min_uptime                : "15s",
      next_gen_js               : false,  
      restart_delay             : 4000,
      env: {
        COMMON_VARIABLE: "true",
        NODE_ENV: "production",
        APP_PORT: 8443,
        APP_IP: "127.0.0.1"
      },
      env_production : {
        NODE_ENV: "production"
      }
    }
  ]

How looks pm2 show ID

Describing process with id 7 - name MyApp
┌───────────────────┬──────────────────────────────────────────────────────────────────────┐
│ status            │ stopped                                                              │
│ name              │ MyApp                                                              │
│ id                │ 7                                                                    │
│ path              │ /home/deployment/APP/MyApp/production/current/app.js           │
│ args              │                                                                      │
│ exec cwd          │ /home/deployment/APP/MyApp/production/current                      │
│ error log path    │ /home/deployment/APP/MyApp/production/shared/logs/app_error-7.log  │
│ out log path      │ /home/deployment/APP/MyApp/production/shared/logs/app_output-7.log │
│ pid path          │ /home/deployment/.pm2/pids/MyApp-7.pid                             │
│ mode              │ cluster_mode                                                         │
│ node v8 arguments │                                                                      │
│ watch & reload    │ ✘                                                                    │
│ interpreter       │ node                                                                 │
│ restarts          │ 0                                                                    │
│ unstable restarts │ 1                                                                    │
│ uptime            │ 0                                                                    │
│ created at        │ 2015-09-19T19:45:03.739Z                                             │
└───────────────────┴──────────────────────────────────────────────────────────────────────┘

Log in pm2.log:

[PM2] Tailing last 20 lines for [all] processes

PM2:     var cwd = process.cwd();
PM2:                       ^
PM2: Error: ENOENT, no such file or directory
PM2:     at Error (native)
PM2:     at Function.startup.resolveArgv0 (node.js:720:23)
PM2:     at startup (node.js:63:13)
PM2:     at node.js:814:3
PM2: 2015-09-19 21:40:54: App name:MyApp id:7 disconnected
PM2: 2015-09-19 21:40:54: App name:MyApp id:7 exited with code 1
PM2: 2015-09-19 21:45:03: Starting execution sequence in -cluster mode- for app name:MyApp id:7
PM2: node.js:720
PM2:     var cwd = process.cwd();
PM2:                       ^
PM2: Error: ENOENT, no such file or directory
PM2:     at Error (native)
PM2:     at Function.startup.resolveArgv0 (node.js:720:23)
PM2:     at startup (node.js:63:13)
PM2:     at node.js:814:3
PM2: 2015-09-19 21:45:03: App name:MyApp id:7 disconnected
PM2: 2015-09-19 21:45:03: App name:MyApp id:7 exited with code 1

[PM2] Streaming realtime logs for [all] processes

This happens only in cluster mode, fork work pretty well. This mean PM2 itself caching some folders locations from 1st deployment and when 7 deployment triggered 1st folder (which was initial) removed, then i can see this problem with ENOENT, frankly stack trace totally not informative :) i was not able locate those lines in any files (guess this is normal behaviour for JS in general).
@Unitech it will be awesome if we can locate and fix this issue.

As conclusion , i think this issue easy to duplicate if do next:
Use capistrano3 (i can give you my Capistrano project i use) or
create folder releases inside create folder 00001, drop code into it, symlink it to any location for example as ./current which pointing to 00001, npm install (actually tried to cached folder which have persistent location wont help), pm2 start ecosystem.json5, then create under releases new folder 00002
drop code there, change symlink current to this new folder, npm install to install dependencies, restart cluster ( but i use delete), and repeat this situation until you will have 6 releases, then create release 00007, point current symlink to this new release, remove 00001 folder, and try do pm2 delete ecosystem.json5, pm2 start ecosystem.json5 (didnt tested it with pm2 start path/to/app.js -i 3, because i should use ecosystem due project setup and multi env)

Those steps above always lead to error with ENOENT

@md2k

This comment has been minimized.

Show comment
Hide comment
@md2k

md2k Sep 19, 2015

Below folder structure for project which is under capistrano3 deployment tool

.
└── production
    ├── current -> /home/deployment/APP/MyApp/production/releases/20150919193256
    ├── releases
    │   ├── 20150919193205
    │   ├── 20150919193215
    │   ├── 20150919193232
    │   ├── 20150919193239
    │   ├── 20150919193249
    │   └── 20150919193256
    ├── repo
    │   ├── branches
    │   ├── hooks
    │   ├── info
    │   ├── objects
    │   └── refs
    └── shared
        ├── app_configs
        ├── logs
        └── node_modules

checked with lower history value and it is same, 2 releases + 1 current, 4th deployment hang with ENOENT error. This looks like problem exactly in PM2 and how it handling cluster mode :(
If release folder is same (like it with Pm2 deployment plugin) all is fine, but unfortunately pm2 deployment plugin very basic and not so flexible as capistrano :(

md2k commented Sep 19, 2015

Below folder structure for project which is under capistrano3 deployment tool

.
└── production
    ├── current -> /home/deployment/APP/MyApp/production/releases/20150919193256
    ├── releases
    │   ├── 20150919193205
    │   ├── 20150919193215
    │   ├── 20150919193232
    │   ├── 20150919193239
    │   ├── 20150919193249
    │   └── 20150919193256
    ├── repo
    │   ├── branches
    │   ├── hooks
    │   ├── info
    │   ├── objects
    │   └── refs
    └── shared
        ├── app_configs
        ├── logs
        └── node_modules

checked with lower history value and it is same, 2 releases + 1 current, 4th deployment hang with ENOENT error. This looks like problem exactly in PM2 and how it handling cluster mode :(
If release folder is same (like it with Pm2 deployment plugin) all is fine, but unfortunately pm2 deployment plugin very basic and not so flexible as capistrano :(

@iam4x

This comment has been minimized.

Show comment
Hide comment
@iam4x

iam4x Sep 20, 2015

@md2k Exact same set up, but it's weird because I'm having this issue with digitalocean, but not with my EC2 instances.

What system are you running?

iam4x commented Sep 20, 2015

@md2k Exact same set up, but it's weird because I'm having this issue with digitalocean, but not with my EC2 instances.

What system are you running?

@Jokero

This comment has been minimized.

Show comment
Hide comment
@Jokero

Jokero Sep 20, 2015

@md2k I use https://github.com/shipitjs/shipit (like Capistrano, written in JS) and have same issue but It is unpredictable

Jokero commented Sep 20, 2015

@md2k I use https://github.com/shipitjs/shipit (like Capistrano, written in JS) and have same issue but It is unpredictable

@md2k

This comment has been minimized.

Show comment
Hide comment
@md2k

md2k Sep 21, 2015

@iam4x it is dedicated servers, with Ubuntu 14.04 freshly installed, all 6 servers where i use PM2 have same issue. so it is 100% in PM2 side for cluster mode, it simply cached some path, and when folder removed simply unable to locate what it looking for, no other explanation. only for me really hard debug nodejs projects.
Hope @Unitech will check this issue soon.

@Jokero tried it before, but when i tried it, it was pretty buggy :) and as you said you also have issue, so we need to do something with PM2 and not jump over different deployment tools :)

md2k commented Sep 21, 2015

@iam4x it is dedicated servers, with Ubuntu 14.04 freshly installed, all 6 servers where i use PM2 have same issue. so it is 100% in PM2 side for cluster mode, it simply cached some path, and when folder removed simply unable to locate what it looking for, no other explanation. only for me really hard debug nodejs projects.
Hope @Unitech will check this issue soon.

@Jokero tried it before, but when i tried it, it was pretty buggy :) and as you said you also have issue, so we need to do something with PM2 and not jump over different deployment tools :)

@yinrong

This comment has been minimized.

Show comment
Hide comment
@yinrong

yinrong Sep 22, 2015

I'm using node v0.12.4. @iam4x

yinrong commented Sep 22, 2015

I'm using node v0.12.4. @iam4x

@Unitech

This comment has been minimized.

Show comment
Hide comment
@Unitech

Unitech Sep 23, 2015

Owner

Thanks for such detailed report,

It should come from here || here || here

Owner

Unitech commented Sep 23, 2015

Thanks for such detailed report,

It should come from here || here || here

@md2k

This comment has been minimized.

Show comment
Hide comment
@md2k

md2k Sep 24, 2015

Thanks for links @Unitech , i will check over weekends and post here my findings.

md2k commented Sep 24, 2015

Thanks for links @Unitech , i will check over weekends and post here my findings.

@md2k

This comment has been minimized.

Show comment
Hide comment
@md2k

md2k Sep 25, 2015

my first finding that i unable duplicate problem when pm2 started with --no-daemon option :(
what makes harder to debug it.

md2k commented Sep 25, 2015

my first finding that i unable duplicate problem when pm2 started with --no-daemon option :(
what makes harder to debug it.

@md2k

This comment has been minimized.

Show comment
Hide comment
@md2k

md2k Sep 25, 2015

@Unitech seems issue not from those 3 locations you mentioned above.
Looking more, but if you have more ideas from where it can come, it will be good to know :)
By the way, issue still exist with PM2 0.15.6

md2k commented Sep 25, 2015

@Unitech seems issue not from those 3 locations you mentioned above.
Looking more, but if you have more ideas from where it can come, it will be good to know :)
By the way, issue still exist with PM2 0.15.6

@md2k

This comment has been minimized.

Show comment
Hide comment
@md2k

md2k Sep 25, 2015

unfortunately didn't found anything till now :(

md2k commented Sep 25, 2015

unfortunately didn't found anything till now :(

@md2k

This comment has been minimized.

Show comment
Hide comment
@md2k

md2k Sep 25, 2015

Seems i found something.
By strace i found next:

wait4(29587, 0x7ffe9f7cec50, WNOHANG, NULL) = 0
wait4(30216, [{WIFEXITED(s) && WEXITSTATUS(s) == 1}], WNOHANG, NULL) = 30216
write(1, "2015-09-26 01:43:46: App name:St"..., 61) = 61
unlink("/home/deployment/.pm2/pids/Stream-1.pid") = -1 ENOENT (No such file or directory)
epoll_wait(5, {{EPOLLIN, {u32=21, u64=21}}}, 1024, 24156) = 1

It is failing trying unlink pid file for application ID 1. only not clear why it trying to do so if application already stopped ? o_0

@Unitech can you point me to part of code where PM2 cleaning pid files and other similar staff?

md2k commented Sep 25, 2015

Seems i found something.
By strace i found next:

wait4(29587, 0x7ffe9f7cec50, WNOHANG, NULL) = 0
wait4(30216, [{WIFEXITED(s) && WEXITSTATUS(s) == 1}], WNOHANG, NULL) = 30216
write(1, "2015-09-26 01:43:46: App name:St"..., 61) = 61
unlink("/home/deployment/.pm2/pids/Stream-1.pid") = -1 ENOENT (No such file or directory)
epoll_wait(5, {{EPOLLIN, {u32=21, u64=21}}}, 1024, 24156) = 1

It is failing trying unlink pid file for application ID 1. only not clear why it trying to do so if application already stopped ? o_0

@Unitech can you point me to part of code where PM2 cleaning pid files and other similar staff?

@md2k

This comment has been minimized.

Show comment
Hide comment
@md2k

md2k Sep 25, 2015

shoot me... when strace connected to pid i unable to duplicate issue.
i assume something wrong with some delay in events/loop or so, because on foreground mode and with strace all work perfect. maybe some events fired faster then they should be , and this broke functionality at some point?

P.S. by strace i still can see No such file or directory with unlink command, but at least it not fail.
so maybe issue not in unlink, will play more with strace

md2k commented Sep 25, 2015

shoot me... when strace connected to pid i unable to duplicate issue.
i assume something wrong with some delay in events/loop or so, because on foreground mode and with strace all work perfect. maybe some events fired faster then they should be , and this broke functionality at some point?

P.S. by strace i still can see No such file or directory with unlink command, but at least it not fail.
so maybe issue not in unlink, will play more with strace

@md2k

This comment has been minimized.

Show comment
Hide comment
@md2k

md2k Sep 26, 2015

Not very informative...

[pid   797] getcwd(0x7ffe5b99d330, 4096) = -1 ENOENT (No such file or directory)
[pid   797] write(2, "node.js:720\n    var cwd = process.cwd();\n                      ^\nError: ENOENT, no such file or directory\n    at Error (native)\n    at Function.startup.resolveArgv0 (node.js:720:23)\n    at startup (node.js:63:13)\n    at node.js:814:3\n", 234) = 234

md2k commented Sep 26, 2015

Not very informative...

[pid   797] getcwd(0x7ffe5b99d330, 4096) = -1 ENOENT (No such file or directory)
[pid   797] write(2, "node.js:720\n    var cwd = process.cwd();\n                      ^\nError: ENOENT, no such file or directory\n    at Error (native)\n    at Function.startup.resolveArgv0 (node.js:720:23)\n    at startup (node.js:63:13)\n    at node.js:814:3\n", 234) = 234
@md2k

This comment has been minimized.

Show comment
Hide comment
@md2k

md2k Sep 26, 2015

hm. one more finding:
when i kill PM2 and start it inside directory current (which is symlink) , then PM2 will have this weird issue, most probably PM2 some where convert symlink to realpath (or maybe not PM2 but NodeJs itself).
if i kill pm2 and run it outside of current folder (just from users home folder), then it looks like issue not exists anymore.
In my case, i always did tests like:

  1. kill pm2
  2. let capistrano start pm2 with application
    as result PM2 always been started from current App folder.

@Unitech any idea how it possible to avoid (except pre start PM2 outside of App folder)?
like allow PM2 start itself from specific location and then proceed with Application start ?

Long story short, it fail on getcwd(), duplicate issue possible even without nodejs.
create folder in terminal, cd into it, do cd . it should cd into same folder, open another window in terminal and remove folder, or replace it with same name folder and in 1st terminal do cd .
it will return cd: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory

I assume this what happens with PM2 when it started from application folder which is symlink and PM2 formally started from real folder. But this only assumption at this moment, need do more tests.

md2k commented Sep 26, 2015

hm. one more finding:
when i kill PM2 and start it inside directory current (which is symlink) , then PM2 will have this weird issue, most probably PM2 some where convert symlink to realpath (or maybe not PM2 but NodeJs itself).
if i kill pm2 and run it outside of current folder (just from users home folder), then it looks like issue not exists anymore.
In my case, i always did tests like:

  1. kill pm2
  2. let capistrano start pm2 with application
    as result PM2 always been started from current App folder.

@Unitech any idea how it possible to avoid (except pre start PM2 outside of App folder)?
like allow PM2 start itself from specific location and then proceed with Application start ?

Long story short, it fail on getcwd(), duplicate issue possible even without nodejs.
create folder in terminal, cd into it, do cd . it should cd into same folder, open another window in terminal and remove folder, or replace it with same name folder and in 1st terminal do cd .
it will return cd: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory

I assume this what happens with PM2 when it started from application folder which is symlink and PM2 formally started from real folder. But this only assumption at this moment, need do more tests.

@md2k

This comment has been minimized.

Show comment
Hide comment
@md2k

md2k Sep 26, 2015

seems my assumption is correct:

started pm from current

production/current$ pm2 list
[PM2] Spawning PM2 daemon

after several deployments PM2 start throw error ENOENT

trying get list of apps while i in same directory:

production/current$ pm2 list
node.js:720
    var cwd = process.cwd();
                      ^
Error: ENOENT, no such file or directory
    at Error (native)
    at Function.startup.resolveArgv0 (node.js:720:23)
    at startup (node.js:63:13)
    at node.js:814:3

As you can see, ENOENT occurred even for pm2 list command (actually for any command from CLI)

Then i do cd ..; cd current/ in same directory (to refresh folder in terminal)
and now i can see list of failed apps:

production/current$ pm2 list
┌──────────┬────┬─────────┬─────┬─────────┬─────────┬────────┬────────┬──────────┐
│ App name │ id │ mode    │ pid │ status  │ restart │ uptime │ memory │ watching │
├──────────┼────┼─────────┼─────┼─────────┼─────────┼────────┼────────┼──────────┤
│ Stream   │ 1  │ cluster │ 0   │ stopped │ 0       │ 0      │ 0 B    │ disabled │

but unable restart them, most probably because PM2 Home dir still points to not-exists folder which was removed due history limit of releases by capistrano (or whatever deployment tool which similar to capistrano's folder layout)

Only it is not explain why with attached strace all was good, but frankly not remember from where i started PM2 yesterday during tracing process or no-damon option. But tested many times and if i pre-start PM2 outside symlink folder current - i not have this problem, if i let capistrano initial start of PM2 - i have this problem.

md2k commented Sep 26, 2015

seems my assumption is correct:

started pm from current

production/current$ pm2 list
[PM2] Spawning PM2 daemon

after several deployments PM2 start throw error ENOENT

trying get list of apps while i in same directory:

production/current$ pm2 list
node.js:720
    var cwd = process.cwd();
                      ^
Error: ENOENT, no such file or directory
    at Error (native)
    at Function.startup.resolveArgv0 (node.js:720:23)
    at startup (node.js:63:13)
    at node.js:814:3

As you can see, ENOENT occurred even for pm2 list command (actually for any command from CLI)

Then i do cd ..; cd current/ in same directory (to refresh folder in terminal)
and now i can see list of failed apps:

production/current$ pm2 list
┌──────────┬────┬─────────┬─────┬─────────┬─────────┬────────┬────────┬──────────┐
│ App name │ id │ mode    │ pid │ status  │ restart │ uptime │ memory │ watching │
├──────────┼────┼─────────┼─────┼─────────┼─────────┼────────┼────────┼──────────┤
│ Stream   │ 1  │ cluster │ 0   │ stopped │ 0       │ 0      │ 0 B    │ disabled │

but unable restart them, most probably because PM2 Home dir still points to not-exists folder which was removed due history limit of releases by capistrano (or whatever deployment tool which similar to capistrano's folder layout)

Only it is not explain why with attached strace all was good, but frankly not remember from where i started PM2 yesterday during tracing process or no-damon option. But tested many times and if i pre-start PM2 outside symlink folder current - i not have this problem, if i let capistrano initial start of PM2 - i have this problem.

@md2k

This comment has been minimized.

Show comment
Hide comment
@md2k

md2k Sep 26, 2015

100% what i said in my last message is true for this kind of issue, question if we can some how programmatically lock PM2 during start to some specific folder without pre-start it manually outside of current release folder , PM2_HOME not affecting any how to this problem. i think issue in process.env.PWD for PM2 main process, question how to override it before start if it possible.
As other solution never do pm2 kill, and full initial PM2 start by deployment tool, because it starts PM2 in application folder and this lead to our issue, or instruct deployment tool to cd outside of application folder, start pm2 and only then cd back to app and run application.

@Unitech it is possible to add to PM2 one more command like 'pm2 up' which just starts PM2 manager process? currently do pm2 list to start manager :)

md2k commented Sep 26, 2015

100% what i said in my last message is true for this kind of issue, question if we can some how programmatically lock PM2 during start to some specific folder without pre-start it manually outside of current release folder , PM2_HOME not affecting any how to this problem. i think issue in process.env.PWD for PM2 main process, question how to override it before start if it possible.
As other solution never do pm2 kill, and full initial PM2 start by deployment tool, because it starts PM2 in application folder and this lead to our issue, or instruct deployment tool to cd outside of application folder, start pm2 and only then cd back to app and run application.

@Unitech it is possible to add to PM2 one more command like 'pm2 up' which just starts PM2 manager process? currently do pm2 list to start manager :)

@yinrong

This comment has been minimized.

Show comment
Hide comment
@yinrong

yinrong Oct 30, 2015

@Unitech Hi, I think this is a major bug. Users have to kill PM2 to avoid this problem. In such way, most PM2 promising functionalities do not need to exist any more.

yinrong commented Oct 30, 2015

@Unitech Hi, I think this is a major bug. Users have to kill PM2 to avoid this problem. In such way, most PM2 promising functionalities do not need to exist any more.

@iam4x

This comment has been minimized.

Show comment
Hide comment
@iam4x

iam4x Oct 30, 2015

@yinrong You can use the workaround from @md2k, spawn pm2 into the parent directory and reload from the parent directory as well. It works correctly for me.

iam4x commented Oct 30, 2015

@yinrong You can use the workaround from @md2k, spawn pm2 into the parent directory and reload from the parent directory as well. It works correctly for me.

@yinrong

This comment has been minimized.

Show comment
Hide comment
@yinrong

yinrong Nov 2, 2015

@iam4x thanks for noticing. It's really a good workaround.

yinrong commented Nov 2, 2015

@iam4x thanks for noticing. It's really a good workaround.

@gansbrest

This comment has been minimized.

Show comment
Hide comment
@gansbrest

gansbrest Jan 20, 2016

Great finding @md2k we were just hit by the same issue. Using symlinks to current and after old build clean up pm2 starts to complain with:

 var path = (i >= 0) ? arguments[i] : process.cwd();
                                                 ^

Error: ENOENT: no such file or directory, uv_cwd
    at Error (native)
    at Object.posix.resolve (path.js:424:50)
    at Function.Module._resolveLookupPaths (module.js:250:17)
    at Function.Module._resolveFilename (module.js:317:31)
    at Function.Module._load (module.js:277:25)
    at Module.require (module.js:354:17)
    at require (internal/module.js:12:17)
    at Object.<anonymous> (/usr/local/lib/node_modules/pm2/lib/ProcessContainer.js:7:15)
    at Module._compile (module.js:398:26)
    at Object.Module._extensions..js (module.js:405:10)

Any progress on this issue? Seem pretty critical and dangerous, because you would think it works, roll to production boxes, start releasing and then suddenly whole thing breaks after 7th release and old builds cleanup..

gansbrest commented Jan 20, 2016

Great finding @md2k we were just hit by the same issue. Using symlinks to current and after old build clean up pm2 starts to complain with:

 var path = (i >= 0) ? arguments[i] : process.cwd();
                                                 ^

Error: ENOENT: no such file or directory, uv_cwd
    at Error (native)
    at Object.posix.resolve (path.js:424:50)
    at Function.Module._resolveLookupPaths (module.js:250:17)
    at Function.Module._resolveFilename (module.js:317:31)
    at Function.Module._load (module.js:277:25)
    at Module.require (module.js:354:17)
    at require (internal/module.js:12:17)
    at Object.<anonymous> (/usr/local/lib/node_modules/pm2/lib/ProcessContainer.js:7:15)
    at Module._compile (module.js:398:26)
    at Object.Module._extensions..js (module.js:405:10)

Any progress on this issue? Seem pretty critical and dangerous, because you would think it works, roll to production boxes, start releasing and then suddenly whole thing breaks after 7th release and old builds cleanup..

@soyuka soyuka added the T: Bug label Jan 20, 2016

@hguillermo

This comment has been minimized.

Show comment
Hide comment
@hguillermo

hguillermo Jan 20, 2016

We need a fix ASAP!!! Can we help? @soyuka?

hguillermo commented Jan 20, 2016

We need a fix ASAP!!! Can we help? @soyuka?

@soyuka

This comment has been minimized.

Show comment
Hide comment
@soyuka

soyuka Jan 22, 2016

Collaborator

I won't say that it's a bug, it's just how cwd works. Proof:

mkdir testdir
cd testdir
node
# now in another terminal do rm -r testdir
> process.cwd()
Error: ENOENT: no such file or directory, uv_cwd
    at Error (native)
    at repl:1:9
    at REPLServer.defaultEval (repl.js:248:27)
    at bound (domain.js:280:14)
    at REPLServer.runBound [as eval] (domain.js:293:12)
    at REPLServer.<anonymous> (repl.js:412:12)
    at emitOne (events.js:82:20)
    at REPLServer.emit (events.js:169:7)
    at REPLServer.Interface._onLine (readline.js:210:10)
    at REPLServer.Interface._line (readline.js:549:8)

I'm willing to fix this but I don't see any solution, I can wrap a try/catch around process.cwd but then what?

Collaborator

soyuka commented Jan 22, 2016

I won't say that it's a bug, it's just how cwd works. Proof:

mkdir testdir
cd testdir
node
# now in another terminal do rm -r testdir
> process.cwd()
Error: ENOENT: no such file or directory, uv_cwd
    at Error (native)
    at repl:1:9
    at REPLServer.defaultEval (repl.js:248:27)
    at bound (domain.js:280:14)
    at REPLServer.runBound [as eval] (domain.js:293:12)
    at REPLServer.<anonymous> (repl.js:412:12)
    at emitOne (events.js:82:20)
    at REPLServer.emit (events.js:169:7)
    at REPLServer.Interface._onLine (readline.js:210:10)
    at REPLServer.Interface._line (readline.js:549:8)

I'm willing to fix this but I don't see any solution, I can wrap a try/catch around process.cwd but then what?

@Turneliusz

This comment has been minimized.

Show comment
Hide comment
@Turneliusz

Turneliusz Jan 28, 2016

Anybody solved the issue?

Turneliusz commented Jan 28, 2016

Anybody solved the issue?

@gansbrest

This comment has been minimized.

Show comment
Hide comment
@gansbrest

gansbrest Jan 28, 2016

We ended up using upstart with chef to start intitial pm2 process on boot, before we run pm2 in the build folder. This approach seem to solve the problem, becaue pm2 is not tied to the build folder any more.

But as more genearl solution (@soyuka) I was thinking maybe we should change current dir to / if it's anything else but / ?

gansbrest commented Jan 28, 2016

We ended up using upstart with chef to start intitial pm2 process on boot, before we run pm2 in the build folder. This approach seem to solve the problem, becaue pm2 is not tied to the build folder any more.

But as more genearl solution (@soyuka) I was thinking maybe we should change current dir to / if it's anything else but / ?

@soyuka

This comment has been minimized.

Show comment
Hide comment
@soyuka

soyuka Jan 28, 2016

Collaborator

You (users) are the only ones who're able to change the pm2 cwd because it's the directory from where pm2 is launched. So, doing pkill -f pm2; cd / && pm2 start might be a workaround for this.

Collaborator

soyuka commented Jan 28, 2016

You (users) are the only ones who're able to change the pm2 cwd because it's the directory from where pm2 is launched. So, doing pkill -f pm2; cd / && pm2 start might be a workaround for this.

@vjpr

This comment has been minimized.

Show comment
Hide comment
@vjpr

vjpr Jan 31, 2016

Just ran into this too. I am using shipit to deploy which symlinks the app dir to a release for rollback purposes. By default only the latest 2 releases stick around. So after 2 deploys the directory that pm2 was started in will be killed - causing the error.

So @soyuka's solution is the way to go. I think the return value is a failure though because it requires the user to specify an app.json file to start.

However, if you cd / && pm2 start /path/to/app.json, your app.json file will use relative paths from /. There is not a cwd CLI flag that I can find either.

This should be fixed in PM2 though.

vjpr commented Jan 31, 2016

Just ran into this too. I am using shipit to deploy which symlinks the app dir to a release for rollback purposes. By default only the latest 2 releases stick around. So after 2 deploys the directory that pm2 was started in will be killed - causing the error.

So @soyuka's solution is the way to go. I think the return value is a failure though because it requires the user to specify an app.json file to start.

However, if you cd / && pm2 start /path/to/app.json, your app.json file will use relative paths from /. There is not a cwd CLI flag that I can find either.

This should be fixed in PM2 though.

@gansbrest

This comment has been minimized.

Show comment
Hide comment
@gansbrest

gansbrest Jan 31, 2016

Yes, but it's not really viable solution as most users will be unaware of this problem.. Still thinking for better and reliable approach/solution. At the moment we use upstart to start pm2 daemon on new box creation and then run pm2 within the apps, but it's all seems like walking on the thin ice..

gansbrest commented Jan 31, 2016

Yes, but it's not really viable solution as most users will be unaware of this problem.. Still thinking for better and reliable approach/solution. At the moment we use upstart to start pm2 daemon on new box creation and then run pm2 within the apps, but it's all seems like walking on the thin ice..

@vjpr

This comment has been minimized.

Show comment
Hide comment
@vjpr

vjpr Feb 1, 2016

So it appears to happen anytime the directory of the configured app is deleted :( I think I have to move to forever.

EDIT

Just realized PM2 follows symlinks when recording the app dir, so instead of pm2 restart, I use delete then start config.json when working in a Capistrano-style environment.

vjpr commented Feb 1, 2016

So it appears to happen anytime the directory of the configured app is deleted :( I think I have to move to forever.

EDIT

Just realized PM2 follows symlinks when recording the app dir, so instead of pm2 restart, I use delete then start config.json when working in a Capistrano-style environment.

@hguillermo

This comment has been minimized.

Show comment
Hide comment
@hguillermo

hguillermo Feb 1, 2016

@vjpr: In our case delete and start didn't work. After pm2 gets the error (the one in this thread) there is no way to make it work unless you kill the pm2 daemon. Also, delete and start doesn't take the advantage of a nice graceful reload.
@ganbrest solution seems the best way to solve this issue.

hguillermo commented Feb 1, 2016

@vjpr: In our case delete and start didn't work. After pm2 gets the error (the one in this thread) there is no way to make it work unless you kill the pm2 daemon. Also, delete and start doesn't take the advantage of a nice graceful reload.
@ganbrest solution seems the best way to solve this issue.

@hguillermo

This comment has been minimized.

Show comment
Hide comment
@hguillermo

hguillermo Feb 2, 2016

@gansbrest @vjpr I think our solution didn't work. It was okay for a few releases then back to the issue.

hguillermo commented Feb 2, 2016

@gansbrest @vjpr I think our solution didn't work. It was okay for a few releases then back to the issue.

@gansbrest

This comment has been minimized.

Show comment
Hide comment
@gansbrest

gansbrest Feb 2, 2016

Yes, once the folder is gone - some issue. When we kill pm2, remove .pm2 folder and then issue /etc/init.d/pm2 start, it displays start message and pm2 process launches (with no processes), but later when we go into the app and issue reload command, pm2 seem to memorize cwd at that very moment. Is there a way to use main process ( the one started with upstart with empty process list) pwd? Any thoughts @soyuka?

gansbrest commented Feb 2, 2016

Yes, once the folder is gone - some issue. When we kill pm2, remove .pm2 folder and then issue /etc/init.d/pm2 start, it displays start message and pm2 process launches (with no processes), but later when we go into the app and issue reload command, pm2 seem to memorize cwd at that very moment. Is there a way to use main process ( the one started with upstart with empty process list) pwd? Any thoughts @soyuka?

@gansbrest

This comment has been minimized.

Show comment
Hide comment
@gansbrest

gansbrest Feb 2, 2016

Or maybe when we issue reload (and other commands) use fetch new cwd for started processes? Is it possible?

gansbrest commented Feb 2, 2016

Or maybe when we issue reload (and other commands) use fetch new cwd for started processes? Is it possible?

@soyuka

This comment has been minimized.

Show comment
Hide comment
@soyuka

soyuka Feb 3, 2016

Collaborator

I'm running a capistrano-like deployment too, and I'm not facing this issue. To avoid this cwd issue, as I said before, pm2 must not be started from the current repository if the linked directory might get removed.

This is why it could break:

1 - First deployment:

.
└── production
    ├── current -> /home/deployment/APP/MyApp/production/releases/20150919193256
    ├── releases
    │   ├── 20150919193256

$- cd  /home/deployment/APP/MyApp/production/current && pm2 start app.js

2 - A bit later, fifth deployment, keeping only 4 releases, the first deployment release will be destroyed and so, the pm2 cwd will not exist anymore and it'll throw the cwd error.


The fix, or at least what I'm using:

1 - First deployment:

.
└── production
    ├── current -> /home/deployment/APP/MyApp/production/releases/20150919193256
    ├── releases
    │   ├── 20150919193256

# I know this path will always exist through deployments:
$- cd  /home/deployment/APP/MyApp/production && pm2 start current/pm2.json 

Where app.json specifies the cwd (which is the current release symlink):

{
  "apps": [{
    "name": "my-app",
    "script": "./index.js",
    "cwd": "/home/deployment/APP/MyApp/production/current",
    "error_file": "./logs/app.err.log",
    "out_file": "./logs/out.err.log",
    "exec_mode": "fork_mode",
    "treekill": false,
    "vizion": false
  }]
}

FIY, I'm symlinking logs to the previous directory too, because I want to keep them accross different releases.


@gansbrest
Now about reload. It's not possible to change the cwd of the script with reload. It doesn't really act on the main process, in other terms it doesn't remove the fork from memory. The script cwd is set during the fork instanciation and we then only restart cluster child processes. On the opposite, doing a restart (with fork or cluster), will instanciate a new child process after deleting the old one, and there the cwd can change.

Are you facing two different issues? One related to pm2 loosing it's cwd the other one it's the process that's lost in the meanders of your hard drive?
If there's a case where your app has an unexisting cwd, you can simply restart the process (not reload).

Collaborator

soyuka commented Feb 3, 2016

I'm running a capistrano-like deployment too, and I'm not facing this issue. To avoid this cwd issue, as I said before, pm2 must not be started from the current repository if the linked directory might get removed.

This is why it could break:

1 - First deployment:

.
└── production
    ├── current -> /home/deployment/APP/MyApp/production/releases/20150919193256
    ├── releases
    │   ├── 20150919193256

$- cd  /home/deployment/APP/MyApp/production/current && pm2 start app.js

2 - A bit later, fifth deployment, keeping only 4 releases, the first deployment release will be destroyed and so, the pm2 cwd will not exist anymore and it'll throw the cwd error.


The fix, or at least what I'm using:

1 - First deployment:

.
└── production
    ├── current -> /home/deployment/APP/MyApp/production/releases/20150919193256
    ├── releases
    │   ├── 20150919193256

# I know this path will always exist through deployments:
$- cd  /home/deployment/APP/MyApp/production && pm2 start current/pm2.json 

Where app.json specifies the cwd (which is the current release symlink):

{
  "apps": [{
    "name": "my-app",
    "script": "./index.js",
    "cwd": "/home/deployment/APP/MyApp/production/current",
    "error_file": "./logs/app.err.log",
    "out_file": "./logs/out.err.log",
    "exec_mode": "fork_mode",
    "treekill": false,
    "vizion": false
  }]
}

FIY, I'm symlinking logs to the previous directory too, because I want to keep them accross different releases.


@gansbrest
Now about reload. It's not possible to change the cwd of the script with reload. It doesn't really act on the main process, in other terms it doesn't remove the fork from memory. The script cwd is set during the fork instanciation and we then only restart cluster child processes. On the opposite, doing a restart (with fork or cluster), will instanciate a new child process after deleting the old one, and there the cwd can change.

Are you facing two different issues? One related to pm2 loosing it's cwd the other one it's the process that's lost in the meanders of your hard drive?
If there's a case where your app has an unexisting cwd, you can simply restart the process (not reload).

soyuka added a commit to soyuka/pm2-hive.github.io that referenced this issue Mar 1, 2016

@soyuka soyuka closed this Mar 1, 2016

soyuka added a commit to pm2-hive/pm2-hive.github.io that referenced this issue Mar 1, 2016

soyuka added a commit to pm2-hive/pm2-hive.github.io that referenced this issue Mar 1, 2016

@huangyingjie

This comment has been minimized.

Show comment
Hide comment
@huangyingjie

huangyingjie Nov 22, 2016

pm2 update solves.

huangyingjie commented Nov 22, 2016

pm2 update solves.

@ThiagoMiranda

This comment has been minimized.

Show comment
Hide comment
@ThiagoMiranda

ThiagoMiranda Mar 6, 2017

@huangyingjie I had the same problem ( using a webpack env it was giving me a path is not a absolute path error ). Update solved it. It was a version issue?

ThiagoMiranda commented Mar 6, 2017

@huangyingjie I had the same problem ( using a webpack env it was giving me a path is not a absolute path error ). Update solved it. It was a version issue?

@Magnie

This comment has been minimized.

Show comment
Hide comment
@Magnie

Magnie May 5, 2017

I had this same error pop up. I hadn't run the code for about a month (nor had I rebooted my laptop). I ran rm -rf ~/.pm2/ and that fixed it for me. I didn't do any in depth debugging and was just trying to get it to work. Here is the error I was getting:

Error: ENOENT: no such file or directory, uv_chdir
    at /usr/local/lib/node_modules/pm2/lib/ProcessContainer.js:299:13
    at /usr/local/lib/node_modules/pm2/node_modules/async/lib/async.js:52:16
    at /usr/local/lib/node_modules/pm2/node_modules/async/lib/async.js:1209:30
    at WriteStream.<anonymous> (/usr/local/lib/node_modules/pm2/lib/Utility.js:147:13)
    at emitOne (events.js:96:13)
    at WriteStream.emit (events.js:191:7)
    at WriteStream.<anonymous> (fs.js:2042:10)
    at FSReqWrap.oncomplete (fs.js:114:15)

Mac OS Sierra
In memory PM2 version: 2.4.4
Local PM2 version: 2.4.6
node -v: v7.7.3

Hopefully that helps someone.

Magnie commented May 5, 2017

I had this same error pop up. I hadn't run the code for about a month (nor had I rebooted my laptop). I ran rm -rf ~/.pm2/ and that fixed it for me. I didn't do any in depth debugging and was just trying to get it to work. Here is the error I was getting:

Error: ENOENT: no such file or directory, uv_chdir
    at /usr/local/lib/node_modules/pm2/lib/ProcessContainer.js:299:13
    at /usr/local/lib/node_modules/pm2/node_modules/async/lib/async.js:52:16
    at /usr/local/lib/node_modules/pm2/node_modules/async/lib/async.js:1209:30
    at WriteStream.<anonymous> (/usr/local/lib/node_modules/pm2/lib/Utility.js:147:13)
    at emitOne (events.js:96:13)
    at WriteStream.emit (events.js:191:7)
    at WriteStream.<anonymous> (fs.js:2042:10)
    at FSReqWrap.oncomplete (fs.js:114:15)

Mac OS Sierra
In memory PM2 version: 2.4.4
Local PM2 version: 2.4.6
node -v: v7.7.3

Hopefully that helps someone.

@huangyingjie

This comment has been minimized.

Show comment
Hide comment
@huangyingjie

huangyingjie May 6, 2017

@ThiagoMiranda Not update the pm2 version, but the pm2 instance in memory, because some app was deleted.

node node_modules/.bin/pm2 delete someApp || true;
node node_modules/.bin/pm2 update || true;
node node_modules/.bin/pm2 start pm2.json

huangyingjie commented May 6, 2017

@ThiagoMiranda Not update the pm2 version, but the pm2 instance in memory, because some app was deleted.

node node_modules/.bin/pm2 delete someApp || true;
node node_modules/.bin/pm2 update || true;
node node_modules/.bin/pm2 start pm2.json
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment