uWSGI unable to reload when the chdir is a symlink

[vigneshsarma]

In our case chdir is a symlink (/path/code/) to the actual code (/path/build/01), each time uwsgi reloads if it does not resolves it a new, this will keep using the old code.
Sample timeline:

* uwsgi --chdir path/current
  - path/current -> path/build/XX0
* New deploy
* -path/current -> path/build/XX1
* uwsgi reload
  - uwsgi still uses path/current -> path/build/XX0

uWSGI==2.0.6 via PIP

Options:

thunder-lock    = true
master          = true
module          = wsgi
buffer-size     = 65535
socket          = /tmp/app.sock
chmod-socket    = 666
processes       = 3
procname        ="[uwsgi:worker: app]"
procname-master ="[*uwsgi:master: app*]"
chdir           = /path/current
die-on-term     = true
home            = /path/venv

[unbit]

i am not able to reproduce it, can you report the real paths ? Maybe you are using some relative path in some specific way. Are you sure the link destination is changed before respawning uWSGI ?

Are you triggering a graceful reload ?

[vigneshsarma]

@unbit

1. All the paths I am using are full paths, No relative paths are used.
2. I am triggering a graceful reload using kill -HUP, using upstart.

If need be I can try to give you a sample app that replicates the problem.

Just adding more info

All the builds are in /opt/app/builds/<num> directory contains app code. Highest number in the builds directory has the lastest code which is symlinked to /opt/app/current. When the new code is deployed lets say the latest is 23, new directory 24 is created and /opt/app/current is symlinked to 24 which was previously symlinked to 23. The chdir option given to uWSGI is /opt/app/current.

[rdeioris]

All seems pretty standard, if you can build a sample app and a way to reproduce it will be very useful to understand what is going on. Thanks a lot

[vigneshsarma]

Check the script https://gist.github.com/vigneshsarma/f31fcffde04f26c44744

Check the comments in the script to see how to use it.

[unbit]

I have tried it, but the chdir you are seeing is the one used for "rebuilding" the original state when uWSGI was launched. If you start uWSGI from / (as an example, like when you run it as a system daemon) you will always it chdir to / before re-execing. The second chdir() (the one you configure in the ini) is then applied, and this time the link is respected and your app shoud load normally (unless something else is happening in the background, like something hold in the PYTHONPATH). Just for being sure i would print in my django wsgi.py the content of sys.path

[quard8]

Just came with same issue.

I'm using capistrano for deployment. Capistrano uses symlinks, so I have this structure on server:

/www/app/current -> /www/app/releases/5
                 -> /www/app/releases/4
                 -> /www/app/releases/3
                 -> /www/app/releases/2
                 -> /www/app/releases/1

After deploy, capistrano runs command:

ln -s /www/app/releases/6 /www/app/current

Then I touch file

touch /www/app/current/reload

I see that uwsgi reloading, but code still old.

uwsgi.ini

[uwsgi]
master = true
current_release = /www/app/current
chdir = %(current_release)/gazelle

gevent = 100
gevent-monkey-patch = true

http = :8888
processes = 10
pidfile = /www/app/current/tmp/pids/app.pid

reload-mercy = 5
harakiri = 10
module = app:get_app()
lazy = true
thunder-lock = true

env = APP_ENV=production

buffer-size = 32768
touch-reload = %(current_release)/reload

I have tried to change chdir to /www/app and etc, nothing helps. What I'm doing wrong?

[unbit]

@quard8 Do not use --lazy as it changes the way reloading works. Use --lazy-apps

[quard8]

Hm, still now working.

Changed to --lazy-apps in ini file, used --touch-reload, --touch-chain-reload, nothing works.

It works only if I manually change code in current path and then reload.

[unbit]

the problem is that chdir is called at the start of the process, (not of the worker) so in lazy/lazy-apps mode it will not be called during chain reload. It is a bit strange that touch-reload does not work, but maybe i am missing something.

Why you do not simply chdir in each worker using a post_fork hook ? (this will allow chain reloading)

Can you double check the touch-reload usage ? are you sure it happens after the /www/app/current/ re-linkage ?

[quard8]

Why you do not simply chdir in each worker using a post_fork hook ? (this will allow chain reloading)

I'm not sure I understand what you mean. I can't find post_fork hook in options, only in API. So you suggest to do like this?

from uwsgidecorators import postfork

@postfork
def get_app():
    from current import app
    return app

And in uwsgi config:

module = master_app:get_app()

Can you double check the touch-reload usage ? are you sure it happens after the /www/app/current/ re-linkage ?

Yes, everything is correct

NFO[0eb99de0] Running /usr/bin/env rm -rf /www/app/current on host
INFO[0eb99de0] Finished in 0.006 seconds with exit status 0 (successful).
INFO[843a6f64] Running /usr/bin/env ln -s /www/app/releases/20141024080544 /www/app/current on host
INFO[843a6f64] Finished in 0.006 seconds with exit status 0 (successful).
INFO[2a655ffa] Running /usr/bin/env touch  /www/app/current/reload on host
INFO[2a655ffa] Finished in 0.006 seconds with exit status 0 (successful).

[unbit]

Well in get_app() you should call os.chdir() to fix the environment, but yes this is the idea.

Btw in 2.0.8 (will be release tomorrow) i have added the --hook-post-fork option so you can do:

hook-post-fork = chdir:/www/app/current

[quard8]

Awesome! hook-post-fork works perfectly with capistrano.

Thanks!