New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

No exception / error logged though starting fails #36

Closed
webdevilopers opened this Issue Mar 10, 2018 · 13 comments

Comments

Projects
None yet
3 participants
@webdevilopers

webdevilopers commented Mar 10, 2018

We use capistrano for deployment. It seems to have some problems with symlinks or similar.
But we could break it down to the main problem.

When we config our supervisor path to e.g. tmp or below the Symfony kernel root dir everything works as expected and supervisor gets started:

rabbit_mq_supervisor:
    supervisor_instance_identifier:     "%supervisor_instance_identifier%"
    paths:
        #workspace_directory:            "/tmp/supervisor/%supervisor_instance_identifier%/"
        workspace_directory:            "%kernel.root_dir%/../../../supervisor/%supervisor_instance_identifier%/"

But if we keep the supervisor dir inside the app folder for instance the starting fails.

In both cases the files are created correctly.

When I run the command on the server with the non-working dir I don't get any response - no log entries - no process starting:

SYMFONY_ENV=prod app/console rabbitmq-supervisor:control start

Though the exception has been added in #22 there still seems to be something missing.

Is there a debug or trace argument I can use? Or some way to hack into this:

@webdevilopers

This comment has been minimized.

webdevilopers commented Mar 10, 2018

Just recognized we were running the following version:

"phobetor/rabbitmq-supervisor-bundle": "2.0.0",

It thought that release
https://github.com/Phobetor/rabbitmq-supervisor-bundle/releases/tag/2.0.0
would alread include the Logger seen here:
https://github.com/Phobetor/rabbitmq-supervisor-bundle/blob/master/Services/Supervisor.php#L98-L100

But it doesn't. I must use dev-master to have it?!

@webdevilopers

This comment has been minimized.

webdevilopers commented Mar 10, 2018

Finally after using dev-master I get some logs in Symfony:

[2018-03-10 10:18:27] app.DEBUG: Executing: supervisorctl --configuration=/var/www/vhosts/***/htdocs/releases/20180310091756/app/../supervisor/sps_intranet/supervisord.conf status [] []
[2018-03-10 10:18:27] app.CRITICAL: supervisorctl returns code: Misuse of shell builtins [] []
[2018-03-10 10:18:27] app.DEBUG: Output: error: <class 'socket.error'>, AF_UNIX path too long: file: /usr/lib64/python2.7/socket.py line: 224  [] []
[2018-03-10 10:18:27] console.ERROR: Exception thrown while running command "rabbitmq-supervisor:rebuild". Message: "Misuse of shell builtins" {"exception":"[object] (Phobetor\\RabbitMqSupervisorBundle\\Exception\\ProcessException(code: 2): Misuse of shell builtins at /var/www/vhosts/***/htdocs/releases/20180310091756/vendor/phobetor/rabbitmq-supervisor-bundle/Phobetor/RabbitMqSupervisorBundle/Services/Supervisor.php:67)","command":"rabbitmq-supervisor:rebuild","message":"Misuse of shell builtins"} []
[2018-03-10 10:18:27] console.ERROR: Command "rabbitmq-supervisor:rebuild" exited with code "2" {"command":"rabbitmq-supervisor:rebuild","code":2} []

The command:

supervisorctl --configuration=/var/www/vhosts/***/htdocs/releases/20180310100334/app/../supervisor/sps_intranet/supervisord.conf status

The error does not occur because of using "../". Since it does work when using "../../../".

The absolute path will fail too:

supervisorctl --configuration=/var/www/vhosts/***/htdocs/releases/20180310100334/supervisor/sps_intranet/supervisord.conf status

And again I want to note: The supervisor files are ALWAYS created. The path is correct. That means the permissions should be correct too.

Maybe it is not the path itself. But that the way it is configured causes this bundle to crash when trying to start supervisor.

@webdevilopers

This comment has been minimized.

webdevilopers commented Mar 10, 2018

Misuse of shell builtins is thrown by:

Similar error messages here:

And here where it looks like it also happens while deployment to @anacona16, @The-Don-Himself or @Turtlesfr?

@webdevilopers

This comment has been minimized.

webdevilopers commented Mar 10, 2018

At the bottom line it looks like the error occurs if the configured directories are none-permanent.
While a deployment with Capistrano a new folder is added under "releases". After deployment finished it is symlinked to "current".

Maybe the link does not exist for a short period of time - thought the absolute path inside the error looks correct and should work.

What I did not is that I moved the "supervisor" built files through other directories.
Event directories that would have worked as you can see in my original configuration above.

The error would still occur!

That makes me think that there is a problem inside the built files. I will post them here:

@webdevilopers

This comment has been minimized.

webdevilopers commented Mar 10, 2018

; supervisor config file

[unix_http_server]
file=/var/www/vhosts/***/htdocs/releases/20180310101225/app/../supervisor/sps_intranet/supervisord.sock   ; (the path to the socket file)
chmod=0700             ; sockef file mode (default 0700)

[supervisord]
logfile=/var/www/vhosts/***/20180310101225/app/../supervisor/sps_intranet/supervisord.log ; (main log file;default $CWD/supervisord.log)
pidfile=/var/www/vhosts/***/htdocs/releases/20180310101225/app/../supervisor/sps_intranet/supervisord.pid ; (supervisord pidfile;default supervisord.pid)

; the below section must remain in the config file for RPC
; (supervisorctl/web interface) to work, additional interfaces may be
; added by defining them in separate rpcinterface: sections
[rpcinterface:supervisor]
supervisor.rpcinterface_factory = supervisor.rpcinterface:make_main_rpcinterface

[supervisorctl]
serverurl=unix:///var/www/vhosts/***/htdocs/releases/20180310101225/app/../supervisor/sps_intranet/supervisord.sock ; use a unix:// URL  for a unix socket

; The [include] section can just contain the "files" setting.  This
; setting can list multiple files (separated by whitespace or
; newlines).  It can also contain wildcards.  The filenames are
; interpreted as relative to this file.  Included files *cannot*
; include files themselves.

[include]
files = /var/www/vhosts/***/htdocs/releases/20180310101225/app/../supervisor/sps_intranet/worker/*.conf

Anything strange so far?

@webdevilopers

This comment has been minimized.

webdevilopers commented Mar 10, 2018

Now i finally recognized that onky the .conf file was created. log, pid and .sock were missing.
That will cause the exception.

What I cannot tell is what causes them not to be created. Well I know that it does not work in newly created and later symlinked folders.

Maybe a permission problem while deployment?

@Phobetor

This comment has been minimized.

Owner

Phobetor commented May 10, 2018

Hi @webdevilopers

Thanks for all the information and first of all sorry for the confusion with 2.0.0 vs. dev-master. There is a new up-to-date version 3.0.0 out now.
You can use that version and console rabbitmq-supervisor:rebuild --wait-for-supervisord to get error output.

The .conf file is created by the commands of this bundle. The .log, .pid and .sock files are created by supervisord on start. If the latter do not exist the start of supervisord may have failed. It is hard to tell why that happened.

You write "it does work when using ../../../". That sounds like the other path is invalid for some reason. Maybe the directory does not exist or is not writable for the user that runs the command?

@Phobetor Phobetor added the question label May 18, 2018

@Phobetor

This comment has been minimized.

Owner

Phobetor commented Jun 2, 2018

As there is no new information on this issue and it should be fixed in 3.0.0 I will close it. Feel free to open a new one if there are still problems with the current version.

@Phobetor Phobetor closed this Jun 2, 2018

@webdevilopers

This comment has been minimized.

webdevilopers commented Jun 5, 2018

I tried to deploy with the new --wait-for-supervisord flag but get no new error messages.

The .conf file is created by the commands of this bundle. The .log, .pid and .sock files are created by supervisord on start.
If the latter do not exist the start of supervisord may have failed. It is hard to tell why that happened.

I agree. It really looks like a problem with the server process. The bundle looks fine actually.

Thanks for the help so far.

@webdevilopers

This comment has been minimized.

webdevilopers commented Jun 20, 2018

I could now break down and re-produce our issue.

When our capistrano deployment executed the rebuild command it gives no error and supervisor is not restarted on the remote server.
When I use your new flag "wait-for-supervisord" supervisor IS restarted. But capistrano freezes waiting for a response infinitely.

However similar happens if I execute the commands outside of the deployment directly on the server.
Without the flag supervisor is not started. With the flag it is start and it DOES NOT HANG UP.

The conflict seems to be running the rebuild while being in the capistrano deployment process.

Ever stumbled upon this with other deployment tools?

Maybe these consumer options could help?

What really disturbs me is the fact the the rebuild command doesn't even work on the server and supervisor is never restarted though the files are correctly rebuild.
The error logs are empty!

@vvasiloi

This comment has been minimized.

vvasiloi commented Nov 6, 2018

Just had the same issue: seems like it fails when it cannot read or write the configuration files.
I was running rabbitmq-supervisor:rebuild --wait-for-supervisord command with www-data user in a Docker container. Executing chown -R www-data:www-data on the configuration folder solved the issue.

@webdevilopers

This comment has been minimized.

webdevilopers commented Nov 8, 2018

Thanks @vvasiloi for your feedback. I no longer feel alone now! :)
TLDR I think we moved the configuration files to a different location where indeed no permission problems can occur. It worked out so far. Since then we never touched it again.

@vvasiloi

This comment has been minimized.

vvasiloi commented Nov 8, 2018

If the issue is caused by not being able to read/write configuration files then it should throw a specific error, not one like "Misuse of shell builtins".
The bundle should check if it can read from/write to the configuration folder before running the supporvisor commands.
Will try to reproduce the issue and come with a PR later.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment