can't find warden.sock? #51

zhouxinyu · 2013-11-22T05:32:58Z

I can't find the warden.sock in "/var/vcap/data/warden/", so the warden component can't be started.

Anyone can help me?

yudai · 2013-11-25T06:25:39Z

Could you let me know some more information of your instllation?

zhouxinyu · 2013-11-25T06:29:55Z

Thanks for your reply. I'm trying to reinstall CF. If the problem still exists after I complete the installation, I will consult you again.

zhouxinyu · 2013-11-25T07:29:02Z

Hi,yudai.
The reinstallation can't solve this question. I just run this command to install cf:
"bash < <(curl -s -k -B https://raw.github.com/yudai/cf_nise_installer/${INSTALLER_BRANCH:-master}/local/bootstrap.sh)"

When i completed this command, I run "sudo ./local/start_processes.sh" to start it, then i got the first error:

Error 1): monit: action failed -- There is no service by that name(nats_stream_forwarder).

Then i delete "nats_stream_forwarder" in your script and rerun the command.
Unfortunately, The health_manager_next,cloud_controller_ng,cloud_controller_jobs,warden,dea_next can't be started.

When I changed "nats://nats:c1oudc0w@:4222" to "nats://nats:c1oudc0w@192.168.2.219:4222" in health_manager_next.yml, the health_manager_next can start normally.

Now let me paste the error logs of other four components.

2)warden.stderr.log:

rake aborted!
command exited with failure

Tasks: TOP => warden:start
(See full trace by running task with --trace)
rake aborted!
command exited with failure

3)cloud_controller_ng.stderr.log
/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/sequel- 3.48.0/lib/sequel/model/base.rb:1386:in save': total_routes presence (Sequel::ValidationFailed) from /var/vcap/packages/cloud_controller_ng/cloud_controller_ng/lib/sequel_plugins/vcap_serialization.rb:72:inupdate_from_hash'
from /var/vcap/data/packages/cloud_controller_ng/29/cloud_controller_ng/lib/cloud_controller/seeds.rb:14:in block (2 levels) in create_seed_quota_definitions' from /var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/sequel-3.48.0/lib/sequel/model/base.rb:1031:ininitialize'
from /var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/sequel-3.48.0/lib/sequel/model/base.rb:168:in new' from /var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/sequel-3.48.0/lib/sequel/model/base.rb:168:increate'
from /var/vcap/packages/cloud_controller_ng/cloud_controller_ng/lib/sequel_plugins/update_or_create.rb:9:in update_or_create' from /var/vcap/data/packages/cloud_controller_ng/29/cloud_controller_ng/lib/cloud_controller/seeds.rb:13:inblock in create_seed_quota_definitions'
from /var/vcap/data/packages/cloud_controller_ng/29/cloud_controller_ng/lib/cloud_controller/seeds.rb:12:in each' from /var/vcap/data/packages/cloud_controller_ng/29/cloud_controller_ng/lib/cloud_controller/seeds.rb:12:increate_seed_quota_definitions'
from /var/vcap/data/packages/cloud_controller_ng/29/cloud_controller_ng/lib/cloud_controller/seeds.rb:5:in write_seed_data' from /var/vcap/packages/cloud_controller_ng/cloud_controller_ng/lib/cloud_controller/runner.rb:114:inblock in run!'
from /var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/eventmachine-1.0.3/lib/eventmachine.rb:187:in call' from /var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/eventmachine-1.0.3/lib/eventmachine.rb:187:inrun_machine'
from /var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/eventmachine-1.0.3/lib/eventmachine.rb:187:in run' from /var/vcap/packages/cloud_controller_ng/cloud_controller_ng/lib/cloud_controller/runner.rb:105:inrun!'
from /var/vcap/packages/cloud_controller_ng/cloud_controller_ng/bin/cloud_controller:12:in `

'

4)jobs_work.stderr.log
rake aborted!
SIGTERM
/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/net-ssh- 2.7.0/lib/net/ssh/key_factory.rb:1:in <top (required)>' /var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/net-ssh- 2.7.0/lib/net/ssh/authentication/key_manager.rb:2:inrequire'
/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/net-ssh-2.7.0/lib/net/ssh/authentication/key_manager.rb:2:in <top (required)>' /var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/net-ssh-2.7.0/lib/net/ssh/authentication/session.rb:4:inrequire'
/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/net-ssh-2.7.0/lib/net/ssh/authentication/session.rb:4:in <top (required)>' /var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/net-ssh-2.7.0/lib/net/ssh.rb:11:inrequire'
/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/net-ssh-2.7.0/lib/net/ssh.rb:11:in <top (required)>' /var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/fog-1.16.0/lib/fog/joyent/compute.rb:4:inrequire'
/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/fog-1.16.0/lib/fog/joyent/compute.rb:4:in <top (required)>' /var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/fog-1.16.0/lib/fog/core/provider.rb:27:inrequire'
/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/fog-1.16.0/lib/fog/core/provider.rb:27:in service' /var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/fog-1.16.0/lib/fog/joyent.rb:7:inmodule:Joyent'
/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/fog-1.16.0/lib/fog/joyent.rb:4:in <module:Fog>' /var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/fog-1.16.0/lib/fog/joyent.rb:3:in<top (required)>'
/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/fog-1.16.0/lib/fog.rb:26:in require' /var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/fog-1.16.0/lib/fog.rb:26:in<top (required)>'
/var/vcap/data/packages/cloud_controller_ng/29/cloud_controller_ng/lib/cloud_controller/blobstore/blobstore.rb:3:in require' /var/vcap/data/packages/cloud_controller_ng/29/cloud_controller_ng/lib/cloud_controller/blobstore/blobstore.rb:3:in<top (required)>'
/var/vcap/data/packages/cloud_controller_ng/29/cloud_controller_ng/app/controllers/runtime/stagings_controller.rb:2:in require' /var/vcap/data/packages/cloud_controller_ng/29/cloud_controller_ng/app/controllers/runtime/stagings_controller.rb:2:in<top (required)>'
/var/vcap/data/packages/cloud_controller_ng/29/cloud_controller_ng/lib/cloud_controller/controllers.rb:4:in require' /var/vcap/data/packages/cloud_controller_ng/29/cloud_controller_ng/lib/cloud_controller/controllers.rb:4:inblock in <top (required)>'
/var/vcap/data/packages/cloud_controller_ng/29/cloud_controller_ng/lib/cloud_controller/controllers.rb:3:in each' /var/vcap/data/packages/cloud_controller_ng/29/cloud_controller_ng/lib/cloud_controller/controllers.rb:3:in<top (required)>'
/var/vcap/data/packages/cloud_controller_ng/29/cloud_controller_ng/lib/cloud_controller.rb:111:in require' /var/vcap/data/packages/cloud_controller_ng/29/cloud_controller_ng/lib/cloud_controller.rb:111:in<top (required)>'
/var/vcap/data/packages/cloud_controller_ng/29/cloud_controller_ng/Rakefile:7:in require' /var/vcap/data/packages/cloud_controller_ng/29/cloud_controller_ng/Rakefile:7:in<top (required)>'

5)dea.stderr.log
There is nothing in this file.

So,yudai,Can you give me some suggestions?

yudai · 2013-11-25T08:07:13Z

Thank you for the report
Did you restart your server before starting the processes? That's mandatory to use the linux namepsaces function, which is used by warden.

Error 1): monit: action failed -- There is no service by that name(nats_stream_forwarder).

This is a bug of the installer. I've fixed it in the testing branch and now I am testing it.
However, you can ignore this error for basic usage.

I'm not sure the cause of the error of CC yet. This might be solved by using the latest cf-release.
Could you please try export CF_RELEASE_USE_HEAD=yes?

zhouxinyu · 2013-11-25T08:51:05Z

Thanks for your reply!
I am sure I have restarted my server before starting the processes shell.

I found a error log in warden.log,like this:

{
"data": {
"stdout": " * Unloading AppArmor profiles\n ...done.\n",
"stderr": "quotacheck: lstat Cannot stat `//home/cfgroup/.gvfs': Permission denied\nGuess you'd better run fsck first !\nexiting...\nquotaon: Cannot find quota file on / [/dev/sda1] to turn quotas on/off.\nquotaon: Cannot find quota file on / [/dev/sda1] to turn quotas on/off.\n"
},
}

Then i run three commands:
$umount /home/yourname/.gvfs
$find . -inum 554009 -exec rm{} ;
$rm -rf .gvfs

Then the warden and dea_next components can run normally.

Also i found a error log in db_migrate.stderr.log:
PG::ConnectionBad: FATAL: could not open relation mapping file "global/pg_filenode.map": No such file or directory

I guess that this error is related to cloud_controller_ng?

yudai · 2013-11-26T09:24:48Z

I've pushed some commit to the testing branch. It should fix the problem of nats_stream_forwarder and health_manager.

I'm now investigating the problem of CCNG. I'll let you know when I figure out the cause.

zhouxinyu · 2013-11-26T10:16:25Z

Ok, Thank you very much,yudai!

yudai · 2013-11-27T06:55:00Z

I've pushed some commits to the master branch. It should work wit the latest cf-release. Please give it a try.

Digit-Elektra · 2013-11-27T09:48:19Z

Now the warden seems to bounce between failed:

Process 'nats' running
Process 'nats_stream_forwarder' running
Process 'health_manager_next' running
Process 'uaa' running
Process 'uaa_cf-registrar' running
Process 'cloud_controller_ng' running
Process 'cloud_controller_jobs' running
Process 'nginx_ccng' running
Process 'gorouter' running
Process 'warden' Execution failed
Process 'dea_next' initializing
Process 'dir_server' running
Process 'postgres' running
System 'system_ubuntu' running

and not monitored:

Process 'nats' running
Process 'nats_stream_forwarder' running
Process 'health_manager_next' running
Process 'uaa' running
Process 'uaa_cf-registrar' running
Process 'cloud_controller_ng' running
Process 'cloud_controller_jobs' running
Process 'nginx_ccng' running
Process 'gorouter' running
Process 'warden' not monitored
Process 'dea_next' not monitored
Process 'dir_server' running
Process 'postgres' running
System 'system_ubuntu' running

in the warden log, i can see:

Tasks: TOP => warden:start
(See full trace by running task with --trace)
rake aborted!
command exited with failure

The dea.stderr.log and dea.stdout.log is empty.
At the end of the dea_next.log:

{"timestamp":1385545368.31445,"message":"nats.message.received","log_level":"debug","source":"Dea::Nats","data":{"subject":"response to router.greet","data":{"id":"0fe940da5326453c42314d6f7f6f4f13","hosts":["192.168.128.133"],"minimumRegisterIntervalInSeconds":5}},"thread_id":16121980,"fiber_id":24397740,"process_id":4405,"file":"/var/vcap/packages/dea_next/lib/dea/nats.rb","lineno":144,"method":"handle_incoming_message"}
{"timestamp":1385545372.5409744,"message":"caught SIGTERM","log_level":"warn","source":"Dea::Bootstrap","data":{},"thread_id":16121980,"fiber_id":24397740,"process_id":4405,"file":"/var/vcap/packages/dea_next/lib/dea/bootstrap.rb","lineno":192,"method":"block (2 levels) in setup_signal_handlers"}
{"timestamp":1385545372.5411825,"message":"Shutting down","log_level":"info","source":"Dea::Bootstrap","data":{},"thread_id":16121980,"fiber_id":24397740,"process_id":4405,"file":"/var/vcap/packages/dea_next/lib/dea/bootstrap.rb","lineno":541,"method":"shutdown"}
{"timestamp":1385545372.5416038,"message":"All instances and staging tasks stopped, exiting.","log_level":"info","source":"Dea::Bootstrap","data":{},"thread_id":16121980,"fiber_id":24397740,"process_id":4405,"file":"/var/vcap/packages/dea_next/lib/dea/bootstrap.rb","lineno":555,"method":"block in shutdown"}
{"timestamp":1385545500.5391474,"message":"Dea started","log_level":"info","source":"Dea::Bootstrap","data":{},"thread_id":10064000,"fiber_id":18339800,"process_id":4754,"file":"/var/vcap/packages/dea_next/lib/dea/bootstrap.rb","lineno":133,"method":"setup_logging"}
{"timestamp":1385545500.5849876,"message":"nats.connecting","log_level":"info","source":"Dea::Nats","data":{"servers":["nats://nats:c1oudc0w@192.168.128.133:4222"]},"thread_id":10064000,"fiber_id":18339800,"process_id":4754,"file":"/var/vcap/packages/dea_next/lib/dea/nats.rb","lineno":99,"method":"create_nats_client"}
{"timestamp":1385545500.605978,"message":"nats.message.received","log_level":"debug","source":"Dea::Nats","data":{"subject":"response to router.greet","data":{"id":"7c7476d5b6a670078809e8b5d2d3f7b1","hosts":["192.168.128.133"],"minimumRegisterIntervalInSeconds":5}},"thread_id":10064000,"fiber_id":18339800,"process_id":4754,"file":"/var/vcap/packages/dea_next/lib/dea/nats.rb","lineno":144,"method":"handle_incoming_message"}
{"timestamp":1385545504.6554058,"message":"caught SIGTERM","log_level":"warn","source":"Dea::Bootstrap","data":{},"thread_id":10064000,"fiber_id":18339800,"process_id":4754,"file":"/var/vcap/packages/dea_next/lib/dea/bootstrap.rb","lineno":192,"method":"block (2 levels) in setup_signal_handlers"}
{"timestamp":1385545504.6556695,"message":"Shutting down","log_level":"info","source":"Dea::Bootstrap","data":{},"thread_id":10064000,"fiber_id":18339800,"process_id":4754,"file":"/var/vcap/packages/dea_next/lib/dea/bootstrap.rb","lineno":541,"method":"shutdown"}
{"timestamp":1385545504.65607,"message":"All instances and staging tasks stopped, exiting.","log_level":"info","source":"Dea::Bootstrap","data":{},"thread_id":10064000,"fiber_id":18339800,"process_id":4754,"file":"/var/vcap/packages/dea_next/lib/dea/bootstrap.rb","lineno":555,"method":"block in shutdown"}

Regards,

yudai · 2013-11-27T09:54:54Z

Please confirm that you have restarted your server.

Digit-Elektra · 2013-11-27T19:02:42Z

Hello,

yes i can confirm i restarted before trying.
I even installed from scratch on a new vm box.
What is even more strange is that after the second time I rebooted,
the system ubuntu server service diden't even start anymore.

Please let me know if i can do anything else.
I want to assist, but i'm afraid that my knowlegde doesn't reach far enough to find the root cause.

Regards,

jhiemer · 2013-11-27T21:49:18Z

@Digit-Elektra which Ubuntu version are you using?

yudai · 2013-11-28T04:53:16Z

Log of Warden process may have some helpful information. Please check /var/vcap/sys/log/monit directory for it.

zhouxinyu · 2013-11-28T08:49:01Z

Now,everything is ok in my server！

Thanks,yudai.

Digit-Elektra · 2013-12-08T15:41:28Z

Hello,

everything is working now. I reinstalled today from the trunk, and that worked!
For the OS, I am using Ubuntu 10.04 (64-bit).

Many thanks.

wzssyqa · 2014-04-07T06:46:30Z

I try installing on Debian Wheezy, and meet something similar, while after uninstall apparmor, it works well now.

wangzhezhe · 2014-05-26T02:13:04Z

Did anyone meet the same problem?(I'm using ubuntu 10.04 64bit and I already have restarted my server)
I'm a novice,could anyone give me some sugesttions?

yudai · 2014-05-26T02:44:04Z

What revision of cf_nise_installer are you using?

wangzhezhe · 2014-05-27T01:10:48Z

Thanks for your reply and this tools, it's the problem of my network, I finished my deployment processes.

yudai · 2014-05-27T01:37:11Z

So your CF works fine now?

ghost assigned yudai Nov 25, 2013

zhouxinyu mentioned this issue Nov 26, 2013

Nise Bosh not starting properly #52

Closed

yudai mentioned this issue Nov 26, 2013

Target refused connection problem. #43

Closed

chrismadison49 mentioned this issue Feb 23, 2014

cf log - bad credentials #66

Closed

yudai closed this as completed Jul 22, 2014

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

can't find warden.sock? #51

can't find warden.sock? #51

zhouxinyu commented Nov 22, 2013

yudai commented Nov 25, 2013

zhouxinyu commented Nov 25, 2013

zhouxinyu commented Nov 25, 2013

yudai commented Nov 25, 2013

zhouxinyu commented Nov 25, 2013

yudai commented Nov 26, 2013

zhouxinyu commented Nov 26, 2013

yudai commented Nov 27, 2013

Digit-Elektra commented Nov 27, 2013

yudai commented Nov 27, 2013

Digit-Elektra commented Nov 27, 2013

jhiemer commented Nov 27, 2013

yudai commented Nov 28, 2013

zhouxinyu commented Nov 28, 2013

Digit-Elektra commented Dec 8, 2013

wzssyqa commented Apr 7, 2014

wangzhezhe commented May 26, 2014

yudai commented May 26, 2014

wangzhezhe commented May 27, 2014

yudai commented May 27, 2014

can't find warden.sock? #51

can't find warden.sock? #51

Comments

zhouxinyu commented Nov 22, 2013

yudai commented Nov 25, 2013

zhouxinyu commented Nov 25, 2013

zhouxinyu commented Nov 25, 2013

yudai commented Nov 25, 2013

zhouxinyu commented Nov 25, 2013

yudai commented Nov 26, 2013

zhouxinyu commented Nov 26, 2013

yudai commented Nov 27, 2013

Digit-Elektra commented Nov 27, 2013

yudai commented Nov 27, 2013

Digit-Elektra commented Nov 27, 2013

jhiemer commented Nov 27, 2013

yudai commented Nov 28, 2013

zhouxinyu commented Nov 28, 2013

Digit-Elektra commented Dec 8, 2013

wzssyqa commented Apr 7, 2014

wangzhezhe commented May 26, 2014

yudai commented May 26, 2014

wangzhezhe commented May 27, 2014

yudai commented May 27, 2014