Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Failed to connect to rabbitmq many times during openstack deployemnt #153

Closed
wenchma opened this issue Oct 20, 2014 · 7 comments

Comments

Projects
None yet
4 participants
@wenchma
Copy link
Contributor

commented Oct 20, 2014

After delopy openstack successfully, open neutron server log
Expected result: In neutron server.log, it should to have such log to tell user you have to connect to AMQP server. " Connected to AMQP server on 127.0.0.1:5671". It should not have errors.
Actual result: In neutron server.log, it try many times to conenct to AMQP server, 10 mintues later, it connect to AMQP server finally.
Notes: Not only neutron have such issue, same as nova.etc.

/var/log/neutron/server.log

2014-08-15 01:36:33.112 19576 WARNING neutron.agent.securitygroups_rpc [-] Driver configuration doesn't match with enable_security_group
2014-08-15 01:36:33.187 19576 INFO oslo.messaging._drivers.impl_rabbit [-] Connecting to AMQP server on 127.0.0.1:5671
2014-08-15 01:36:33.198 19576 ERROR oslo.messaging._drivers.impl_rabbit [-] AMQP server on 127.0.0.1:5671 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds.
2014-08-15 01:36:34.200 19576 INFO oslo.messaging._drivers.impl_rabbit [-] Delaying reconnect for 1.0 seconds...
2014-08-15 01:36:35.202 19576 INFO oslo.messaging._drivers.impl_rabbit [-] Connecting to AMQP server on 127.0.0.1:5671
2014-08-15 01:36:35.214 19576 ERROR oslo.messaging._drivers.impl_rabbit [-] AMQP server on 127.0.0.1:5671 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 3 seconds.
2014-08-15 01:36:38.214 19576 INFO oslo.messaging._drivers.impl_rabbit [-] Delaying reconnect for 1.0 seconds...
2014-08-15 01:36:39.222 19576 INFO oslo.messaging._drivers.impl_rabbit [-] Connecting to AMQP server on 127.0.0.1:5671
2014-08-15 01:36:39.228 19576 ERROR oslo.messaging._drivers.impl_rabbit [-] AMQP server on 127.0.0.1:5671 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 5 seconds.
2014-08-15 01:36:44.232 19576 INFO oslo.messaging._drivers.impl_rabbit [-] Delaying reconnect for 1.0 seconds...
2014-08-15 01:36:45.234 19576 INFO oslo.messaging._drivers.impl_rabbit [-] Connecting to AMQP server on 127.0.0.1:5671
2014-08-15 01:36:45.240 19576 ERROR oslo.messaging._drivers.impl_rabbit [-] AMQP server on 127.0.0.1:5671 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 7 seconds.

@wenchma

This comment has been minimized.

Copy link
Contributor Author

commented Oct 20, 2014

It is due to delay to restart rabbitmq service, should notify the service to restart immediately to fix this issue.
Related line:
https://github.com/kennonkwok/rabbitmq/blob/master/recipes/default.rb#L157

@michaelklishin

This comment has been minimized.

Copy link
Member

commented Oct 20, 2014

While it may be desirable to notify with :immediate in OpenStack installations, you don't want to do it in every case. That said, RabbitMQ restarts are quick (unless you have broken message indices and millions of messages in durable queues), so it may be fine to do this for config changes.

@wenchma why don't you submit a pull request?

wenchma pushed a commit to wenchma/rabbitmq that referenced this issue Oct 29, 2014

@wenchma

This comment has been minimized.

Copy link
Contributor Author

commented Oct 29, 2014

@michaelklishin I have sumitted a pull request, #154 ,
can you have a review or make others also review ? thanks!

wenchma pushed a commit to wenchma/rabbitmq that referenced this issue Nov 14, 2014

@jjasghar

This comment has been minimized.

Copy link
Collaborator

commented Nov 26, 2014

These issues have been merged.

@jjasghar jjasghar closed this Nov 26, 2014

@zhang-jianwei

This comment has been minimized.

Copy link

commented Apr 18, 2018

2018-04-18 07:28:58.156 3125 ERROR oslo.messaging._drivers.impl_rabbit [-] AMQP server on 172.19.14.6:5673 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 5 seconds.
2018-04-18 07:28:58.231 3142 ERROR oslo.messaging._drivers.impl_rabbit [-] AMQP server on 172.19.14.6:5673 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 5 seconds.
2018-04-18 07:29:02.382 2834 INFO oslo.messaging._drivers.impl_rabbit [-] Reconnected to AMQP server on 172.19.14.8:5673
2018-04-18 07:29:02.415 3143 INFO oslo.messaging._drivers.impl_rabbit [-] Reconnected to AMQP server on 172.19.14.8:5673
2018-04-18 07:29:02.477 2834 ERROR oslo.messaging._drivers.impl_rabbit [-] AMQP server on 172.19.14.7:5673 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 5 seconds.

rabbitmq install as cluster, when restart the cluster, the openstack service will reconnect it, but neutron metadata service sometimes unreachable error, when try connection every node many times, it connected. this issue lead to neutron service instability.

@michaelklishin

This comment has been minimized.

Copy link
Member

commented Apr 18, 2018

This is not a support forum. "ECONNREFUSED" is pretty unambiguous: the client failed to open a TCP connection. Make sure that the broker is running on the target node, then follow the methodology recommended by RabbitMQ docs.

@rabbitmq rabbitmq locked and limited conversation to collaborators Apr 18, 2018

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
You can’t perform that action at this time.