Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Only register the first web-dispatch listener #3021

Merged
merged 1 commit into from
Jun 14, 2021

Conversation

lhoguin
Copy link
Contributor

@lhoguin lhoguin commented May 3, 2021

The problem: on certain double IPv4 and IPv6 environments, web-dispatch
will start a single listener (for management for example) but will
register two listeners in the Rabbit Mnesia table. Later, RabbitMQ
will crash when trying to stop or suspend them (rabbitmq-upgrade drain)
because only one of the two listeners exist in Ranch.

The fix: we register only the first listener, because that's the one
we used to create the Ranch ref (see rabbit_networking:ranch_ref/1).

Types of Changes

What types of changes does your code introduce to this project?
Put an x in the boxes that apply

  • Bug fix (non-breaking change which fixes issue #NNNN)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause an observable behavior change in existing systems)
  • Documentation improvements (corrections, new content, etc)
  • Cosmetic change (whitespace, formatting, etc)

Checklist

Put an x in the boxes that apply. You can also fill these out after creating
the PR. If you're unsure about any of them, don't hesitate to ask on the
mailing list. We're here to help! This is simply a reminder of what we are
going to look for before merging your code.

  • I have read the CONTRIBUTING.md document
  • I have signed the CA (see https://cla.pivotal.io/sign/rabbitmq)
  • All tests pass locally with my changes
  • I have added tests that prove my fix is effective or that my feature works
  • I have added necessary documentation (if appropriate)
  • Any dependent changes have been merged and published in related repositories

Further Comments

Present in at least 3.8.11, tested on Windows.

@dumbbell is all over the git blame for the original implementation so a review would be great. Also I am not sure this is the right fix, I am wondering if it worked before and if perhaps something else broke it. Or perhaps the drain command did not exist at the time.

RabbitMQ and Ranch disagreeing on what listeners are available:

PS C:\Windows\system32> rabbitmqctl.bat eval 'rabbit_networking:node_listeners(node()).'
[{listener,rabbit@WinDev2104Eval,http,"WinDev2104Eval",
           {0,0,0,0,0,0,0,0},
           15672,
           [{cowboy_opts,[{sendfile,false}]},{port,15672}]},
 {listener,rabbit@WinDev2104Eval,http,"WinDev2104Eval",
           {0,0,0,0},
           15672,
           [{cowboy_opts,[{sendfile,false}]},{port,15672}]},
 {listener,rabbit@WinDev2104Eval,clustering,"WinDev2104Eval",
           {0,0,0,0,0,0,0,0},
           25672,[]},
 {listener,rabbit@WinDev2104Eval,amqp,"WinDev2104Eval",
           {0,0,0,0,0,0,0,0},
           5672,
           [{backlog,128},
            {nodelay,true},
            {linger,{true,0}},
            {exit_on_close,false}]},
 {listener,rabbit@WinDev2104Eval,amqp,"WinDev2104Eval",
           {0,0,0,0},
           5672,
           [{backlog,128},
            {nodelay,true},
            {linger,{true,0}},
            {exit_on_close,false}]}]
PS C:\Windows\system32> rabbitmqctl.bat eval 'ets:tab2list(ranch_server).'
[{{addr,{acceptor,{0,0,0,0},5672}},{{0,0,0,0},5672}},
 {{addr,{acceptor,{0,0,0,0,0,0,0,0},5672}},{{0,0,0,0,0,0,0,0},5672}},
 {{addr,{acceptor,{0,0,0,0,0,0,0,0},15672}},{{0,0,0,0},15672}},
 {{conns_sup,{acceptor,{0,0,0,0},5672}},<11164.749.0>},
 {{conns_sup,{acceptor,{0,0,0,0,0,0,0,0},5672}},<11164.734.0>},
 {{conns_sup,{acceptor,{0,0,0,0,0,0,0,0},15672}},<11164.621.0>},
...

25672 is the distribution so it's expected that it's not in Ranch. 5672 has two in both cases. 15672 has one in Ranch and two in RabbitMQ: this is the problem.

The problem: on certain double IPv4 and IPv6 environments, web-dispatch
will start a single listener (for management for example) but will
register two listeners in the Rabbit Mnesia table. Later, RabbitMQ
will crash when trying to stop or suspend them (rabbitmq-upgrade drain)
because only one of the two listeners exist in Ranch.

The fix: we register only the first listener, because that's the one
we used to create the Ranch ref (see rabbit_networking:ranch_ref/1).
@lhoguin lhoguin marked this pull request as draft May 3, 2021 12:03
@lhoguin
Copy link
Contributor Author

lhoguin commented May 4, 2021

The same issue exists for the Prometheus listener. I will need to double check whether this PR fixes it as well.

@lhoguin lhoguin changed the title WIP: Only register the first web-dispatch listener Only register the first web-dispatch listener Jun 14, 2021
@lhoguin lhoguin marked this pull request as ready for review June 14, 2021 10:42
@lhoguin
Copy link
Contributor Author

lhoguin commented Jun 14, 2021

I have now tested that this fixes the original issue. Please review.

@michaelklishin michaelklishin merged commit c67e7e9 into master Jun 14, 2021
michaelklishin added a commit that referenced this pull request Jun 14, 2021
…ment-ipv4v6

Only register the first web-dispatch listener

(cherry picked from commit c67e7e9)
@michaelklishin michaelklishin deleted the fix-double-registration-management-ipv4v6 branch June 14, 2021 15:13
@michaelklishin
Copy link
Member

Backported to v3.8.x.

@michaelklishin michaelklishin added this to the 3.8.18 milestone Jun 14, 2021
This pull request was closed.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants