Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Don't crash listing listeners when a node is shutting down #6719

Merged
merged 1 commit into from Dec 20, 2022

Conversation

gomoripeti
Copy link
Contributor

Previously a management api request could crash as below while the overview page was open during a node restart.

crasher:
  initial call: cowboy_stream_h:request_process/3
  pid: <0.1902.0>
  registered_name: []
  exception error: bad argument
    in operator  ++/2
       called as {badrpc,
                  {'EXIT',
                   {badarg,
                    [{ets,match_object,
                      [rabbit_listener_ets,'_'],
                      [{error_info,
                        #{cause => id,module => erl_stdlib_errors}}]},
                     {ets,tab2list,1,[{file,"ets.erl"},{line,771}]}]}}}
                 ++
                 [{listener,'rabbit@<hostname>',http,
                      "<hostname>",
                      {0,0,0,0,0,0,0,0},
                      15672,
                      [{cowboy_opts,[{sendfile,false}]},
                       {ip,"0.0.0.0"},
                       {port,15672},
                       {ssl,false}]},
...
    in call from lists:append/1 (lists.erl, line 155)
    in call from rabbit_mgmt_wm_overview:listeners/0 (rabbit_mgmt_wm_overview.erl, line 111)
    in call from rabbit_mgmt_wm_overview:to_json/2 (rabbit_mgmt_wm_overview.erl, line 66)
    in call from cowboy_rest:call/3 (src/cowboy_rest.erl, line 1575)
    in call from cowboy_rest:set_resp_body/2 (src/cowboy_rest.erl, line 1464)
    in call from cowboy_rest:upgrade/4 (src/cowboy_rest.erl, line 284)
    in call from cowboy_stream_h:execute/3 (src/cowboy_stream_h.erl, line 300)

Proposed Changes

Please describe the big picture of your changes here to communicate to the RabbitMQ team why we should accept this pull request.
If it fixes a bug or resolves a feature request, be sure to link to that issue.

A pull request that doesn't explain why the change was made has a much lower chance of being accepted.

If English isn't your first language, don't worry about it and try to communicate the problem you are trying to solve to the best of your abilities.
As long as we can understand the intent, it's all good.

Types of Changes

What types of changes does your code introduce to this project?
Put an x in the boxes that apply

  • Bug fix (non-breaking change which fixes issue #NNNN)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause an observable behavior change in existing systems)
  • Documentation improvements (corrections, new content, etc)
  • Cosmetic change (whitespace, formatting, etc)
  • Build system and/or CI

Checklist

Put an x in the boxes that apply.
You can also fill these out after creating the PR.
If you're unsure about any of them, don't hesitate to ask on the mailing list.
We're here to help!
This is simply a reminder of what we are going to look for before merging your code.

  • I have read the CONTRIBUTING.md document
  • I have signed the CA (see https://cla.pivotal.io/sign/rabbitmq)
  • I have added tests that prove my fix is effective or that my feature works
  • All tests pass locally with my changes
  • If relevant, I have added necessary documentation to https://github.com/rabbitmq/rabbitmq-website
  • If relevant, I have added this change to the first version(s) in release-notes that I expect to introduce it

Further Comments

If this is a relatively large or complex change, kick off the discussion by explaining why you chose the solution you did and what alternatives you considered, etc.

Previously a management api request could crash as below while the overview
page was open during a node restart.

```
crasher:
  initial call: cowboy_stream_h:request_process/3
  pid: <0.1902.0>
  registered_name: []
  exception error: bad argument
    in operator  ++/2
       called as {badrpc,
                  {'EXIT',
                   {badarg,
                    [{ets,match_object,
                      [rabbit_listener_ets,'_'],
                      [{error_info,
                        #{cause => id,module => erl_stdlib_errors}}]},
                     {ets,tab2list,1,[{file,"ets.erl"},{line,771}]}]}}}
                 ++
                 [{listener,'rabbit@<hostname>',http,
                      "<hostname>",
                      {0,0,0,0,0,0,0,0},
                      15672,
                      [{cowboy_opts,[{sendfile,false}]},
                       {ip,"0.0.0.0"},
                       {port,15672},
                       {ssl,false}]},
...
    in call from lists:append/1 (lists.erl, line 155)
    in call from rabbit_mgmt_wm_overview:listeners/0 (rabbit_mgmt_wm_overview.erl, line 111)
    in call from rabbit_mgmt_wm_overview:to_json/2 (rabbit_mgmt_wm_overview.erl, line 66)
    in call from cowboy_rest:call/3 (src/cowboy_rest.erl, line 1575)
    in call from cowboy_rest:set_resp_body/2 (src/cowboy_rest.erl, line 1464)
    in call from cowboy_rest:upgrade/4 (src/cowboy_rest.erl, line 284)
    in call from cowboy_stream_h:execute/3 (src/cowboy_stream_h.erl, line 300)
```
@michaelklishin
Copy link
Member

Mixed version tests fail right now, AFAICT due to Bazel 6.0 changes.

@michaelklishin michaelklishin added this to the 3.11.6 milestone Dec 20, 2022
@michaelklishin michaelklishin merged commit 2ceb0b6 into rabbitmq:main Dec 20, 2022
michaelklishin added a commit that referenced this pull request Dec 20, 2022
Don't crash listing listeners when a node is shutting down (backport #6719)
@michaelklishin
Copy link
Member

This does not affect 3.10 and 3.9 as they do not use erpc to fetch node listeners.

@gomoripeti
Copy link
Contributor Author

indeed this only happens when feature flag listener_records_in_ets is enabled (introduced in 3.11). I forgot to specify that I encountered this after upgrading to 3.11.5. thank you.

@gomoripeti gomoripeti deleted the badrpc_listeners branch December 20, 2022 22:17
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants