Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Crash in channel for 3.8 / 3.9 mixed version cluster #5141

Closed
ansd opened this issue Jul 4, 2022 · 2 comments · Fixed by #5144
Closed

Crash in channel for 3.8 / 3.9 mixed version cluster #5141

ansd opened this issue Jul 4, 2022 · 2 comments · Fixed by #5144
Assignees
@ansd
Labels
bug
Milestone
3.10.6

Comments

@ansd
Copy link
Member

ansd commented Jul 4, 2022

Reported in https://rabbitmq.slack.com/archives/C1EDN83PA/p1656904564289129

Hello. I'm trying to upgrade a 3 node cluster. 3.8.19 w/erlang 23.3.1. I upgraded 1 node to 3.9.20 and it seems ok. But if I point a client to it, the updated node crashes.
The issue seems to be with routing a message to a quorum queue.

{function_clause,
     [{rabbit_channel,'-find_queue_name_from_quorum_name/2-fun-0-',
          [{resource,<<"prod">>,queue,<<"02.olaper.pickups">>},
           {ctx,rabbit_quorum_queue,
               {resource,<<"prod">>,queue,<<"02.olaper.pickups">>},
               {state,
                   {cfg,
                       {resource,<<"prod">>,queue,<<"02.olaper.pickups">>},
                       [{'prod_02.olaper.pickups','rabbit@prod02.internal'},
                        {'prod_02.olaper.pickups','rabbit@prod03.internal'},
                        {'prod_02.olaper.pickups','rabbit@prod01.internal'}],
                       32,#Fun<rabbit_quorum_queue.0.2401250>,
                       #Fun<rabbit_quorum_queue.1.2401250>,65000,0},
                   {'prod_02.olaper.pickups','rabbit@prod02.internal'},
                   go,3,0,4,false,#{},
                   #{1 =>
                         {12,
                          {enqueue,<0.8611.3>,2,
                              {basic_message,
                                  {resource,<<"prod">>,exchange,<<"main">>},
                                  [<<"02.v1.transformer.pickup.update">>],
                                  {content,60,none,
                                      <<48,0,0,0,0,0,1>>,
                                      rabbit_framing_amqp_0_9_1,
                                      [<<"c0a67a21-fb42-11ec-8f8a-95161b833ee6">>]},
                                  <<94,244,76,180,0,161,60,236,33,35,3,163,219,
                                    155,175,198>>,
                                  false}}},
                     2 =>
                         {18,
                          {enqueue,<0.8611.3>,3,
                              {basic_message,
                                  {resource,<<"prod">>,exchange,<<"main">>},
                                  [<<"02.v1.transformer.pickup.update">>],
                                  {content,60,none,
                                      <<48,0,0,0,0,0,1>>,
                                      rabbit_framing_amqp_0_9_1,
                                      [<<"c0a67a21-fb42-11ec-8f8a-95161b833ee6">>]},
                                  <<189,241,19,90,226,47,29,116,168,75,167,251,
                                    72,226,127,184>>,
                                  false}}}},
                   #{},undefined}},
           {resource,<<"prod">>,queue,<<"01.olaper.pickups">>}],
          [{file,"rabbit_channel.erl"},{line,2921}]},
      {maps,fold_1,3,[{file,"maps.erl"},{line,233}]},
      {rabbit_channel,handle_info,2,[{file,"rabbit_channel.erl"},{line,791}]},
      {gen_server2,handle_msg,2,[{file,"gen_server2.erl"},{line,1067}]},
      {proc_lib,wake_up,3,[{file,"proc_lib.erl"},{line,236}]}]}
     supervisor: {<0.8608.3>,rabbit_channel_sup}
     errorContext: child_terminated
     reason: {function_clause,
              [{rabbit_channel,'-find_queue_name_from_quorum_name/2-fun-0-',
                [{resource,<<"prod">>,queue,<<"02.olaper.pickups">>},
                 {ctx,rabbit_quorum_queue,
                  {resource,<<"prod">>,queue,<<"02.olaper.pickups">>},
                  {state,
                   {cfg,
                    {resource,<<"prod">>,queue,<<"02.olaper.pickups">>},
                    [{'prod_02.olaper.pickups','rabbit@prod02.internal'},
                     {'prod_02.olaper.pickups','rabbit@prod03.internal'},
                     {'prod_02.olaper.pickups','rabbit@prod01.internal'}],
                    32,#Fun<rabbit_quorum_queue.0.2401250>,
                    #Fun<rabbit_quorum_queue.1.2401250>,65000,0},
                   {'prod_02.olaper.pickups','rabbit@prod02.internal'},
                   go,3,0,4,false,#{},
                   #{1 =>
                      {12,
                       {enqueue,<0.8611.3>,2,
                        {basic_message,
                         {resource,<<"prod">>,exchange,<<"main">>},
                         [<<"02.v1.transformer.pickup.update">>],
                         {content,60,none,
                          <<48,0,0,0,0,0,1>>,
                          rabbit_framing_amqp_0_9_1,
                          [<<"c0a67a21-fb42-11ec-8f8a-95161b833ee6">>]},
                         <<94,244,76,180,0,161,60,236,33,35,3,163,219,155,175,
                           198>>,
                         false}}},
                     2 =>
                      {18,
                       {enqueue,<0.8611.3>,3,
                        {basic_message,
                         {resource,<<"prod">>,exchange,<<"main">>},
                         [<<"02.v1.transformer.pickup.update">>],
                         {content,60,none,
                          <<48,0,0,0,0,0,1>>,
                          rabbit_framing_amqp_0_9_1,
                          [<<"c0a67a21-fb42-11ec-8f8a-95161b833ee6">>]},
                         <<189,241,19,90,226,47,29,116,168,75,167,251,72,226,
                           127,184>>,
                         false}}}},
                   #{},undefined}},
                 {resource,<<"prod">>,queue,<<"01.olaper.pickups">>}],
                [{file,"rabbit_channel.erl"},{line,2921}]},
               {maps,fold_1,3,[{file,"maps.erl"},{line,233}]},
               {rabbit_channel,handle_info,2,
                [{file,"rabbit_channel.erl"},{line,791}]},
               {gen_server2,handle_msg,2,
                [{file,"gen_server2.erl"},{line,1067}]},
               {proc_lib,wake_up,3,[{file,"proc_lib.erl"},{line,236}]}]}
     offender: [{pid,<0.8611.3>},
                {id,channel},
                {mfargs,
                    {rabbit_channel,start_link,
                        [1,<0.8595.3>,<0.8609.3>,<0.8595.3>,
                         <<"app03:22520 -> prod03.internal:5672">>,
                         rabbit_framing_amqp_0_9_1,
                         {user,<<"transformer_prod">>,[],
                             [{rabbit_auth_backend_internal,none}]},
                         <<"prod">>,
                         [{<<"publisher_confirms">>,bool,true},
                          {<<"exchange_exchange_bindings">>,bool,true},
                          {<<"basic.nack">>,bool,true},
                          {<<"consumer_cancel_notify">>,bool,true},
                          {<<"connection.blocked">>,bool,true},
                          {<<"authentication_failure_close">>,bool,true}],
                         <0.8596.3>,<0.8610.3>]}},
                {restart_type,intrinsic},
                {shutdown,70000},
                {child_type,worker}]
     supervisor: {<0.8608.3>,rabbit_channel_sup}
     errorContext: shutdown
     reason: reached_max_restart_intensity
     offender: [{pid,<0.8611.3>},
                {id,channel},
                {mfargs,
                    {rabbit_channel,start_link,
                        [1,<0.8595.3>,<0.8609.3>,<0.8595.3>,
                         <<"app03:22520 -> prod03.internal:5672">>,
                         rabbit_framing_amqp_0_9_1,
                         {user,<<"transformer_prod">>,[],
                             [{rabbit_auth_backend_internal,none}]},
                         <<"prod">>,
                         [{<<"publisher_confirms">>,bool,true},
                          {<<"exchange_exchange_bindings">>,bool,true},
                          {<<"basic.nack">>,bool,true},
                          {<<"consumer_cancel_notify">>,bool,true},
                          {<<"connection.blocked">>,bool,true},
                          {<<"authentication_failure_close">>,bool,true}],
                         <0.8596.3>,<0.8610.3>]}},
                {restart_type,intrinsic},
                {shutdown,70000},
                {child_type,worker}]
@ansd ansd added the bug label Jul 4, 2022
@ansd ansd self-assigned this Jul 4, 2022
ansd added a commit that referenced this issue Jul 4, 2022
@ansd
Fix mixed version channel crash
a8442cc 
Fixes #5141
@ansd ansd mentioned this issue Jul 4, 2022
Merged
@ansd
Copy link
Member Author

ansd commented Jul 4, 2022

Can also be triggered when upgrading from 3.8 to 3.10

@michaelklishin michaelklishin added this to the 3.10.6 milestone Jul 4, 2022
mergify bot pushed a commit that referenced this issue Jul 4, 2022
@ansd @mergify
Fix mixed version channel crash
b72cd31 
Fixes #5141

(cherry picked from commit a8442cc)
mergify bot pushed a commit that referenced this issue Jul 4, 2022
@ansd @mergify
Fix mixed version channel crash
dfea8c6 
Fixes #5141

(cherry picked from commit a8442cc)
(cherry picked from commit b72cd31)
@michaelklishin
Copy link
Member

For better visibility: this will ship in 3.9.21 as well.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@ansd ansd

Labels
bug
Projects
None yet
Milestone
3.10.6
Development

Successfully merging a pull request may close this issue.

Fix mixed version channel crash rabbitmq/rabbitmq-server
2 participants
@michaelklishin @ansd