Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Don't use 3.7.x-specific CLI flags on 3.6.x versions installed when no version was explicitly configured #508

Closed
jharbott opened this issue Mar 12, 2019 · 12 comments

Comments

Projects
None yet
2 participants
@jharbott
Copy link

commented Mar 12, 2019

Failure when running with 5.7.1:

[2019-03-12T13:39:51+00:00] FATAL: Mixlib::ShellOut::ShellCommandFailed: rabbitmq_plugin[rabbitmq_management] (openstack-ops-messaging::rabbitmq-server line 66) had an error: Mixlib::ShellOut::ShellCommandFailed:
 Expected process to exit with [0], but received '70'               
---- Begin output of rabbitmq-plugins list -q -e 'rabbitmq_management\b' ----                                                                                                                                      
STDOUT:                                                               
STDERR: Error: {'EXIT',                                                    
           {function_clause,                                                                                                                                                                                       
               [{rabbit_plugins_main,action,                                                                                                                                                                       
                    [list,'rabbit@controller-node11',                       
                     ["-q","rabbitmq_management\\b"],                                                                     
                     [{"-m",false},                                                                                                                                                                                
                      {"-n",'rabbit@controller-node11'},                     
                      {"-e",true},                                    
                      {"-E",false},                                                                                               
                      {"-v",false}],                                                                                                                                                                               
                     {cli,"/etc/rabbitmq/enabled_plugins",      
                         "/usr/lib/rabbitmq/plugins:/usr/lib/rabbitmq/lib/rabbitmq_server-3.6.10/plugins",              
                         [{plugin,amqp_client,"3.6.10",                                                                                                                                                            
                              "RabbitMQ AMQP Client",ez,[],                 
                              "/usr/lib/rabbitmq/lib/rabbitmq_server-3.6.10/plugins/amqp_client-3.6.10.ez"},
                          {plugin,cowboy,"1.0.4", 
                              "Small, fast, modular HTTP server.",ez,                                                                                                                                              
                              [cowlib],                           
                              "/usr/lib/rabbitmq/lib/rabbitmq_server-3.6.10/plugins/cowboy-1.0.4.ez"},                                                                                                             
                          {plugin,cowlib,"1.0.2",                                                                              
                              "Support library for manipulating Web protocols.",                                                                                                                                   
                              ez,[],                                        
                              "/usr/lib/rabbitmq/lib/rabbitmq_server-3.6.10/plugins/cowlib-1.0.2.ez"},                                                                                                             
                          {plugin,rabbitmq_amqp1_0,"3.6.10",                                                                
                              "AMQP 1.0 support for RabbitMQ",ez,                                                                                                                                                   
                              [amqp_client],                        
                              "/usr/lib/rabbitmq/lib/rabbitmq_server-3.6.10/plugins/rabbitmq_amqp1_0-3.6.10.ez"},                                                                                                  
                          {plugin,rabbitmq_auth_backend_ldap,"3.6.10",
                              "RabbitMQ LDAP Authentication Backend",ez,[],
                              "/usr/lib/rabbitmq/lib/rabbitmq_server-3.6.10/plugins/rabbitmq_auth_backend_ldap-3.6.10.ez"},                                                                                        
                          {plugin,rabbitmq_auth_mechanism_ssl,"3.6.10",                                                                                                                                            
                              "RabbitMQ SSL authentication (SASL EXTERNAL)",
                              ez,[],                                                                                      
                              "/usr/lib/rabbitmq/lib/rabbitmq_server-3.6.10/plugins/rabbitmq_auth_mechanism_ssl-3.6.10.ez"},                                                                                       
                          {plugin,rabbitmq_consistent_hash_exchange,"3.6.10",
                              "Consistent Hash Exchange Type",ez,[],  
                              "/usr/lib/rabbitmq/lib/rabbitmq_server-3.6.10/plugins/rabbitmq_consistent_hash_exchange-3.6.10.ez"},
                          {plugin,rabbitmq_event_exchange,"3.6.10",                                                                                                                                                
                              "Event Exchange Type",ez,[],      
                              "/usr/lib/rabbitmq/lib/rabbitmq_server-3.6.10/plugins/rabbitmq_event_exchange-3.6.10.ez"},
                          {plugin,rabbitmq_federation,"3.6.10",                                                                                                                                                    
                              "RabbitMQ Federation",ez,                     
                              [amqp_client],                                                                
                              "/usr/lib/rabbitmq/lib/rabbitmq_server-3.6.10/plugins/rabbitmq_federation-3.6.10.ez"},
                          {plugin,rabbitmq_federation_management,"3.6.10",                                                                                                                                         
                              "RabbitMQ Federation Management",ez,
                              [rabbitmq_management,rabbitmq_federation],                                                                                                                                           
                              "/usr/lib/rabbitmq/lib/rabbitmq_server-3.6.10/plugins/rabbitmq_federation_management-3.6.10.ez"},
                          {plugin,rabbitmq_jms_topic_exchange,"3.6.10",                                                                                                                                            
                              "RabbitMQ JMS topic selector exchange plugin",
                              ez,[],                                                                                                                                                                               
                              "/usr/lib/rabbitmq/lib/rabbitmq_server-3.6.10/plugins/rabbitmq_jms_topic_exchange-3.6.10.ez"},
                          {plugin,rabbitmq_management,"3.6.10",                                                                                                                                                     
                              "RabbitMQ Management Console",ez,     
                              [amqp_client,cowboy,cowlib,                                                                                                                                                          
                               rabbitmq_web_dispatch,                 
                               rabbitmq_management_agent],                 
                              "/usr/lib/rabbitmq/lib/rabbitmq_server-3.6.10/plugins/rabbitmq_management-3.6.10.ez"},                                                                                               
                          {plugin,rabbitmq_management_agent,"3.6.10",                                                                                                                                              
                              "RabbitMQ Management Agent",ez,[],            
                              "/usr/lib/rabbitmq/lib/rabbitmq_server-3.6.10/plugins/rabbitmq_management_agent-3.6.10.ez"},
                          {plugin,rabbitmq_management_visualiser,"3.6.10",                                                                                                                                         
                              "RabbitMQ Visualiser",ez,                      
                              [rabbitmq_management],                  
                              "/usr/lib/rabbitmq/lib/rabbitmq_server-3.6.10/plugins/rabbitmq_management_visualiser-3.6.10.ez"},   
                          {plugin,rabbitmq_mqtt,"3.6.10",                                                                                                                                                          
                              "RabbitMQ MQTT Adapter",ez,       
                              [amqp_client],                                                                            
                              "/usr/lib/rabbitmq/lib/rabbitmq_server-3.6.10/plugins/rabbitmq_mqtt-3.6.10.ez"},                                                                                                     
                          {plugin,rabbitmq_recent_history_exchange,"3.6.10",
                              "RabbitMQ Recent History Exchange",ez,[],                                     
                              "/usr/lib/rabbitmq/lib/rabbitmq_server-3.6.10/plugins/rabbitmq_recent_history_exchange-3.6.10.ez"},
                          {plugin,rabbitmq_sharding,"3.6.10",                                                                                                                                                      
                              "RabbitMQ Sharding Plugin",ez,[],   
                              "/usr/lib/rabbitmq/lib/rabbitmq_server-3.6.10/plugins/rabbitmq_sharding-3.6.10.ez"},                                                                                                 
                          {plugin,rabbitmq_shovel,"3.6.10",                                                                    
                              "Data Shovel for RabbitMQ",ez,                                                                                                                                                       
                              [amqp_client],                                
                              "/usr/lib/rabbitmq/lib/rabbitmq_server-3.6.10/plugins/rabbitmq_shovel-3.6.10.ez"},                                                                                                   
                          {plugin,rabbitmq_shovel_management,"3.6.10",                                                      
                              "Management extension for the Shovel plugin",ez,                                                                                                                                      
                              [rabbitmq_management,rabbitmq_shovel],
                              "/usr/lib/rabbitmq/lib/rabbitmq_server-3.6.10/plugins/rabbitmq_shovel_management-3.6.10.ez"},                                                                                        
                          {plugin,rabbitmq_stomp,"3.6.10",            
                              "RabbitMQ STOMP plugin",ez,                  
                              [amqp_client],                                                                                                                                                                       
                              "/usr/lib/rabbitmq/lib/rabbitmq_server-3.6.10/plugins/rabbitmq_stomp-3.6.10.ez"},                                                                                                    
                          {plugin,rabbitmq_top,"3.6.10","RabbitMQ Top",ez,  
                              [amqp_client,rabbitmq_management],                                                          
                              "/usr/lib/rabbitmq/lib/rabbitmq_server-3.6.10/plugins/rabbitmq_top-3.6.10.ez"},                                                                                                      
                          {plugin,rabbitmq_tracing,"3.6.10",                 
                              "RabbitMQ message logging / tracing",ez,
                              [rabbitmq_management],                                                                              
                              "/usr/lib/rabbitmq/lib/rabbitmq_server-3.6.10/plugins/rabbitmq_tracing-3.6.10.ez"},                                                                                                  
                          {plugin,rabbitmq_trust_store,"3.6.10",
                              "Client X.509 certificates trust store",ez,[],                                            
                              "/usr/lib/rabbitmq/lib/rabbitmq_server-3.6.10/plugins/rabbitmq_trust_store-3.6.10.ez"},                                                                                              
                          {plugin,rabbitmq_web_dispatch,"3.6.10",           
                              "RabbitMQ Web Dispatcher",ez,                                                 
                              [cowboy],
                              "/usr/lib/rabbitmq/lib/rabbitmq_server-3.6.10/plugins/rabbitmq_web_dispatch-3.6.10.ez"},                                                                                             
                          {plugin,rabbitmq_web_mqtt,"3.6.10",     
                              "RabbitMQ MQTT-over-WebSockets adapter",ez,                                                                                                                                          
                              [cowboy,rabbitmq_mqtt],                                                                          
                              "/usr/lib/rabbitmq/lib/rabbitmq_server-3.6.10/plugins/rabbitmq_web_mqtt-3.6.10.ez"},                                                                                                 
                          {plugin,rabbitmq_web_mqtt_examples,"3.6.10",      
                              "Rabbit WEB-MQTT - examples",ez,                                                                                                                                                     
                              [rabbitmq_web_dispatch,rabbitmq_web_mqtt],                                                    
                              "/usr/lib/rabbitmq/lib/rabbitmq_server-3.6.10/plugins/rabbitmq_web_mqtt_examples-3.6.10.ez"},                                                                                         
                          {plugin,rabbitmq_web_stomp,"3.6.10",      
                              "Rabbit WEB-STOMP - WebSockets to Stomp adapter",                                                                                                                                    
                              ez,                                     
                              [cowboy,sockjs,rabbitmq_stomp],              
                              "/usr/lib/rabbitmq/lib/rabbitmq_server-3.6.10/plugins/rabbitmq_web_stomp-3.6.10.ez"},                                                                                                
                          {plugin,rabbitmq_web_stomp_examples,"3.6.10",                                                                                                                                            
                              "Rabbit WEB-STOMP - examples",ez,             
                              [rabbitmq_web_dispatch,rabbitmq_web_stomp],                                                 
                              "/usr/lib/rabbitmq/lib/rabbitmq_server-3.6.10/plugins/rabbitmq_web_stomp_examples-3.6.10.ez"},                                                                                       
                          {plugin,sockjs,"0.3.4","SockJS",ez,[],             
                              "/usr/lib/rabbitmq/lib/rabbitmq_server-3.6.10/plugins/sockjs-0.3.4.ez"}],
                         [],[]}],                                                                                                 
                    [{file,"src/rabbit_plugins_main.erl"},{line,79}]},                                                                                                                                             
                {rabbit_cli,main,3,[{file,"src/rabbit_cli.erl"},{line,84}]},
                {init,start_it,1,[]},                                                                                   
                {init,start_em,1,[]}]}}                                                                                                                                                                            
---- End output of rabbitmq-plugins list -q -e 'rabbitmq_management\b' ---- 
Ran rabbitmq-plugins list -q -e 'rabbitmq_management\b' returned 70

The same setup runs fine with cookbook version 5.7.0 or earlier. Also see http://logs.openstack.org/45/642745/1/check/openstack-chef-integration/92b9e8d/ubuntu-xenial/chef-client-pass1.txt for a sample of a full log. Environment is an OpenStack deployment on Xenial using the Ubuntu Cloud Archive for Queens.

@jharbott

This comment has been minimized.

Copy link
Author

commented Mar 12, 2019

Seems to be triggered by f6f08c3 which doesn't correctly detect that the installed distro version is 3.6.10, i.e. < 3.7.10 and doesn't support the "-q" option.

@michaelklishin michaelklishin changed the title Upgrade to 5.7.1 breaks OpenStack deployment 5.7.1 doesn't correctly detect 3.6.x versions Mar 12, 2019

@michaelklishin

This comment has been minimized.

Copy link
Member

commented Mar 12, 2019

Hm, the LWRP test in f6f08c3 does include a rabbitmq_plugin resource. So how come it passes.

@michaelklishin

This comment has been minimized.

Copy link
Member

commented Mar 12, 2019

@jharbott can you please share a little more information about your system? What resources or recipes/attributes are used? rabbitmq-plugins list is invoked with -q for 3.7.x only, for example.

@michaelklishin

This comment has been minimized.

Copy link
Member

commented Mar 12, 2019

According to the stack trace,

{function_clause,                                                                                                                                                                                       
               [{rabbit_plugins_main,action,                                                                                                                                                                       
                    [list,'rabbit@controller-node11',                       
                     ["-q","rabbitmq_management\\b"],                                                                     
                     [{"-m",false},                                                                                                                                                                                
                      {"-n",'rabbit@controller-node11'},                     
                      {"-e",true},                                    
                      {"-E",false},                                                                                               
                      {"-v",false}],   

The command was invoked as

rabbitmq-plugins list -q -e rabbitmq_management\\b -n rabbit@controller-node11

And the only provider I can find that could do that is

def plugin_enabled?(name)
  ENV['PATH'] = "#{ENV['PATH']}:/usr/lib/rabbitmq/bin"
  cmdstr = if rabbitmq_37?
             "rabbitmq-plugins list -q -e '#{name}\\b'"
           else
             "rabbitmq-plugins list -e '#{name}\\b'"
           end
  cmd = Mixlib::ShellOut.new(cmdstr, :env => shell_environment)
  cmd.run_command
  Chef::Log.debug "rabbitmq_plugin_enabled?: #{cmdstr}"
  Chef::Log.debug "rabbitmq_plugin_enabled?: #{cmd.stdout}"
  cmd.error!
  cmd.stdout =~ /\b#{name}\b/
end

and it does not include -q for non-3.7.x nodes. The 3.6.x LWRP Kitchen test passes.

@michaelklishin

This comment has been minimized.

Copy link
Member

commented Mar 12, 2019

Just spotted End output of rabbitmq-plugins list -q -e 'rabbitmq_management\b' which confirms the hypothesis.

michaelklishin added a commit that referenced this issue Mar 12, 2019

@jharbott

This comment has been minimized.

Copy link
Author

commented Mar 12, 2019

I think that the only relevant attribute being set is

override['rabbitmq']['use_distro_version'] = true

and then

include_recipe 'rabbitmq'
include_recipe 'rabbitmq::mgmt_console'

is called in the recipe. See the whole wrapper cookbook here: https://opendev.org/openstack/cookbook-openstack-ops-messaging

@michaelklishin

This comment has been minimized.

Copy link
Member

commented Mar 12, 2019

This is very unfortunate. When distro version is used, the cookbook cannot know what version is to be provisioned and what assumptions it can or cannot make. This is one of the reasons why in 6.0 installing distribution-provided version won't be possible any more.

The OpenStack cookbook also assumes that it doesn't have to control what versions may be installed, so as soon as the default changed to 3.7.13, it immediately failed.

Setting node['rabbitmq']['version'] to 3.6.16 or similar should help. The rabbitmq_37? helper uses that attribute to determine what switches to use.

@jharbott I will see what I can do but there's only so much time I'm willing to spend on this scenario, for a few reasons:

  • RabbitMQ 3.6.x has been out of support for almost 10 months now. Soon all focus in this cookbook will switch to 6.0 which won't support RabbitMQ 3.6.x.
  • Using distribution version makes it impossible to know what can and cannot be relied on, from CLI flags to entire new tools and commands (such as rabbitmq-diagnostics)
  • cookbook-openstack-ops-messaging doesn't do any version pinning. I'm afraid that's not my responsibility to make sure it is not affected by the changes in this cookbook.

@michaelklishin michaelklishin changed the title 5.7.1 doesn't correctly detect 3.6.x versions 5.7.1 doesn't detect 3.6.x versions when no version is explicitly configured Mar 12, 2019

michaelklishin added a commit that referenced this issue Mar 12, 2019

Assume that "distro version" means "3.6.x"
That's the case at the moment and for the next year or so.
This cookbook will drop the option of using distro version in 6.0.

References #508.
@michaelklishin

This comment has been minimized.

Copy link
Member

commented Mar 12, 2019

This has been addressed to the extent that is practical: node['rabbitmq']['use_distro_version'] = true will imply that the node is a 3.6.x one. Note that -q is supported by rabbitmqctl in 3.6.x, so only the plugin provider has been updated.

@jharbott I suspect that you don't have a way to safely try the tip of v5.x in your environment but if you do, it would be greatly appreciated. FWIW there is now a Test Kitchen suite for this scenario.

@michaelklishin michaelklishin added this to the 5.7.2 milestone Mar 12, 2019

@michaelklishin michaelklishin self-assigned this Mar 12, 2019

michaelklishin added a commit that referenced this issue Mar 13, 2019

Make sure the test for #508 enables a plugin
Which discovers a typo that's now corrected.
@jharbott

This comment has been minimized.

Copy link
Author

commented Mar 13, 2019

Thanks for the fast response and fix.
I could test the current tip in a development environment and it seems that the original issue is resolved, but I'm now seeing this failure when adding a user:

    ================================================================================                                                                                                                       [70/2190]
    Error executing action `add` on resource 'rabbitmq_user[add openstack rabbit user]'                                                                                                                            
    ================================================================================                                                                                                                               
                                                                                                                                                                                                                   
    Mixlib::ShellOut::ShellCommandFailed                                                                                                                                                                           
    ------------------------------------                                                                                                                                                                           
    execute[rabbitmqctl -q add_user rabbit] (/var/chef/cache/cookbooks/rabbitmq/providers/user.rb line 104) had an error: Mixlib::ShellOut::ShellCommandFailed: Command execution failed. STDOUT/STDERR suppressed f
or sensitive resource                                  
                                                                                             
    Resource Declaration:                             
    ---------------------                                                                  
    # In /var/chef/cache/cookbooks/openstack-ops-messaging/recipes/rabbitmq-server.rb                                                                                                                              
                                                                                                                                                                                                                   
     65: rabbitmq_user 'add openstack rabbit user' do                                                                                                                                                              
     66:   user user                                                                                                                                                                                               
     67:   password pass                                                                          
     68:   action :add
     69: end
     70:

    Compiled Resource:
    ------------------
    # Declared in /var/chef/cache/cookbooks/openstack-ops-messaging/recipes/rabbitmq-server.rb:65:in `from_file'

    rabbitmq_user("add openstack rabbit user") do
      action [:add]
      default_guard_interpreter :default
      cookbook_name "openstack-ops-messaging"                                                                                                                                                                       
      recipe_name "rabbitmq-server"                                                                                                                                                                                
      user "rabbit"                                                                                                                                                                                                
      password "REDACTED"
    end                                                                                                                                                                                                            
                                                                                                                                                                                                                   
    System Info:                                                                                                                                                                                                    
    ------------                                       
    chef_version=13.10.0                                                                     
    platform=ubuntu                                   
    platform_version=16.04                                                                 
    ruby=ruby 2.4.4p296 (2018-03-28 revision 63013) [x86_64-linux]                                                                                                                                                 
    program_name=chef-client worker: ppid=27312;start=12:53:58;                                                                                                                                                    
    executable=/opt/chef/bin/chef-client                                                                                                                                                                           

Running with debug output and fixing the bug in the logging shows that user_exists? still tries to run the 3.7.x version rabbitmqctl -s list_users which fails.

@michaelklishin michaelklishin modified the milestones: 5.7.2, 5.7.3 Mar 13, 2019

michaelklishin added a commit that referenced this issue Mar 13, 2019

michaelklishin added a commit that referenced this issue Mar 13, 2019

@michaelklishin

This comment has been minimized.

Copy link
Member

commented Mar 13, 2019

@jharbott ready for another round.

openstack-gerrit pushed a commit to openstack/cookbook-openstack-ops-messaging that referenced this issue Mar 14, 2019

Pin rabbitmq cookbook to avoid broken version
Version 5.7.1 seems to break things when running with the distro version
as in our setup [0]. Pin to 5.7.0 until there is a fix.

[0] rabbitmq/chef-cookbook#508

Change-Id: I10059588716dea49a5a0ad996db20323d458bd43

openstack-gerrit added a commit to openstack/openstack that referenced this issue Mar 14, 2019

Update git submodules
* Update cookbook-openstack-ops-messaging from branch 'master'
  - Pin rabbitmq cookbook to avoid broken version
    
    Version 5.7.1 seems to break things when running with the distro version
    as in our setup [0]. Pin to 5.7.0 until there is a fix.
    
    [0] rabbitmq/chef-cookbook#508
    
    Change-Id: I10059588716dea49a5a0ad996db20323d458bd43

openstack-gerrit pushed a commit to openstack/cookbook-openstack-ops-messaging that referenced this issue Mar 14, 2019

Pin rabbitmq cookbook to avoid broken version
Version 5.7.1 seems to break things when running with the distro version
as in our setup [0]. Pin to 5.7.0 until there is a fix.

[0] rabbitmq/chef-cookbook#508

Change-Id: I10059588716dea49a5a0ad996db20323d458bd43
(cherry picked from commit 300bb04)

@michaelklishin michaelklishin changed the title 5.7.1 doesn't detect 3.6.x versions when no version is explicitly configured Don't use 3.7.x-specific CLI flags on 3.6.x versions installed when no version was explicitly configured Mar 15, 2019

@jharbott

This comment has been minimized.

Copy link
Author

commented Mar 18, 2019

@michaelklishin thanks, I can confirm that with 5.7.3 the cookbook is working for us again. We are also looking into running more current erlang and rabbitmq versions as a long-term solution.

@michaelklishin

This comment has been minimized.

Copy link
Member

commented Mar 18, 2019

@jharbott thank you for confirming and for the update.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.