Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Solaris: default plugins directory is listed twice, breaking rabbitmq-plugins commands that detect duplicate plugins #3155

Closed
tropikhajma opened this issue Jun 28, 2021 · 12 comments · Fixed by #3156
Closed

Solaris: default plugins directory is listed twice, breaking rabbitmq-plugins commands that detect duplicate plugins #3155

tropikhajma opened this issue Jun 28, 2021 · 12 comments · Fixed by #3156
Labels
bug effort-tiny usability
Milestone
3.8.19

Comments

@tropikhajma
Copy link
Contributor

tropikhajma commented Jun 28, 2021 •
edited
Loading

In RabbitMQ 3.8.17 the CLI commands spew a long list of warnings *
Using the truss(1) command I found the process sets the PLUGINS_DIR variable with the same path twice:
15545/5: P L U G I N S _ D I R = / u s r / l i b / r a b b i t m q / p l
15545/5: u g i n s : / u s r / l i b / r a b b i t m q / p l u g i n s\0

and I found that prefixing the command with
PLUGINS_DIR="/usr/lib/rabbitmq/plugins"
avoids the issue.

The attached patch fixes the issue for me, but I do not speak Erlang nor am I familiar with RabbitMQ enough to be sure it's a correct approach.
02.patch.txt

This is not happening with RabbitMQ 3.8.3. I haven't tried other versions.
This is on Solaris 11.4, both architectures, Erlang 0.24.

*
# rabbitmq-plugins list
Listing plugins with pattern ".*" ...
Problem reading some plugins: [{"/usr/lib/rabbitmq/plugins/accept-0.3.5.ez",
                                duplicate_plugin},
                               {"/usr/lib/rabbitmq/plugins/amqp10_client-3.8.17.ez",
                                duplicate_plugin},
                               {"/usr/lib/rabbitmq/plugins/amqp10_common-3.8.17.ez",
                                duplicate_plugin},
                               {"/usr/lib/rabbitmq/plugins/amqp_client-3.8.17.ez",
                                duplicate_plugin},
                               {"/usr/lib/rabbitmq/plugins/aten-0.5.5.ez",
                                duplicate_plugin},
                               {"/usr/lib/rabbitmq/plugins/base64url-1.0.1.ez",
                                duplicate_plugin},
                               {"/usr/lib/rabbitmq/plugins/cowboy-2.8.0.ez",
                                duplicate_plugin},
                               {"/usr/lib/rabbitmq/plugins/cowlib-2.9.1.ez",
                                duplicate_plugin},
                               {"/usr/lib/rabbitmq/plugins/credentials_obfuscation-2.4.0.ez",
                                duplicate_plugin},
                               {"/usr/lib/rabbitmq/plugins/cuttlefish-3.0.0.ez",
                                duplicate_plugin},
                               {"/usr/lib/rabbitmq/plugins/eetcd-0.3.3.ez",
                                duplicate_plugin},
                               {"/usr/lib/rabbitmq/plugins/gen_batch_server-0.8.4.ez",
                                duplicate_plugin},
                               {"/usr/lib/rabbitmq/plugins/goldrush-0.1.9.ez",
                                duplicate_plugin},
                               {"/usr/lib/rabbitmq/plugins/gun-1.3.3.ez",
                                duplicate_plugin},
                               {"/usr/lib/rabbitmq/plugins/jose-1.11.1.ez",
                                duplicate_plugin},
                               {"/usr/lib/rabbitmq/plugins/jsx-2.11.0.ez",
                                duplicate_plugin},
                               {"/usr/lib/rabbitmq/plugins/lager-3.9.1.ez",
                                duplicate_plugin},
                               {"/usr/lib/rabbitmq/plugins/observer_cli-1.6.1.ez",
                                duplicate_plugin},
                               {"/usr/lib/rabbitmq/plugins/prometheus-4.6.0.ez",
                                duplicate_plugin},
                               {"/usr/lib/rabbitmq/plugins/ra-1.1.8.ez",
                                duplicate_plugin},
                               {"/usr/lib/rabbitmq/plugins/rabbit-3.8.17.ez",
                                duplicate_plugin},
                               {"/usr/lib/rabbitmq/plugins/rabbit_common-3.8.17.ez",
                                duplicate_plugin},
                               {"/usr/lib/rabbitmq/plugins/rabbitmq_amqp1_0-3.8.17.ez",
                                duplicate_plugin},
                               {"/usr/lib/rabbitmq/plugins/rabbitmq_auth_backend_cache-3.8.17.ez",
                                duplicate_plugin},
                               {"/usr/lib/rabbitmq/plugins/rabbitmq_auth_backend_http-3.8.17.ez",
                                duplicate_plugin},
                               {"/usr/lib/rabbitmq/plugins/rabbitmq_auth_backend_ldap-3.8.17.ez",
                                duplicate_plugin},
                               {"/usr/lib/rabbitmq/plugins/rabbitmq_auth_backend_oauth2-3.8.17.ez",
                                duplicate_plugin},
                               {"/usr/lib/rabbitmq/plugins/rabbitmq_auth_mechanism_ssl-3.8.17.ez",
                                duplicate_plugin},
                               {"/usr/lib/rabbitmq/plugins/rabbitmq_aws-3.8.17.ez",
                                duplicate_plugin},
                               {"/usr/lib/rabbitmq/plugins/rabbitmq_consistent_hash_exchange-3.8.17.ez",
                                duplicate_plugin},
                               {"/usr/lib/rabbitmq/plugins/rabbitmq_event_exchange-3.8.17.ez",
                                duplicate_plugin},
                               {"/usr/lib/rabbitmq/plugins/rabbitmq_federation-3.8.17.ez",
                                duplicate_plugin},
                               {"/usr/lib/rabbitmq/plugins/rabbitmq_federation_management-3.8.17.ez",
                                duplicate_plugin},
                               {"/usr/lib/rabbitmq/plugins/rabbitmq_jms_topic_exchange-3.8.17.ez",
                                duplicate_plugin},
                               {"/usr/lib/rabbitmq/plugins/rabbitmq_management-3.8.17.ez",
                                duplicate_plugin},
                               {"/usr/lib/rabbitmq/plugins/rabbitmq_management_agent-3.8.17.ez",
                                duplicate_plugin},
                               {"/usr/lib/rabbitmq/plugins/rabbitmq_mqtt-3.8.17.ez",
                                duplicate_plugin},
                               {"/usr/lib/rabbitmq/plugins/rabbitmq_peer_discovery_aws-3.8.17.ez",
                                duplicate_plugin},
                               {"/usr/lib/rabbitmq/plugins/rabbitmq_peer_discovery_common-3.8.17.ez",
                                duplicate_plugin},
                               {"/usr/lib/rabbitmq/plugins/rabbitmq_peer_discovery_consul-3.8.17.ez",
                                duplicate_plugin},
                               {"/usr/lib/rabbitmq/plugins/rabbitmq_peer_discovery_etcd-3.8.17.ez",
                                duplicate_plugin},
                               {"/usr/lib/rabbitmq/plugins/rabbitmq_peer_discovery_k8s-3.8.17.ez",
                                duplicate_plugin},
                               {"/usr/lib/rabbitmq/plugins/rabbitmq_prelaunch-3.8.17.ez",
                                duplicate_plugin},
                               {"/usr/lib/rabbitmq/plugins/rabbitmq_prometheus-3.8.17.ez",
                                duplicate_plugin},
                               {"/usr/lib/rabbitmq/plugins/rabbitmq_random_exchange-3.8.17.ez",
                                duplicate_plugin},
                               {"/usr/lib/rabbitmq/plugins/rabbitmq_recent_history_exchange-3.8.17.ez",
                                duplicate_plugin},
                               {"/usr/lib/rabbitmq/plugins/rabbitmq_sharding-3.8.17.ez",
                                duplicate_plugin},
                               {"/usr/lib/rabbitmq/plugins/rabbitmq_shovel-3.8.17.ez",
                                duplicate_plugin},
                               {"/usr/lib/rabbitmq/plugins/rabbitmq_shovel_management-3.8.17.ez",
                                duplicate_plugin},
                               {"/usr/lib/rabbitmq/plugins/rabbitmq_stomp-3.8.17.ez",
                                duplicate_plugin},
                               {"/usr/lib/rabbitmq/plugins/rabbitmq_top-3.8.17.ez",
                                duplicate_plugin},
                               {"/usr/lib/rabbitmq/plugins/rabbitmq_tracing-3.8.17.ez",
                                duplicate_plugin},
                               {"/usr/lib/rabbitmq/plugins/rabbitmq_trust_store-3.8.17.ez",
                                duplicate_plugin},
                               {"/usr/lib/rabbitmq/plugins/rabbitmq_web_dispatch-3.8.17.ez",
                                duplicate_plugin},
                               {"/usr/lib/rabbitmq/plugins/rabbitmq_web_mqtt-3.8.17.ez",
                                duplicate_plugin},
                               {"/usr/lib/rabbitmq/plugins/rabbitmq_web_stomp-3.8.17.ez",
                                duplicate_plugin},
                               {"/usr/lib/rabbitmq/plugins/ranch-2.0.0.ez",
                                duplicate_plugin},
                               {"/usr/lib/rabbitmq/plugins/recon-2.5.1.ez",
                                duplicate_plugin},
                               {"/usr/lib/rabbitmq/plugins/stdout_formatter-0.2.4.ez",
                                duplicate_plugin},
                               {"/usr/lib/rabbitmq/plugins/syslog-3.4.5.ez",
                                duplicate_plugin},
                               {"/usr/lib/rabbitmq/plugins/sysmon_handler-1.3.0.ez",
                                duplicate_plugin}]
 Configured: E = explicitly enabled; e = implicitly enabled
 | Status: * = running on rabbit@sudo
 |/
[  ] rabbitmq_amqp1_0                  3.8.17
[  ] rabbitmq_auth_backend_cache       3.8.17
[  ] rabbitmq_auth_backend_http        3.8.17
[  ] rabbitmq_auth_backend_ldap        3.8.17
[  ] rabbitmq_auth_backend_oauth2      3.8.17
[  ] rabbitmq_auth_mechanism_ssl       3.8.17
[  ] rabbitmq_consistent_hash_exchange 3.8.17
[  ] rabbitmq_event_exchange           3.8.17
[  ] rabbitmq_federation               3.8.17
[  ] rabbitmq_federation_management    3.8.17
[  ] rabbitmq_jms_topic_exchange       3.8.17
[  ] rabbitmq_management               3.8.17
[  ] rabbitmq_management_agent         3.8.17
[  ] rabbitmq_mqtt                     3.8.17
[  ] rabbitmq_peer_discovery_aws       3.8.17
[  ] rabbitmq_peer_discovery_common    3.8.17
[  ] rabbitmq_peer_discovery_consul    3.8.17
[  ] rabbitmq_peer_discovery_etcd      3.8.17
[  ] rabbitmq_peer_discovery_k8s       3.8.17
[  ] rabbitmq_prometheus               3.8.17
[  ] rabbitmq_random_exchange          3.8.17
[  ] rabbitmq_recent_history_exchange  3.8.17
[  ] rabbitmq_sharding                 3.8.17
[  ] rabbitmq_shovel                   3.8.17
[  ] rabbitmq_shovel_management        3.8.17
[  ] rabbitmq_stomp                    3.8.17
[  ] rabbitmq_top                      3.8.17
[  ] rabbitmq_tracing                  3.8.17
[  ] rabbitmq_trust_store              3.8.17
[  ] rabbitmq_web_dispatch             3.8.17
[  ] rabbitmq_web_mqtt                 3.8.17
[  ] rabbitmq_web_stomp                3.8.17
#
@michaelklishin
Copy link
Member

This seems to be Solaris-specific as I don't remember seeing this elsewhere.

I can't tell what the case expression may be returning on Solaris that isn't matched by an existing clause. It would be very interesting to find out.

Please submit a PR with this change. It's harmless but without knowing what value is returned on Solaris, we can't really
tell if there is a better approach. But we would consider accepting this blunt solution, too.

@michaelklishin
Copy link
Member

Actually if the issue is with an env variable value, it would be great to investigate what in the shell script handling may go wrong. And there is nothing wrong with overriding an env variable in your package/environment. I'm afraid accounting for and testing on less popular distributions will be quite difficult for us to cover in terms of future maintenance.

@michaelklishin
Copy link
Member

and the reason why 3.8.3 doesn't exhibit this behavior is that in 3.8.4, the way scripts, env variables and environment "setup" nodes do have undergone a lot of changes. There was a couple of side effects addressed by 3.8.6 or so. Admittedly we don't have any Solaris users on our team.

@michaelklishin
Copy link
Member

Another idea for a small PR: we can remove duplicates from the PLUGINS_DIR value right after we parse it to a list of locations.

By the way, nodes should log quite a bit of their env variable and "setup" steps at debug level, maybe there would be something that would help narrow the issue down quicker.

@michaelklishin michaelklishin changed the title Problem reading some plugins: duplicate_plugin Solaris: default plugins directory is listed twice, breaking rabbitmq-plugins commands that detect duplicate plugins Jun 28, 2021
@michaelklishin
Copy link
Member

This function seems to be a good candidate to perform deduplication after PLUGINS_DIR value has been split using a platform-specific separator (e.g. :).

michaelklishin added a commit that referenced this issue Jun 28, 2021
@michaelklishin
Deduplicate parsed list of plugin directories
d0ad77f 
See #3155 for context. It is also possible that someone might
override the path with duplicate values.

Note that the list of directories will lose its original
ordering. That should be fine as we expect unique plugin
paths: it is not a "preference list" (ordering should
not matter).

Closes #3155
@michaelklishin michaelklishin mentioned this issue Jun 28, 2021
Merged
@michaelklishin
Copy link
Member

#3156 deduplicates the parsed list of plugin directories (locations). @tropikhajma any chance you can give this a shot? If you need us to produce a one-off package with this PR, of what type should it be, generic UNIX? compiled on Erlang 23.3?

@tropikhajma
Copy link
Contributor Author

I'm still seeing this issue with rabbitmq 3.8.17 with #3156 applied.

@michaelklishin
Copy link
Member

@tropikhajma any debug level logs you can share? We don't have a Solaris environment I'm afraid.

@michaelklishin
Copy link
Member

Apparently there is an OVA of Solaris 11.4. Setting it up would take a while for me as I haven't used Solaris in a decade. So it would be nice to have some debug logs to see what rabbit_prelaunch produces when it sets up the environment.

@michaelklishin
Copy link
Member

michaelklishin commented Jun 28, 2021 •
edited
Loading

I found another place where we parse the list of plugins and it is specific to the rabbitmq-plugins commands API. @tropikhajma please try again in a few minute, I'm about to push a revision to #3156.

michaelklishin added a commit that referenced this issue Jun 28, 2021
@michaelklishin
Deduplicate parsed plugin directory list in one more place
17a1f4f 
References #3155
@tropikhajma
Copy link
Contributor Author

Thank you, the second one did the trick.

@michaelklishin michaelklishin added this to the 3.8.19 milestone Jun 28, 2021
@michaelklishin
Copy link
Member

Backported to v3.8.x and v3.9.x.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug effort-tiny usability
Projects
None yet
Milestone
3.8.19
Development

Successfully merging a pull request may close this issue.

Deduplicate parsed list of plugin directories rabbitmq/rabbitmq-server
2 participants
@michaelklishin @tropikhajma