RabbitMQ commands fail due to wrong locale

[jodok]

Description of Issue/Question

it seems that the salt-runner on the minion doesn't set the proper locale for the rabbitmq commands of erlang-20.1.7-1, rabbitmq-server-3.7.1

Steps to Reproduce Issue

whenever i run a rabbitmq command on a minion from salt-master (e.g. salt 'queue*' rabbitmq.status I receive:

ERROR: RabbitMQ command failed: warning: the VM is running with native name encoding of latin1 which may cause Elixir to malfunction as it expects utf8. Please ensure your locale is set to UTF-8 (which can be verified by running "locale" in your shell)

if i run the command on the minion directly (salt-call rabbitmq.status) everything is fine.

running cmd.run locale from salt-master:

    LANG=en_US.UTF-8
    LC_CTYPE=C
    LC_NUMERIC=C
    LC_TIME=C
    LC_COLLATE=C
    LC_MONETARY=C
    LC_MESSAGES=C
    LC_PAPER=C
    LC_NAME=C
    LC_ADDRESS=C
    LC_TELEPHONE=C
    LC_MEASUREMENT=C
    LC_IDENTIFICATION=C
    LC_ALL=

running salt-call cmd.run locale on salt-minion:

local:
    LANG=en_US.UTF-8
    LC_CTYPE="en_US.UTF-8"
    LC_NUMERIC="en_US.UTF-8"
    LC_TIME="en_US.UTF-8"
    LC_COLLATE="en_US.UTF-8"
    LC_MONETARY="en_US.UTF-8"
    LC_MESSAGES="en_US.UTF-8"
    LC_PAPER="en_US.UTF-8"
    LC_NAME="en_US.UTF-8"
    LC_ADDRESS="en_US.UTF-8"
    LC_TELEPHONE="en_US.UTF-8"
    LC_MEASUREMENT="en_US.UTF-8"
    LC_IDENTIFICATION="en_US.UTF-8"
    LC_ALL=en_US.UTF-8

Versions Report

CentOS 7, erlang-20.1.7-1, rabbitmq-server-3.7.1

salt-master:

Salt Version:
           Salt: 2017.7.2

Dependency Versions:
           cffi: 1.6.0
       cherrypy: unknown
       dateutil: Not Installed
      docker-py: Not Installed
          gitdb: 2.0.3
      gitpython: Not Installed
          ioflo: Not Installed
         Jinja2: 2.7.2
        libgit2: 0.24.6
        libnacl: Not Installed
       M2Crypto: Not Installed
           Mako: Not Installed
   msgpack-pure: Not Installed
 msgpack-python: 0.4.8
   mysql-python: Not Installed
      pycparser: 2.14
       pycrypto: 2.6.1
   pycryptodome: 3.4.3
         pygit2: 0.24.2
         Python: 2.7.5 (default, Aug  4 2017, 00:39:18)
   python-gnupg: Not Installed
         PyYAML: 3.11
          PyZMQ: 15.3.0
           RAET: Not Installed
          smmap: 2.0.3
        timelib: Not Installed
        Tornado: 4.2.1
            ZMQ: 4.1.4

System Versions:
           dist: centos 7.4.1708 Core
         locale: UTF-8
        machine: x86_64
        release: 3.10.0-693.5.2.el7.x86_64
         system: Linux
        version: CentOS Linux 7.4.1708 Core

salt-minion:

Salt Version:
           Salt: 2017.7.2

Dependency Versions:
           cffi: Not Installed
       cherrypy: Not Installed
       dateutil: Not Installed
      docker-py: Not Installed
          gitdb: Not Installed
      gitpython: Not Installed
          ioflo: Not Installed
         Jinja2: 2.7.2
        libgit2: Not Installed
        libnacl: Not Installed
       M2Crypto: Not Installed
           Mako: Not Installed
   msgpack-pure: Not Installed
 msgpack-python: 0.4.8
   mysql-python: Not Installed
      pycparser: Not Installed
       pycrypto: 2.6.1
   pycryptodome: 3.4.3
         pygit2: Not Installed
         Python: 2.7.5 (default, Aug  4 2017, 00:39:18)
   python-gnupg: Not Installed
         PyYAML: 3.11
          PyZMQ: 15.3.0
           RAET: Not Installed
          smmap: Not Installed
        timelib: Not Installed
        Tornado: 4.2.1
            ZMQ: 4.1.4

System Versions:
           dist: centos 7.4.1708 Core
         locale: UTF-8
        machine: x86_64
        release: 3.10.0-693.11.6.el7.x86_64
         system: Linux
        version: CentOS Linux 7.4.1708 Core

[gtmanfred]

If you set the locale using https://docs.saltstack.com/en/latest/ref/states/all/salt.states.locale.html#salt.states.locale.system does the problem go away?

salt-call inherits everything from the environment variables of the environment you are running, because it starts a whole minion process in that context.

the salt command talks to the salt-minion process that is started by systemd, so it needs the locale setup correctly for systemd to get it.

[jodok]

i've additionally set the locale as outlined - running the highstate didn't trigger a change (so it was already set).
unfortunately the problem persists. as mentioned - if i run cmd.run locale i see that LANG is correct, but LC_CTYPE=C (only via salt-master, not with a local salt-call)

[fastjohnny]

Hi @jodok
Did you solve the issue?
Have this error too, nothing helps

[gtmanfred]

I recently found out that reset_system_locale=True is the default on cmd.run, so that is the problem here.

Sounds like rabbit might need it to be set to false in some cases.

I am marking this as a bug.

Daniel

[fastjohnny]

Ok, eventually i'v found a workaround -
in /etc/locale.conf insert this
LANG=en_US.utf8
LC_ALL=en_US.UTF-8

[kissifrot]

Well I can confirm the problem on Debian stretch too with RabbitMQ 3.7.4 (and erlang 20.3-1)

[michaelklishin]

This is an Elixir requirement and their team seems to feel strongly about this.

Configuration software that uses CLI tools can override locale environment variables for the calls that expect an output that is parsed:

LANG="en_US.UTF-8" LC_ALL="en_US.UTF-8" rabbitmqctl …

Team RabbitMQ will consider forcing this in our shell scripts (rabbitmq/rabbitmq-server#1573) but IIRC this has been considered before and rejected.

[mdeneen]

This request was rejected by the rabbitmq folks. Is there any other workaround?

[gtmanfred]

pass reset_system_locale=False to any cmd.run commands in the rabbitmq module

If someone more familiar with rabbitmq wanted to modify and test that and submit a pr it would be greatly appreciated

[michaelklishin]

Set the locale to be a UTF-8 one for RabbitMQ CLI calls.

[cblackworth-anl]

This is marked as closed, but as is documented in the comments on #47453 its not fixed in 2018.3.x, currently this causes rabbitmq functions to fail on all minions.

[rallytime]

It will be in the 2018.3.3 release.