LP1892983: ipmi_test crashes when ipmi_config output is not utf-8

[beliaev-maksim]

This issue was migrated from https://bugs.launchpad.net/plainbox-provider-checkbox/+bug/1892983

Summary

Status	Created on	Heat	Importance	Security related
In Progress	2020-08-26 05:18:03	24	Medium	False

Description

We have some servers which have non utf-8 bytes in ipmi-config output and cause ipmi_test to crash

The non utf-8 is in Lan_Conf_Security_Keys and Popen.communicate will try to decode it as utf-8.

$ sudo /usr/sbin/ipmi-config --checkout --lan-channel-number 1 | grep -a -A5 '^Section Lan_Conf_Security_Keys'
Section Lan_Conf_Security_Keys

Give string or blank to clear. Max 20 chars

K_R Ӷ:΁,ء�Z��*�N��

Give string or blank to clear. Max 20 bytes, prefix with 0x to enter hex

K_G 0x0000000000000000000000000000000000000000
EndSection

Traceback (most recent call last):
File "/tmp/nest-jh8wu8ym.14373cc61f93798d1a9ac09341ae554943f71bd04dad06f96b13defe4aee3784/ipmi_test.py", line 324, in
sys.exit(main())
File "/tmp/nest-jh8wu8ym.14373cc61f93798d1a9ac09341ae554943f71bd04dad06f96b13defe4aee3784/ipmi_test.py", line 306, in main
results = ipmi_test.run_test()
File "/tmp/nest-jh8wu8ym.14373cc61f93798d1a9ac09341ae554943f71bd04dad06f96b13defe4aee3784/ipmi_test.py", line 277, in run_test
self.ipmi_channel(),
File "/tmp/nest-jh8wu8ym.14373cc61f93798d1a9ac09341ae554943f71bd04dad06f96b13defe4aee3784/ipmi_test.py", line 205, in ipmi_channel
self._ipmi_channel_hlpr(i, matches, channel)
File "/tmp/nest-jh8wu8ym.14373cc61f93798d1a9ac09341ae554943f71bd04dad06f96b13defe4aee3784/ipmi_test.py", line 186, in _ipmi_channel_hlpr
output = self._subproc_logging(cmd)
File "/tmp/nest-jh8wu8ym.14373cc61f93798d1a9ac09341ae554943f71bd04dad06f96b13defe4aee3784/ipmi_test.py", line 91, in _subproc_logging
output, error = process.communicate(timeout=self.subproc_timeout)
File "/usr/lib/python3.6/subprocess.py", line 863, in communicate
stdout, stderr = self._communicate(input, endtime, timeout)
File "/usr/lib/python3.6/subprocess.py", line 1574, in _communicate
self.stdout.errors)
File "/usr/lib/python3.6/subprocess.py", line 760, in _translate_newlines
data = data.decode(encoding, errors)
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xe6 in position 21281: invalid continuation byte

Attachments

ipmi.patch

Tags:
['servcert-221', 'sts']

[beliaev-maksim]

This thread was migrated from launchpad.net

https://launchpad.net/~ikepanhc wrote on 2020-08-26 05:19:11:

Manually decode it as cp437 can solve this issue. Please see the attachment

https://launchpad.net/~bladernr wrote on 2020-09-03 13:17:11:

Near as I can tell, this is out of spec. IPMI v2 spec explicitly suggests UTF-8 or UNICODE or ASCII+Latin1 for encoding. Anything outside of that is not within IPMI v2 specs.

However, the section on security keys does not mention specific encoding for the key when returned. This is a bit of a grey area. I'm not a fan of changing the encoding for the entire script to something that's not UTF-8 or Unicode when every other server in the world uses that standard.

https://launchpad.net/~bladernr wrote on 2020-09-03 13:19:38:

I'll also want to verify that Kr is even set on most of the servers I have access to....

https://launchpad.net/~bladernr wrote on 2020-09-03 13:30:56:

Lenovo x3650 M2:
$ sudo /usr/sbin/ipmi-config --checkout --lan-channel-number 1 | grep -a -A5 '^Section Lan_Conf_Security_Keys'
Section Lan_Conf_Security_Keys
## Give string or blank to clear. Max 20 chars
K_R
## Give string or blank to clear. Max 20 bytes, prefix with 0x to enter hex
K_G 0x0000000000000000000000000000000000000000
EndSection

Dell R710
$ sudo /usr/sbin/ipmi-config --checkout --lan-channel-number 1 | grep -a -A5 '^Section Lan_Conf_Security_Keys'
Section Lan_Conf_Security_Keys
## Give string or blank to clear. Max 20 bytes, prefix with 0x to enter hex
K_G 0x0000000000000000000000000000000000000000
EndSection

Section Lan_Conf_User_Security Comments

HP ProLiant SL230
ubuntu@gurley:~$ sudo /usr/sbin/ipmi-config --checkout --lan-channel-number 2 | grep -a -A5 '^Section Lan_Conf_Security_Keys'; sudo dmidecode |grep -A5 "System Information"
Section Lan_Conf_Security_Keys
## Give string or blank to clear. Max 20 bytes, prefix with 0x to enter hex
K_G 0x0000000000000000000000000000000000000000
EndSection

Section Lan_Conf_User_Security Comments

Fujitsu RX2530 M5
$ sudo /usr/sbin/ipmi-config --checkout --lan-channel-number 2 | grep -a -A5 '^Section Lan_Conf_Security_Keys'; sudo dmidecode |grep -A5 "System Information"

Section Lan_Conf_Security_Keys
## Give string or blank to clear. Max 20 chars
K_R
## Give string or blank to clear. Max 20 bytes, prefix with 0x to enter hex
K_G 0x0000000000000000000000000000000000000000
EndSection

https://launchpad.net/~bladernr wrote on 2020-09-03 13:33:24:

Gonna mark this invalid... this feels more like something specific to what they've done in their firmware with non-unicode/utf-8 characters. We could revisit this later, but first I think it's a better approach to fix this in firmware (or verify this on production hardware, not the mishmash of questionable loaners we have for SoC work).

https://launchpad.net/~bladernr wrote on 2020-09-03 13:33:38:

Feel free to reopen if you disagree.

https://launchpad.net/~bladernr wrote on 2020-09-03 14:06:15:

For reference, tyhe IPMI spec only mentions UTF-8, Unicode, and ASCII + Latin 1 as valid character encodings:

file:///home/bladernr/Downloads/ipmi-second-gen-interface-spec-v2-rev1-1.pdf

https://launchpad.net/~bladernr wrote on 2020-09-03 14:08:48:

Also, changed my mind and moved it back to Incomplete for now, plan to check with firmware first, and then revisit if that is not fixable

https://launchpad.net/~ikepanhc wrote on 2020-09-04 07:06:32:

Thanks providing the spec. It also comes to my mind that we can try to clear the Lan security before running checkbox, since most common cases people use utf-8 char for passwords.

Let me try that first.

https://launchpad.net/~andrew-cloke wrote on 2020-09-04 07:49:02:

Even if the "Lan_Conf_Security_Keys" section in the IPMI configuration data has not been (re)set by the user, it would be reasonable to expect it to be empty on new systems. At the very least it shouldn't contain garbage/illegal chars.
Is that fair?

https://launchpad.net/~ikepanhc wrote on 2020-09-07 06:03:28:

Yes, I agree. To have correct default setting is part of firmware's job.

Tried to set empty value for Lan_Conf_Security_Keys and then checkbox passed.

$ cat test.txt
Section Lan_Conf_Security_Keys
## Give string or blank to clear. Max 20 chars
K_R
## Give string or blank to clear. Max 20 bytes, prefix with 0x to enter hex
K_G 0x0000000000000000000000000000000000000000
EndSection
$ sudo ipmi-config --commit --lan-channel-number 1 --filename=test.txt
$ sudo /usr/lib/plainbox-provider-checkbox/bin/ipmi_test.py

Running IPMI Tests

---

Verifying kernel modules:

• ipmi_si already loaded
• ipmi_devintf already loaded

• Unable to load module ipmi_powernv!

  ---

  Warning: proceeding, but in-band IPMI may fail

  ---

• ipmi_ssif already loaded
• ipmi_msghandler already loaded

---

Fetching chassis status:

• Fetched chassis status!

---

Fetching power status:

• Fetched power status!

---

Fetching BMC information:

• Fetched BMC information!

---

Validating IPMI version:

• IPMI version: 2.0
  IPMI version compliant!

---

Fetching IPMI channels:

• Found 1 channel(s)!
  IPMI channel(s): [1]

---

IPMI tests passed!

https://launchpad.net/~ikepanhc wrote on 2020-09-07 06:04:14:

Set status to "Invalid" because this is a firmware issue.

https://launchpad.net/~rodsmith wrote on 2020-09-16 22:11:31:

Even if it's caused by firmware being out of spec, our script should probably do a better job of catching the resulting error. It could then report a test failure with a helpful message, like "unsupported encoding in IPMI output," rather than fail with a stack trace. Alternatively, if we wanted to allow out-of-spec output, the "except:" clause could try other encodings.

https://launchpad.net/~bladernr wrote on 2021-05-07 14:28:48:

reopening as Mao got pinged for this as well, so I'm trying some changes to ipmi_test.py to adjust codecs when necessary.

https://launchpad.net/~seyeongkim wrote on 2022-06-21 08:28:45:

related topic the customer mentioned.

https://discourse.maas.io/t/maas-failed-commissioning/4904

https://launchpad.net/~bladernr wrote on 2022-06-21 16:13:49:

I should add that the original complaint stumbled on this bug while searching the issue. The MAAS task is new, and has been triaged. Fixing this bug from my end (the Checkbox end) WILL NOT fix the issue in MAAS so these are separate issue that have manifested from the same underlying problem with Huawei's use of non-UTF8 characters here. IMO the MAAS part should have been a separate bug.

So just to set expectations, the problem with MAAS commissioning will be addressed separately by the MAAS team and the checkbox part of this bug is very low priority and will likely not be addressed for some time to come, since we are not currently certifying Huawei hardware until we work out how to handle the spun-off x86 server line.

https://launchpad.net/~seyeongkim wrote on 2022-06-22 07:27:59:

another bug related to this

https://bugs.launchpad.net/maas/+bug/1929478

https://launchpad.net/~jhusakowski wrote on 2022-08-25 08:47:38:

The MAAS issue mentioned in comments is reported and fixed here: https://bugs.launchpad.net/maas/+bug/1929478 Is this still reproducible after the fix?

[zhanglei-mao]

The output file for
for channel in seq 1 16; do
ipmi-config --debug --checkout --filename=ipmi-output --lan-channel-number $channel 2>&1
done
ipmi-output.txt

[bladernr]

The output file for for channel in seq 1 16; do ipmi-config --debug --checkout --filename=ipmi-output --lan-channel-number $channel 2>&1 done ipmi-output.txt

Hey Mao, could you ahve them run this modified version of ipmi_test.py? Have them run them side by side like this:

First to get a baseline, run the current version:
sudo /usr/lib/checkbox-provider-base/bin/ipmi_test.py

Observe that it has a traceback because of the non-UTF8 chars in K_R

Then run the modified version:

sudo ./ipmi_test.py

Which should now pass... or at least should not dump a traceback. I do not have access to a system that fails to try it myself.
ipmi_test.py.txt

You'll have to remove the .txt from the end of the filename, github won't let me attach .py files directly, apparenlty.