Allow admin to disable BMC SSH, IPMI, and HTTP access

[joseph-reynolds]

Expected Delivery Dates

• UX: xx/xx/xx
• Backend: 7/1/2020 estimate
• FED: xx/xx/xx

Stakeholders

SME: Joseph Reynolds
Design Researcher: @ParishrutB @priyanka-pillai97
UX Designer: @ParishrutB @priyanka-pillai97
FED: @dixsie

Use Case

The BMC admin should have an option to disable BMC shell access as a way to ensure the system is managed only by its intended interfaces (like Redfish REST APIs). Security conscious users will want to disable shell access when build the OpenBMC image or when provision their BMC. They require that, for example, to better control and log use of the BMC's management functions, and to pass audits. Use cases are (1) large-scale data centers where uniform access is desired, and (2) systems with sensitive (personal, financial, etc.) data where shell access constitutes a back door into the system.

Specifically, when disabled, secure shell (ssh) access to the BMC (ssh -p 22) will fail. Note that ssh access to the host console (via ssh -p 2200) is not affected by this design.

The admin will be able to re-enable access, allow the BMC shell to be used for some function, debugging, or whatever, and then disable access again. Presumably use of the shell will be a rare event and closely watched to ensure no back doors into the BMC are created.

The BMC admin should be able to log the fact that BMC shell access was disabled or re-enabled. For example, if the design implements the Redfish ManagerNetworkProtocol SSH property (reference below), then Redfish REST API logging would suffice. The BMC admin should also be able to log ssh connection attempts, for example, log files written by the ssh server, PAM, etc.

Requirements

Design

We don't want the GUI to turn this function on or off by accident. My crude GUI design sketch: I envision a new status field on the admin page that shows if "BMC shell access is enabled" (and clearly indicate this feature is separate from the "host console ssh" feature). Maybe have a way to change its state, indicating one of:

• You are about to disable BMC shell access via SSH. No SSH shell connections can be made, but existing connections will still work.
• You are about to enable BMC shell access via SSH. Users will be able to connect to the BMC shell via ssh and attempt to authenticate.

Development

Shell access will remain enabled by default in the current OpenBMC releases.

InVision Prototype

---

Design Issue (phosphor-webui)

---

Development Issue

---

References/Resources

• The Redfish ManagerNetworkProtocol SSH property (https://redfish.dmtf.org/schemas/ManagerNetworkProtocol.v1_4_0.json).

[jk-ozlabs]

We'll need a way to perform debugging in the field, and the functionality there won't be all available over REST. Will there be a way to re-enable ssh for this use-case?

[jk-ozlabs]

Also:

Access to the BMC's shell should be disabled because it offers too much power to users.

This seems to indicate an element of contempt for our users. Should we not give them the facility to perform any sort of automation/interaction method that suits their workflow?

If it's a security issue, then let's look at securing it. If it's warranty issue, I'm sure we can find ways to put safeguards in place. But otherwise, this seems like removing flexibility that some users may rely on.

[joseph-reynolds]

The OpenBMC project direction is to provide a complete set of function via Redfish, including diagnostic functions. My understanding is Redfish provides a forward compatible abstraction layer which frees us to modify the implementation while maintaining compatibility, and that providing shell access to D-Bus and kernel interfaces does not support that direction.

The use case for shell access will remain in the OpenBMC project. This story is for the use case of someone who wants to block shell access to the BMC. We'll have both use cases.

The primary use case for being able to lock out shell access to the BMC is for system owners to ensure the system is managed only by its intended interfaces (like Redfish). They require that, for example, to better control and log use of the BMC's management functions, and to pass audits. Use cases are (1) large-scale data centers where uniform access is desired, and (2) systems with sensitive (personal, financial, etc.) data where shell access constitutes a back door into their data.

Debugging in the field should be a rare event, with the escalation path being:

1. Send in logs.
2. Send in more logs.
3. Try to recreate the problem in the development lab.
4. Create "test fixes" (IBM's old PTF terminology) designed to capture more data.
5. Hop onto the customer system via ssh.

We can implement the Redfish ManagerNetworkProtocol SSH property (https://redfish.dmtf.org/schemas/ManagerNetworkProtocol.v1_4_0.json) to control access to the BMC's shell via ssh -p 22. I feel this would be compatible with the use cases described above because:

1. The fact that the BMC's configuration is changing to allow shell access could be logged (example: sent to an external server) in a way that the shell user could not hide their tracks.
2. When done debugging, removing shell access provides an additional layer of assurance that shell access is disabled. (Example: in addition to changing LDAP permissions, passwords, or ssh certificates.)

[joseph-reynolds]

It didn't come across in the template above, but I meant for this epic to identify all of the functions that need to be in place to support the use cases for disabling BMC shell access. I should edit the description. Thanks for pointing that out.

[jk-ozlabs]

I think it's an admirable goal to have all possible debug functionality exposed over standardised interfaces, but we're never going to be able to anticipate the full spectrum of what we need for all debug scenarios.

As an example, here's what I during the my last debug session (last week), on a system that needed to be in field configuration:

1. Check which versions of crypto libraries were installed on the system
2. Inspect the state of the network stack (using ip)
3. Upload a python script to allow me to capture network packets
4. Run that script several times (while triggering various management actions via standard interfaces), outputting debug data to local files
5. Perform initial analysis on the captured packet data, locally to the BMC
6. Run an editor to implement various packet filtering functions within that script
7. Retrieve captured packet data from the BMC

It's possible that some of these would have been possible through other methods, but I don't think we want facilities for all of those over the standard interfaces.

Additionally, we know that particular issue I have been working here is sensitive to code versions and external network environment, so the option to recreate this in our development environment is not always possible.

This is just one example, which happens to be my most recent.

With that in mind, we're absolutely going to need shell access at some points during field-based debugging, otherwise we're completely reliant on being able to reproduce specific problems in our development environment and/or specific BMC code loads.

[joseph-reynolds]

So we need interfaces to:

1. get the BMC's version,
2. query the BMC's network status, and
3. capture network packets.

That's a good start, thanks!

[mikey]

@joseph-reynolds the point @jk-ozlabs is making is that it's infeasible to add all the functionality that shell access provides for developers. Taking his one use case example and adding that functionality is just the start of playing an endless game of whack-a-mole.

[joseph-reynolds]

Yes I got that, thank you. My previous post was an attempt at very dry humor.

Given that we will have customers who want/need to disable BMC shell access over ssh, I am exploring the options. It seems like:

1. Permanently disabling shell access will limit our ability to service the machine.
2. The customer will want the ability to lock out shell access while the system is operational, re-enable it only in exceptional circumstances, and be able to audit the fact that shell access was re-enabled.
3. Therefore, the Redfish ManagerNetworkProtocol schema seems like the way to go.
4. Over the next 20 years we can add more functions to reduce the need for shell access.
5. (Item 4 was my humor again.)

[jk-ozlabs]

OK, phew. As @mikey says, I don't think the whack-a-mole approach would work; there's just too many corner-cases to cover.

I think your new plan is solid. Having the ability to disable shell access in regular scenarios would be good, as long as we can re-enable without too much disruption to the state of the BMC.