Update bluestore OSDs with new format in v14.2.2

[travisn]

Is this a bug report or feature request?

• Feature Request

What should the feature do:
When the underlying data format changes in the OSDs, we need to perform a long running operation to update the OSDs. The requirement as seen here in certain upgrade scenarios is:

• Stop the OSD
• Run the repair on the device: ceph-bluestore-tool repair --path ...
• Start the OSD

This is specifically seen when updating from luminous or mimic to nautilus v14.2.2. There is a warning that:

Legacy BlueStore stats reporting detected on N OSD(s)

More background can seen in this BZ.

Design for updating OSDs

Rook will need to automate this process. There isn't a feasible way for the admin to manually upgrade. Consider the following design:

• Rook runs the orchestration, which triggers the "osd prepare" job on each node
• The osd prepare job detects if the OSDs are in a state that they need to be repaired for the data path update
• The operator starts the OSDs like usual
• If there are OSDs that require repair, start a goroutine to do the following:
  • For each OSD that requires repair:
    • Take a lock on the OSD orchestration (so the operator doesn't try to update the osd spec in parallel)
    • Stop the osd pod (set its deployment replica count to 0)
    • Run a job that runs the repair command on the OSD
    • When the job completes, start the OSD pod again
    • Release the OSD lock

This will generally be a useful pattern going forward for other OSD repairs in the future.
@liewegas Does this seem reasonable or can you think of a simpler repair path?

[mumrau]

I wonder wether a fully automatic process fits the use case. I think a way of triggering it manually would make sense as well. Because in our case (bluestore legacy stat fs), the documentation is extremely clear that we need to proceed on a rolling repair on OSDs.
On the other hand, I can easily imagine someone having an inconsistency of some sort, which cannot be completely analyzed automatically, who would desire to pass a certain command pre-OSD start.
I believe this use case could require an override of the osd-prepare job once and only once (until success obviously). Maybe a ConfigMap that would live temporarily encapsualting the desired command would make sense, the orchestrator would have to watch for it and act accordingly?

What I'm stating isn't incompatible with the automated process for stuff like BLUESTORE_LEGACY_STATFS, it could complete the framework.

[guits]

This is specifically seen when updating from luminous or mimic to nautilus v14.2.2. There is a warning that:

From what I'm understanding this can even happen when upgrading from nautilus v14.2.0 or v14.2.1 to v14.2.2, right?

[liewegas]

See ceph/ceph#30264 ... bluestore will automatically convert itself (quickly) on startup, coming soon to a nautilus point release near you!

[leseb]

@liewegas I'm seeing a similar message 3 OSD(s) reporting legacy (not per-pool) BlueStore omap usage stats form 14.2.4 to 15.0.0-7878-g080d550.

[leseb]

As discussed, this will probably end up being documentation for an upgrade to 1.2.

[travisn]

@leseb Does this only affect upgrade to octopus? In this case it wouldn't affect 1.2 since octopus won't be supported until 1.3.

[leseb]

@leseb Does this only affect upgrade to octopus?

Yes.

In this case it wouldn't affect 1.2 since octopus won't be supported until 1.3.

Yes.

[BlaineEXE]

Is there something in progress here that I don't see being tracked? Maybe this should be moved to project v1.3?

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed in a week if no further activity occurs. Thank you for your contributions.

[stale]

This issue has been automatically closed due to inactivity. Please re-open if this still requires investigation.