Add new command `fluvio cluster resume`

[digikata]

On some systems fluvio cluster start can fail to recognize started spus and timeout despite the spus reporting status. (MacOs Sonoma 14.1.2)

Problem

Starting up with multiple spus, creating a topic with replicas higher than 1, then restarting with a different number of spus causes this problem. This can be fixed by starting the cluster with the original start options or a fluvio cluster delete and a fresh fluvio cluster start

$ fluvio cluster start --spus 3
$ fluvio topic create --replication 3 repltopic
$ fluvio produce repltopic
Ok!
> b
Ok!
> c
Ok!
> ^C
$ fluvio cluster shutdown

$ fluvio cluster start        
📝 Running pre-flight checks
    ✅ Local Fluvio is not installed
🎉 All checks passed!
✅ Local Cluster initialized
👤 Profile set
✅ SC Launched
🤖 Starting SPU: (1/1) \
🖥️ 3/1 SPU confirmed, 15 seconds elapsed 

Solution

disallow fluvio cluster start after a fluvio cluster shutdown. Instead add new command for fluvio cluster resume

Community Note

This issue is of intermediate to expert difficulty, but Infinyon mentoring is available if you are interested in taking it on.

[avikam]

i think the problem is coming from confirm_spu(). It actually confirmed more SPUs (based on the admin interface) than needed and therefore doesn't leave.

if ready_spu /* ~admin.all().filter() */ == spu /* cli request */ as usize {
    ...
    return Ok(()); // never happens 
}

In fact, the bug reproduces whenever start is called with less SPUs and doesn't with greater or equal SPUs, than the previously specified under start.

Admin is listing SPUs from the metadata directory. The bug is also fixed if the spec files are deleted between different starts. I think it makes sense for shutdown to delete the SPU spec files, since the SPU processes are killed anyway. wdyt?

Here is an example of the above:
#3901

[digikata]

Hi @avikam, that's a good analysis of what is causing the immediate problem. But fluvio cluster delete is what should delete spec files and delete data. The root issue is that we should really differentiate a couple of different command pairs for a cluster running in local mode.

fluvio cluster start|delete

• start: creates creates a new cluster, but should fail if there is already a cluster running or in a shutdown state
• delete: this is what should delete local mode spec files

To align with other parts of the cli, start|delete should maybe actually be create|delete (@sehz @nacardin what do you think?)

fluvio cluster shutdown|resume

• shutdown shuts down running processes but should save data and any configuration needed to resume running
• resume (new command for this issue), restarts with previous configuration, should not accept configuration flags, but
• shutdown|resume should only be meaningful for a cluster in 'local' mode. It is pretty much meaningless at the moment for K8 fluvio clusters.

The smallest increment of work to progress this issue might be to add the logic to return an error and refuse to start if a fluvio cluster start is attempted when there are any local spu spec files.

[avikam]

Yeah makes sense!
The reasons I considered this approach are that it looks like it's closer to the local k8s behavior (deleting the CRDs) and it maintains the simple start/shutdown interface. It's also consistent with the rest of the setup types (that don't have 'resume')
But def see the advantage of resuming already defined SPUs.

[sehz]

Consider this issue before: #1672.

Complexity is that each infrastructure types requires different type to save system state. Personal cluster behave different than shared cluster. For personal cluster, it's ok to save in the file but shared cluster like K8 will need to behave differently.

Cleaner solution is to abstract out Cluster management further delegate to implementation specific behavior.

[digikata]

Fully implemented, thanks to the work of @avikam!