Default Shell State Consistency

[jtkech]

In a multi instances env with tenant settings coming from a shared source as the database, if it fails due to a bad connection or the absence of the needed table, the Default tenant is in an inconsistent state.

Not a big issue when we work with only one OC instance (just needs to be restarted), and this is what we recommend before having setup the Default tenant on which we rely e.g. to keep in sync tenants.

But while testing the Tenant Removal with a local K8S deployment allowing multiple OC replicas, when I forget to specify only one OC replica before a fresh setup, it is less easy to find the OC pods to restart.

So here the main point is when _shellSettingsManager.LoadSettingsAsync() fails from different places.

• When it fails from ShellHost.InitializeAsync(), here we still let the exception be thrown but we don't mark the ShellHost as _initialized, so that when the database connection/table are okay, on a new request we can create a setup context and render the setup form in place of a blank page.

• We also call _shellHost.InitializeAsync() In ModularBackgroundService if the .ShellWarmup option is true, here we use a try/catch block to log an error but without interrupting the background service, so that when the Default tenant is finally setup the background service will start its periodic loop.

• TODO: In a multi instances env, the Default setup form may be rendered by one instance, but the setup executed by another one. In that case the Default tenant of the first instance will always be Uninitialized because tenant syncing rely on the Default tenant being running.

  Here the solution, in DistributedShellHostedService, would be to periodically check directly from the shared tenants settings source if the Default tenant was setup by another instance.

  Maybe here or in a separate PR. Updated: Okay done here.