[Explorer] Extract URL state persistence from the component

[weltenwort]

📓 Summary

The LogExplorer component that the log_explorer plugin exports currently inherits the URL state synchronization from Discover. In order to make it reusable in different UIs we want to prevent it from modifying the URL so the consumer has control over that.

• 🛑 blocked by: [Log Explorer] Convert log explorer profile into standalone app #164493

✔️ Acceptance criteria

• The LogExplorer component doesn't interact with the URL in any way.
• It exposes its state such that it can be initialized and reacted-to by the consuming app.
• There is a new Observability Log Explorer state machine that handles initialisation
• The Observability Log Explorer app registered by observability_log_explorer synchronizes the necessary parts of the state to the URL, such that...
  • a page reload restores these parts of the state.
  • it can be migrated as the schema changes.

Please see #165255 (comment) for more specific implementation details.

💡 Implementation hints

• Maybe versioning the url state would help in managing url schema migrations?

[elasticmachine]

Pinging @elastic/infra-monitoring-ui (Team:Infra Monitoring UI)

[Kerry350]

Whilst this issue is focussed around URL state extraction, after discussions with Felix and Mohamed we considered whether this should reach a bit broader to state syncing in general. Here I've tried to provide an overview of how things work currently, and potentially some ideas for how we can move forward.

Here we'll be referring to 3 layers: Observability Log Explorer (top level consumer) > Log Explorer (reusable log explorer component that registers profile customisations etc) > Discover Container (essentially an embeddable wrapper around Discover's main route with an injectable scoped history etc). These are mounted left to right.

Broadly speaking we want a way to:

• Set initial state (for the Discover Container)
• React to state changes (from the Discover Container)
• And possibly provide state changes (to the Discover container)

We can apply some forward thinking here and try to ensure that the Log Explorer layer communicates (with top level consumers) in a manner that will suit our planned persistence layer: https://github.com/elastic/logs-dev/issues/26

The issue we have here is that the Explorers and Discover were developed at very different times, so we are trying to retrofit this state syncing on top of something that's existed for a very long time.

Discover's state container

Discover provides a "state container", this is also provided to the customisation callbacks as a parameter.

This state container wraps several child portions of state. Not everything here is relevant to us, but we would be interested in globalState (time range, refresh interval etc), appState (columns, filters etc), internalState (we need this for the custom filters), stateStorage (this syncs the state containers with a URL state storage container). This state container more or less holds all of the state handling. The global / app / internal containers utilise createStateContainer from Kibana utils. It's important to note the internal state isn't synced to the URL via the state storage container.

How are we interacting with this state now?

Right now our only top level consumer is Observability, and this plugin has no knowledge of Discover state internals. As we move down a layer to the Log Explorer there is knowledge of, and interaction with, Discover's state container.

Most of this interaction is isolated within the url_state_storage_service:

• stateContainer.appState.subscribe - an appState container subscription is setup to monitor for index changes (technically we refer to this as listenUrlChange but this is just on the state container, not state storage)
• const { index } = stateContainer.appState.getState(); - getState is used for initialising state to the machine
• stateContainer.stateStorage.get<ControlPanels>(CONTROL_PANELS_URL_KEY); - get is used here to initialise the control panels. Important note: Discover itself does not sync the control panels to the URL, as it's part of internal state, but as we pass our scoped history instance to the Discover container this state storage call will still retrieve the key via the URL that the consumer (Log Explorer) sets.
• When a custom filter is published this is set on the internal state container, and the fetchData action is called.

const filtersSubscription = context.controlGroupAPI.onFiltersPublished$.subscribe(
	(newFilters) => {
		stateContainer.internalState.transitions.setCustomFilters(newFilters);
		stateContainer.actions.fetchData();
	}
);

• When custom filters change in the URL, these are synced to the machine:

stateContainer.stateStorage.change$<ControlPanels>(CONTROL_PANELS_URL_KEY)
.subscribe((controlPanels) => {
	if (!deepEqual(controlPanels, context.controlPanels)) {
		send({ type: 'UPDATE_CONTROL_PANELS', controlPanels });
	}
});

• updateStateContainer (used to sync some initial values to the state container if not set) has knowledge of appState.getState(), appState.isEmptyURL(), and appState.update().
• set is called on the state storage container to update the controlPanels key (again, this is via our scoped history instance):

stateContainer.stateStorage.set(
	CONTROL_PANELS_URL_KEY,
	cleanControlPanels(controlsPanelsWithId),
	{ replace: true }
);

Outside of the url_state_storage_service there are only two other direct interactions with the state container:

• The data view service- this calls both the createAndAppendAdHocDataView() and setDataView() actions when needed on the top level state container.
• The profile customisation callback uses the app state observable to create a merged state on the state$ observable passed by the consumer (merges Discover's app state with the state machine context). I don't think this state is actually consumed anywhere though.

We can see that the Log Explorer currently has a lot of knowledge of the Discover state container. This is in no way meant to criticise the current implementation and interaction with the Discover state container, this was the only way to facilitate the functionality that the Log Explorer needed based on the architecture that existed at the time (and still exists now).

Whilst we are somewhat protected by types, there is no "formal" contract here.

How can we improve this?

Here are some initial ideas. It's important that we try to create a good solution, but one that is only as complicated as we need for now (with the long term goal being all of the Discover components being extracted).

Consumer <-> Log Explorer interaction isn't as complicated, here we can likely create an actor / state machine that can handle "talking" our future persistence layer language. When the consumer receives state updates they are completely in control of how these are persisted in the URL, as long as they can be translated back to the Log Explorer interface.

The Log Explorer <-> Discover container interaction is more complicated.

• If all we were interested in was setting initial state, and receiving state updates (but not setting them) we could provide a custom state storage container (adhering to the state storage interface, which is really just get, set, change$). Here, rather than all of the state storage updates going to the URL, they'd go to the custom container. However, this only handles the app / global case (not internal) and we'd receive state formatted for the URL, not necessarily "raw". And, whilst setting some state back might work, some cases require calling the specific actions for further processing etc. Not fullproof.

• I'd propose we create some sort of consumer state container, this would extend the DiscoverStateContainer type. It would make the use of the URL state storage container optional (some log explorer consumers might just want that default behaviour). This container would subscribe to the $change observables on the internal containers, and push those state changes back to the log explorer as needed (here, the other layer would pass on these changes to top level consumers if needed). We would also expose the specific actions we want access to (setDataView etc). We could provide this container as a customisation (it's not UI, but it still customises something), or we can just pass it as an optional parameter with history and so on.

This would allow us to move forward in a way that doesn't require big changes to Discover.

[tonyghiani]

@Kerry350 great analysis 👏 The plan looks good, since we moved the Log Explorer into our own app it makes sense to get more control over the state as explained in your proposal.

A quick note on the following:

The profile customisation callback uses the app state observable to create a merged state on the state$ observable passed by the consumer (merges Discover's app state with the state machine context). I don't think this state is actually consumed anywhere though.

The state$ observable was introduced in #165464 to notify the Observability Log Explorer consumer about changes on the internal dataset selection, filters etc to compose the Discover fallback link.
IIRC this is the only use case we have, and I guess this could be removed in favour of the new Consumer <-> Log Explorer interaction system that will be implemented 👌

[Kerry350]

@tonyghiani

The state$ observable was introduced in #165464 to notify the Observability Log Explorer consumer about changes on the internal dataset selection, filters etc to compose the Discover fallback link.

Ah, thank you for the heads up 🙏

[Kerry350]

Felix and I had some good discussions last week about an architecture we can move forward with, taking into account the following: #165255 (comment) and using the original URL issue as the start point: #165255 (comment). Also taking into account we don't want to make any large changes to Discover, as the future would be using the separate Discover components.

This will almost certainly change a bit once implementation begins.

• The Discover container will be changed in much the way described here: [Explorer] Extract URL state persistence from the component #165255 (comment), e.g. a new wrapper that extends the current container, subscribes to changes on the Observables of the inner state containers and emits the changes, and exposes actions in a contractual way.

For the following examples we've tried to consider that consumers might not be using state machines, so there should be an agnostic method for accessing state / actions. Here Observables would be the agnostic communication channel (as these can easily be hooked up to state machines if wanted).

We'll introduce a controller concept (this is just pseudocode to try and show some concepts):

const { state$, event$, actions, controller } = useObservabilityLogExplorerController({
    initialState: {
      filters: serviceFilters,
      timeRange: stateFromUrl.timeRange,
    },
    options: {
      tableRows: {
        rowActions: (previousRowActions) => [firstAction, ...previousRowAction],
      }
    },
  });

The controller would essentially take care of instantiating the relevant Log Explorer state machine, observables etc, and can therefore be passed on to the Log Explorer component. For Observability we'll use a specific Obs version. options shows how we could extend this to override functionality within the Log Explorer (similar to how Discover customisations work), the idea here would be that the previous state is passed in, and a new state is returned (this keeps state machine concepts away from the consumer, but makes hooking this up to actions behind the scenes pretty seamless).

The state$ observable will emit state changes (machine context, and possibly the state string itself, or a consumer friendly version), the event$ observable will emit event notifications (note: these are strictly events that we've deemed "notifications" / public, and will pass also a full state snapshot after the event, plus the slice of state that changed based on the event).

Actions will expose actions that can be undertaken by the consumer, behind the scenes this is essentially dispatching an event to the machine, but we don't want the consumer to know about any state machine internals.

Rendering then could loosely look like this:

<button disabled={isValidAction(actions.reload)} onClick={() => actions.reload()}>Reload Logs</button>
<ObservabilityLogExplorer
  controller={controller}
/>

(<ObservabilityLogExplorer /> would now render a <LogExplorer / forwarding on the controller).

A consumer that isn't using state machines is now more or less ready to go, they've passed some initial state (their job to instantiate this), they have actions to call, and they can observe the two observables in the best way they see fit (React hooks, hooking up to Redux etc).

This explains the changes for the Log Explorer layer, and lastly we'll need the Observability Log Explorer layer. In the Observability Log Explorer we are using state machines. During the Observability Log Explorer initialisation process we will also create a controller, we'll then store these Observables etc into context (this means we can essentially call the same functions that the hook would call for other consumers). We can then use invoke to listen to the observable events and respond as needed. Once we move into an initialised state (and have our controller etc) we can render the <ObservabilityLogExplorer />.

For this issue specifically we'll focus on adding these changes to facilitate URL state extraction. As such, in the Observability Log Explorer, we'll have a new state machine that handles initialising from the URL, creating a controller, and passing the controller the initialState, we can also use the new Observables to synchronise state back to the URL.

[tonyghiani]

@Kerry350 thanks for the in-depth explanation! A quick question:

• Decoupling the controller logic from the Log Explorer means that consumers using actions or reacting to state updates won't have immediate access to those, since the internal machines are not initialized until the internal stateContainer is provided to the controller, is that correct?
  What I mean is, in pseudocode:

const { state$, event$, actions, controller } = useObservabilityLogExplorerController(...);

// `isValidAction(actions.reload)` is meant to control whether the action is already available?
<button disabled={!isValidAction(actions.reload)} onClick={() => actions.reload()}>Reload Logs</button>

// Here actions and state updates are not yet available.
<ObservabilityLogExplorer
  controller={controller}
/>

// Here it rerenders because the controller got access to the internal stateContainer and initialized actions, state$ etc.

Sorry for the primitive and not structured flow example, it my understanding of how it work correct?

[Kerry350]

// isValidAction(actions.reload) is meant to control whether the action is already available?

So, this might end up looking a little different but the idea is that we'd just have a way to know when an action was available / in a valid state to perform. I did a similar thing here in the custom integrations package whereby the "dispatchable event" would be available or undefined. Then the consumer doesn't need to perform a side state check also.

// Here actions and state updates are not yet available.

Roughly (again, this is just hypothetical and may look different after code changes) the actions and the state should be immediately available (using "hot" observables), but there just might not be any actions that can be performed yet, and the state may only contain the initial state. However, the <LogExplorer /> itself (which the <ObservabilityLogExplorer /> would be wrapping) would still only be rendering relevant content once fully initialised.

since the internal machines are not initialized until the internal stateContainer is provided to the controller, is that correct?

I think the hierarchy will change slightly here. You are right, we need to go through the customisation callback flow, state container setup, etc before we're fully initialised. I need to play around a bit to see how this will look with the new "extended" state container. It's possible this may be instantiated a little earlier. Basically, I don't have a 100% answer on this yet 😅 This should become clearer in implementation (which I've started on today) 🤞

We can always have some additional UI here too, whilst the <LogExplorer /> renders the <DiscoverContainer /> directly at the moment, it could also render a loading state if needed (for example). The LogExplorer will have a full understanding of how the machine from the controller works.

[tonyghiani]

Thanks for clarifying those questions!

[elasticmachine]

Pinging @elastic/obs-ux-logs-team (Team:obs-ux-logs)