cluster and rancher stores now use advanced worker

[ghost]

Still requires a queueing system in place since some subscribes are fired prior to the advanced worker creation

Summary

Fixes #8837

Occurred changes and/or fixed issues

Technical notes summary

Areas or cases that should be tested

Areas which could experience regressions

Screenshot/Video

[richard-cox]

@Sean-McQ There's some test failures, can you take a look? I've added the milestone as well, can you create and link the issue as well?

[richard-cox]

• going from a context with no cluster (cluster manager with cluster of _) to one with a cluster (explorer) throws the below error

  (actual error) TypeError: Cannot read properties of undefined (reading 'revision')
    at Object.<anonymous> (subscribe.js:973:1)
  

  (in nextResourceVersion)
• there might be an issue with the way stop requests are handled for steve based stores. At the moment the resource watcher deletes the cached revision and doesn't send one when starting the subscription again. This means we get a resource.create for every matching resource (which doesn't scale). We need the same approach as in current master, where we try to sub with a best effort revision (which should normally be ok). The only way around this, given the current code base, is to bring in some of the sync worker to ui thread requests (not store / getters, but the promises over thread boundry stuff)

As a rough guide, here's what i think should be happening given the following scenarios

steve socket unsubs everything (CATTLE_WATCH_TIMEOUT_SECONDS related)

• we receive resource.stop per type
• in resource.stop handler, as we're still interesting in the type, we call watch with no revision supplied
• watch handles no revision by using a best effort to find one (cache.revision or highest resource revision from store)

ui subscribes with a revision that's too old

• In kind of parallel
  • resource.error received
    • set error state of type
    • call resync
      • resync starts afresh (http request & re-sub with new revision)
  • resource.stop received
    • as per resource.stop above... but as we're in an error state we don't rewatch

From testing it looks like we're handling the later ok, so it's just the former to look at

[richard-cox]

Didn't get too far today. I need to resolve the existing review comments by just understanding your updates. There's one additional comment from this review, containing two points. One regarding an exception in the new batch mutator. The other is to ensure the resource.stop --> watch process works the same as non-advanced worker (uses a revision from the store). that means fetching the revision from cross worker boundary (or alternatively find a way to add socket message resource revisions to the resource cache).

[richard-cox]

I've fixed some bugs

• Ensured advanced worker is used in cluster manager with no current cluster context
• Ensured resource.stop from advanced worker successfully kicks off watch
  • In current master socket state is tracked in the worker thread, but here we're checking state in the ui thread, so need to work around it
• Persist the error state instead of nuking all state when we receive a resource.stop
  • This will ensure the resource.stop given alongside the resource.error TOO_OLD is ignored
• Fix resource.error TOO_OLD --> resync params
  • these weren't correct, so the resync after TOO_OLD never worked

With the above fixes, and a tweak to force TOO_OLD (setting a revision of 1 in subscribe resource.stop if it came from the advanced worker), i've seen a successful set of events

1. resource.stop from BE
2. resource.start (bad revision) to BE
3. resource.stop & resource.error TOO_OLD from BE
4. http request to BE
5. resource.start (new revision) to BE

I had to merge instead of rebase, so the history is crazy. When merging please squash & merge

[git-ival]

@Sean-McQ @richard-cox Is there a description of what the advanced worker is meant to do and how to validate for QA? It's unclear what the current vs. expected behavior is.

[ghost]

@git-ival The advanced worker is a javascript web-worker, essentially a new thread outside of the main UI thread to process CPU intensive work that would otherwise block UX responsiveness in the browser tab.

In the case of the "Advanced Worker" feature, we've offloaded the websockets that would typically be waiting for real-time updates for the cluster, rancher, and management apis into separate worker threads (one for each socket) so that excessive incoming messages are handled separately and don't slow down or block the user's interaction with the Dashboard app in the browser.

Simple validation would probably just mean validating that the three websockets in question do in fact open in webworkers, you can do this by checking the Chrome devtools under "Network" and further selecting the "WS" (Websockets) tab to verify that the websockets in question opened in webworkers as opposed to in the main UI thread.

In the above screenshot, I can tell the Websockets in question are running in webworkers by the gear icon next to their "name" and I can tell they are separate worker threads by verifying that the filename under the "Initiator" column are all different (and random).

Apart from that, you'll want to perform some actions in another browser whose changes should appear in the active browser tab under test to ensure that the messages being received via websocket make their way back into the UI thread and update the app that the user is interacting with.

You can also validate performance improvements by scaling up a large cluster that generates excessive websocket traffic to the point of slowing down the UX and then switching on the "advanced worker" to verify that responsiveness improves with the feature toggled on.

Finally, you should notice a decrease in both the amount of memory consumed in the browser for clusters with a lot of resources but also significantly slowed memory growth over time as well. You'll want to test this on a list page with a lot of resources (such as pods or really anything under workloads).

Also, feel free to contact me on slack and I can send you a link to the recorded support training that Richard and I did recently in which we go into some detail for most of the recent performance features.

[git-ival]

Thanks for the clarification Sean! Just had a couple of follow-up questions.

• When you refer to scaling up a large cluster, are you referring to having a large # of workloads/objects, or a large # of downstream clusters in general?
• When you refer to memory consumption and performance, are you referring to the local machine running the browser or to the local cluster itself?

[floatingman]

@Sean-McQ Do you have answers to Iramis's questions?

[ghost]

Apologies, I thought I had answered these but I'm not seeing that here.

In this case, for scaling I'm referring to large clusters as those with a lot of resources or at least those that produce a lot of events since really what we're addressing here is the web-app's ability to keep up with potentially hundreds or thousands of socket messages per second.

For memory consumption, we're specifically referring to memory used by the web browser itself and not the cluster itself, the only real effect we should see on the cluster is that it shouldn't have to close as many sockets due to a slow client (for which the memory impact would be negligible).