Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix(ui): canvas stuttering, perf improvements #6102

Merged
merged 3 commits into from
Apr 2, 2024

Conversation

psychedelicious
Copy link
Collaborator

Summary

We have reports of canvas "stutters" (UI-blocking GC events) on 3.7.0 and 4.0.0. I'm not sure what changed in v3.7.0, could have always been an issue and it's just a coincidence that it's popped up now.

PR introduces a few perf optimizations for canvas, benchmarked across Chrome, FF and Safari.

  • Prevent canvas history from growing beyond the max (prevents potential memory leaks & reduces blocking GC events)
  • Use more efficient algorithm for deep cloning
  • Reduce max canvas history from 128 to 100

Related Issues / Discussions

Discord thread for canvas "stutters": https://discord.com/channels/1020123559063990373/1224446493323296809

QA Instructions

To replicate the excessively large history on main, draw about 50 mask lines. You can just click in the same spot over and over again rather than actually dragging the mouse to draw a line. Then press Shift + C about 100 times, and finally draw a line. Depending on your hardware and browser, we expect some lag at this point.

Run this in the JS browser console:

let request = window.indexedDB.open("invoke", 1);

request.onsuccess = (event) => {
  const db = event.target.result;
  const transaction = db.transaction(["invoke-store"], "readonly");
  const store = transaction.objectStore("invoke-store");
  const getRequest = store.get("@@invokeai-canvas");
  
  getRequest.onsuccess = (event) => {
    data = JSON.parse(event.target.result)
    console.log("Past layer states:", data.pastLayerStates.length);
  };
};

This prints the size of the undo history - we expect it to be around 150, which seems to be the danger zone. It'll be different depending on hardware and browser, though.

Next, switch to this PR. You'll need to stop the dev server, pnpm i and then start it again.

You should be back to where you left of in the canvas. Draw a single line with the mask. This should reset your history - run the snippet above again, should say 100 (or less). Draw around, we expect improved performance from here on out.

Merge Plan

n/a

Checklist

  • The PR has a short but descriptive title, suitable for a changelog
  • Tests added / updated (if applicable) n/a
  • Documentation added / updated (if applicable)

There were two ways the canvas history could grow too large (past the `MAX_HISTORY` setting):
- Sometimes, when pushing to history, we didn't `shift` an item out when we exceeded the max history size.
- If the max history size was exceeded by more than one item, we still only `shift`, which removes one item.

These issue could appear after an extended canvas session, resulting in a memory leak and recurring major GCs/browser performance issues.

To fix these issues, a helper function is added for both past and future layer states, which uses slicing to ensure history never grows too large.
- Add and use more performant `deepClone` method for deep copying throughout the UI.

Benchmarks indicate the Really Fast Deep Clone library (`rfdc`) is the best all-around way to deep-clone large objects.

This is particularly relevant in canvas. When drawing or otherwise manipulating canvas objects, we need to do a lot of deep cloning of the canvas layer state objects.

Previously, we were using lodash's `cloneDeep`.

I did some fairly realistic benchmarks with a handful of deep-cloning algorithms/libraries (including the native `structuredClone`). I used a snapshot of the canvas state as the data to be copied:

On Chromium, `rfdc` is by far the fastest, over an order of magnitude faster than `cloneDeep`.

On FF, `fastest-json-copy` and `recursiveDeepCopy` are even faster, but are rather limited in data types. `rfdc`, while only half as fast as the former 2, is still nearly an order of magnitude faster than `cloneDeep`.

On Safari, `structuredClone` is the fastest, about 2x as fast as `cloneDeep`. `rfdc` is only 30% faster than `cloneDeep`.

`rfdc`'s peak memory usage is about 10% more than `cloneDeep` on Chrome. I couldn't get memory measurements from FF and Safari, but let's just assume the memory usage is similar relative to the other algos.

Overall, `rfdc` is the best choice for a single algo for all browsers. It's definitely the best for Chromium, by far the most popular desktop browser and thus our primary target.

A future enhancement might be to detect the browser and use that to determine which algorithm to use.
This should further insulate canvas from excessive GCs.
@psychedelicious psychedelicious force-pushed the psyche/fix/ui/canvas-history-leak branch from 217a06b to 35d1a48 Compare April 2, 2024 12:32
@github-actions github-actions bot added frontend-deps PRs that change frontend dependencies frontend PRs that change frontend files labels Apr 2, 2024
@hipsterusername hipsterusername merged commit b6ad33a into main Apr 2, 2024
14 checks passed
@hipsterusername hipsterusername deleted the psyche/fix/ui/canvas-history-leak branch April 2, 2024 12:48
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
frontend PRs that change frontend files frontend-deps PRs that change frontend dependencies
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants