Performance: Stretch and translate instead of redraw

[etowahadams]

Goal

Improve the performance of Gosling by reducing the number of times that shapes are redrawn. Instead of redrawing, we can stretch and translate the shapes when appropriate.

Pixi Basics

Reference tutorial

A PIXI.Container is like an empty box where you display and group display objects. These include PIXI.Sprites which renders a PIXI.Texture (an image), and PIXI.Graphics which are primitive shapes and lines.

For example, all of the marks (except text) in Gosling are PIXI.Graphics objects and text is a PIXI.Sprite.

Laying out the terms here to avoid confusion:

Texture:

• Created from various sources like images, canvases, videos, and even other Textures
• Cannot be directly displayed, but needs to be used with Sprites or Graphics.

Sprite:

• A simple display object that renders a single Texture.

Graphics:

• Low level object that allows you to draw shapes and lines (drawRect, drawCircle)
• Graphics objects can have other Graphics objects as children. Child Graphics objects will be rendered on top

Container:

• A display object that acts as a parent for other display objects, allowing you to group multiple Sprites, Graphics, or even other Containers together.

Pixi in HiGlass

Where does the PIXI.Container used in HiGlass get instantiated? And how do PIXI.Graphics objects that Gosling draws things to (such as Tile.graphics) to get added to that container?

• The main PIXI.Container used in HiGlass is created in the HiGlassComponent. All tracks that use Pixi will eventually add their PIXI.Graphics objects to other PIXI.Graphics objects which are inside of this PIXI.Container.
• Every Tile (the basic unit of data in HiGlass/Gosling) has a graphics property (Tile.graphics) which is set to a PIXI.Graphics object. Gosling adds shapes and lines to this graphics object. TiledPixiTrack, which GoslingTrack extends, adds the Tile.graphics objects to PixiTrack.pMain which is a child of the PIXI.Container created in HiGlassComponent

// In HiGlassComponent
this.pixiRoot = new GLOBALS.PIXI.Container();
this.pixiStage = new GLOBALS.PIXI.Container();
this.pixiRoot.addChild(this.pixiStage);

// In TrackRenderer
this.pStage = new GLOBALS.PIXI.Graphics();
this.currentProps.pixiStage.addChild(this.pStage);
const context = {scene: this.pStage}
new PixiTrack(context, options)

// PixiTrack.js
const { scene } = context 
this.scene = scene // therefore is pStage, a new PIXI.Graphics() object
this.pBase = new GLOBALS.PIXI.Graphics();
this.pMasked = new GLOBALS.PIXI.Graphics();
this.pMain = new GLOBALS.PIXI.Graphics();
this.scene.addChild(this.pBase);
this.pBase.addChild(this.pMasked);
this.pMasked.addChild(this.pMain);

// The hierarchy in summary: 
pixiRoot = new GLOBALS.PIXI.Container();
pixiStage = new GLOBALS.PIXI.Container(); // gets added to pixiRoot
pStage = new GLOBALS.PIXI.Graphics(); // gets added to pixiStage
pBase = new GLOBALS.PIXI.Graphics(); // gets added to pStage
pMasked = new GLOBALS.PIXI.Graphics(); // gets added to pBase
pMain = new GLOBALS.PIXI.Graphics(); // gets added to pMasked
// Then Tile.graphics will get added to pMain 

export type Tile = {
  tileId: string;
  remoteId: string;
  drawnAtScale: Scale;
  graphics: PIXI.Graphics; // Every Tile has a graphics property 
  tileData: TileData;
};

What is the relationship between GoslingTrack and PixiTrack?

• GoslingTrack extends BarTrack extends HorizontalLine1DPixiTrack extends HorizontalTiled1DPixiTrack extends Tiled1DPixiTrack extends TiledPixiTrack extends PixiTrack extends Track. Track is the base class for most HiGlass tracks.

Stretching and translating instead of redrawing

When a user zooms in, we want the marks that are drawn to simply be stretched and translated, rather than redrawn completely.

Currently when a user zooms or pans, all of the shapes and lines which are drawn to the PIXI.Graphics object (Tile.graphics) are cleared (Tile.graphics.clear()) and everything is redrawn.

In other HiGlass tracks, this is avoided by instead stretching or translating the shapes. For example, there is a function called stretchRects that is used by the gene annotation and Bed viewer track that stretches the drawn shapes.

/** When zooming, we want to do fast scaling rather than re-rendering
 * but if we do this too much, shapes (particularly arrowheads) get
 * distorted. This function keeps track of how stretched the track is
 * and redraws it if it becomes too distorted (tileK < 0.5 or tileK > 2) */
function stretchRects(track, graphicsAccessors) {
}

We would like to do something similar in Gosling.

When draw() gets called in GoslingTrack, we want to stretch and translate the marks when appropriate to avoid redrawing them over and over again.

For inspiration:

• HorizontalGeneAnnotationsTrack
• BedLikeTrack
  • Functions like movedY and zoomedY

[etowahadams]

Motivating example:

This multiple sequence alignment visualization often is very laggy.

Looking at the fame chart, we see that most of the time is spent drawing after zooming.

[dvdkouril]

Thanks for the summary, Etowah.

I looked very quickly at how one particular example ("Custom Mouse Events") is rendered, through a graphics debugger. It looks like on PIXI's side, there's really some sort of batching happening, even for the graphics elements. All the Graphics.drawRect etc. calls are combined into one WebGL draw call (drawElements):

In the screenshots you can see 3 stages during a single frame: 1) before the track is drawn, 2) first drawElements rendering 384 triangles, 3) second drawElements rendering another 384 triangles --- now that I think of it, it's a bit weird that the smaller parts is composed of the same number of triangles, but there could be some offscreen drawing, too.

This approach is somewhere in the middle performance-wise. Worst case would be if for each rectangle a separete call would be issued. The best case scenario would be if I saw here just one drawElementsInstanced call, rendering all rectangles in a single draw call. Maybe there's a lot of overhead in merging all the graphics primitives into these common buffers? That would make sense why there's slowdown upon interaction, cause maybe PIXI thinks it has to destroy all these buffers and rebuild.

I'm honestly not sure at this point how stretching the primitives would improve the performance, but it's possible I'm just missing something.

I'll continue investigating.

PS: That's a great example you linked.

[etowahadams]

Oooh that's cool! What is this graphics debugger called?

I do remember reading that small PIXI.Graphics objects are batched (in the performance tips). Not sure if that's the same thing that you're observing or not though.

Here's my current hypothesis for why things are slow:

1. User zooms in
2. With every new frame, drawTile() is called
3. Every time drawTile() is called, the shapes and lines drawn to the PIXI.Graphics object are cleared (function), and are completely redrawn (using for exampledrawRect() for rectangles, which you can see in the frame chart). Calling drawRect() for a bunch of rectangles at every frame is very slow.

override drawTile(tile: Tile) {
      // every time drawTile() is called all of the graphics are cleared and are redrawn
      tile.graphics?.clear();
      tile.graphics?.removeChildren();
      ...
}

Instead of redrawing everything at every frame, what certain HiGlass tracks do is to scale/move the graphics. The gist of it is this:

override drawTile(tile: Tile) {
      // every time drawTile(), the graphics are completely redrawn only some of the time 
      if (this.isSmallZoomChange()) {
           scaleGraphics(tile.graphics)
      } else {
           tile.graphics?.clear();
           tile.graphics?.removeChildren();
           redrawEverything(tile.graphics);
      }
      ...
}

Here is similar logic in HiGlass. Does this make sense?

[dvdkouril]

This is spectre.js, a Chrome extension. For desktop there are some much more comprehensive tools, like RenderDoc, but they don't usually support WebGL.

What you describe definitely sounds plausible. With the higlass approach, do you mean this stretchRects function (and I assume there are equivalent ones for different primitives)? I'll have to dig around more, because the abstraction here is not yet clear to me, i.e., what's behind the graphicsAccessor and how it gets translated through PIXI to actual draw calls.

I've got two more thoughts:

1. from WebGL perspective there has to be anyway a screen clearing at each frame, but it does sound like it's something a bit different than the tile.graphics?.clear() call from above. This one scratches all the objects and rebuilds the buffers it seems (which I guess is where the slowdown happens).
2. In my current understanding, I'm also wondering whether you could simply scale the objects no matter the zoom change. We would not be scaling sprites/textures, but graphics primitives which---while not exactly like vector graphics---still can be stretched. For example, if you have 4 vertices of a quad and you scale those positions, visually you still get the same quad, just bigger.

I guess the question is how PIXI internally deals with this scaling graphics primitives, which is something I still need to dive into.

[dvdkouril]

Looked more into this. I made a separate example to test out two approaches to drawing with the PIXI.Graphics objects.

Essentially I made two versions of the same graphic with some animation to simulate user panning:

1. what happens in gosling (AFAIK): each frame I clear the graphics object (graphics.clear(); graphics.removeChildren();) and redraw all the marks using graphics.drawRect() with different coordinates.
2. what I thought might be more efficient: I make the graphics objects only once (graphics.drawRect()), and then update the x coordinate: graphics.x = newPosition in each frame.

pixi-test-smaller.mov

Code: https://github.com/dvdkouril/pixi-graphics-playground
Live example: https://dvdkouril.github.io/pixi-graphics-playground/

Personally, I don't see a difference. I would expect that the second approach would be faster, but even with the approach used right now in Gosling I can smoothly animate 10.000 objects with no problem. It doesn't seem to me that the problem here is with clearing the Graphics object every frame and then remaking the graphics primitives from scratch. I would guess that there's more work that happens on Gosling side after each pan/zoom that causes the performance drop.

[etowahadams]

Nice experiment! Indeed it doesn't seem that there is much difference.

I see that you are using PIXI@7 here. Gosling uses PIXI@6. I know that 7 was a pretty big modernization rewrite so I wonder if that would make a big difference.

Another difference I see is that in Gosling there are many Graphics elements (one per tile), whereas here there is only one which the 10,000 rectangles get drawn to.

[dvdkouril]

Hm, that's a good idea, I'll try it out with v6.

The second thing should be also pretty easily testable.

[dvdkouril]

Okay, one of those things isolated: with pixi.js@6.3.0 I'm getting the same results:

pixi-v630-smaller.mov

(I'm assuming the flickering with 10k objects is just aliasing in action...)

[dvdkouril]

The second thing tested: still 10k of rectangles but divided into many Graphics objects.

pixi-heatmap-smaller.mov

I did have to do a bit more to free up the allocated child Graphics objects, to prevent memory leaking and crashing the browser. But I would assume this is covered in Gosling already:

//~ frame loop
const render = (frametime: number) => {
    const removed = graphics.removeChildren();
    for (const obj of removed) {
        obj.destroy();
    }
    graphics.clear();

    makeDummyHeatmapChart(graphics, values, offsetX, offsetY);

    requestAnimationFrame(render);
};
const requestID = requestAnimationFrame(render);

All in all, still doesn't seem like rendering is the issue here. I'll probably start looking generally into what happens after interaction.

[etowahadams]

All in all, still doesn't seem like rendering is the issue here

Yeah that looks smooth! Hmm so its not just that there are a bunch of Graphics objects that makes it slow.

I finally got the stretching idea somewhat working in #1012

Here's a rough comparison of zooming around on a laggy visualization. It does seem noticeably faster for some large specs.

Before: drawing takes 39% of the time when zooming

After drawTile takes 4.5% of the time

What I have so far works alright for stretching along the X axis, although it doesn't look great for text or non-square shapes (at least when we allow up to 2X stretching).

I still haven't figured out how to stretch in along the Y axis, which is needed for examples like this one https://gosling.js.org/?example=MATRIX

[dvdkouril]

It's great to see an actual measured improvement! Makes me wonder why I didn't really see a difference in my experiments.
However, this looks like a proof that not rebuilding the graphics objects in every re-draw should result in a performance gain.

Seeing this in action, and especially on the point mark example which turns into ellipses, makes me think: isn't this the best you can do with the stretching approach? Because essentially you scale up a "baked in" graphic, but what you really want in some of these representations is to only scale the positions, i.e., distances between the marks; the marks themselves stay constant in size. A quick illustration:

But I guess you could make it more efficient on the individual mark level, i.e., save the Graphics object for each mark and scale/position instead of clear and draw anew.