Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Investigate WebGL Rendering with Pixi.js #350

Open
samreid opened this issue Jan 17, 2015 · 10 comments
Open

Investigate WebGL Rendering with Pixi.js #350

samreid opened this issue Jan 17, 2015 · 10 comments

Comments

@samreid
Copy link
Member

samreid commented Jan 17, 2015

Last week, we decided to investigate WebGL Rendering with Pixi.js. Our plan is to spend an initial 20 hours or so on it, and to check in with @ariel-phet regarding status after about 10 hours. Some discussion about Pixi took place in #346 and I will move pertinent conversations to this issue in subsequent comments.

There are 3 possibilities for how to proceed based on the results of this investigation:

  1. Scenery WebGL support is exclusively through Pixi.js
  2. Scenery WebGL support is exclusively through our own custom WebGL renderer module
  3. Scenery WebGL support is met through a combination of Pixi.js and our own custom WebGL renderer module (most likely through separate canvas elements though possible through a shared single canvas, when used in the same application)

Work on our own custom WebGL renderer module will proceed in parallel with our Pixi.js investigation.

@samreid
Copy link
Member Author

samreid commented Jan 17, 2015

In #346 @englercj discussed Pixi v3 as a possible target

Just to pop in and say a quick note about pixi, we should be stablizing the API for v3 over the next couple weeks. So integration can start being planned then, and also in this rewrite we made the webgl renderer much more extensible, flexible, and powerful. We can talk more details as we get closer to a stable core.

@samreid responded:

I have read through pixijs/pixijs#1296 and I'll be interested to take a closer look when the API is more stable. In the meantime, we will probably begin experimenting with Pixi v2 to test the waters.

@englercj also described Pixi features that @samreid asked about in #346

  • Gradients, there is no "feature" for it but a canvas texture works and so does a custom shader.
  • Patterns, TilingSprite is a fast tiled sprite implementation
  • Transparency, We support multiple blend modes and transparency on sprites/containers
  • Clip Regions, We support masks via Graphics (primitives) in v2 and we will also support alpha masks in v3
  • Text bounds, supported and will be even better in v3
  • Dynamic spritesshet, not automatically combined into a spritesheet. You can draw them to canvas and use that as a single texture so they are batched.
  • Context loss handle, supported
  • Using SVG as a texture, supported but the scaling factors are not. We rasterize it.
  • Renger webgl scene to a texture, supported.

samreid added a commit that referenced this issue Jan 20, 2015
samreid added a commit that referenced this issue Jan 20, 2015
samreid added a commit that referenced this issue Jan 20, 2015
samreid added a commit to phetsims/sherpa that referenced this issue Jan 20, 2015
samreid added a commit to phetsims/chipper that referenced this issue Jan 21, 2015
samreid added a commit that referenced this issue Jan 21, 2015
samreid added a commit that referenced this issue Jan 21, 2015
jonathanolson added a commit that referenced this issue Jan 21, 2015
jonathanolson added a commit that referenced this issue Jan 21, 2015
jonathanolson added a commit that referenced this issue Jan 21, 2015
@samreid
Copy link
Member Author

samreid commented Jan 22, 2015

Yesterday working with @jonathanolson we determined that Pixi has support for some of the features that would be tricky to implement ourselves: opacity (including for parent nodes) and clipping (including nested clipping regions). Some text glyphs are getting clipped, but this problem seems difficult to avoid with any technology. So pixi seems promising so far.

@samreid
Copy link
Member Author

samreid commented Jan 23, 2015

I've been working on PixiNode. Here is my first attempt at using it to render a sim:

image

@samreid
Copy link
Member Author

samreid commented Jan 23, 2015

From skype:

[1/22/15, 10:55:22 PM] Sam Reid: Using PixiNode strategy it is unclear how to let scenery handle input events for original scenery node but not its rendering. Thoughts?
[1/23/15, 12:09:36 AM] Jonathan Olson: It doesn't handle it well. Probably makes sense to integrate as a block and have Scenery handle somewhat specially
[1/23/15, 12:10:29 AM] Sam Reid: So the original way I was doing renderer:’pixi’ with PixiBlock, or some sort of PixiBlock/PixiNode hybrid?
[1/23/15, 12:10:46 AM] Jonathan Olson: presumably with no PixiNode at all
[1/23/15, 12:10:55 AM] Sam Reid: by the way, I really liked the way PixiNode was working—very straightforward
[1/23/15, 12:11:19 AM] Sam Reid: Will PixiBlock be able to give us the 1:1 node mapping like we were getting with PixiNode?
[1/23/15, 12:11:56 AM] Jonathan Olson: well blocks have the advantage of being able to handle part of a subtree
[1/23/15, 12:12:08 AM] Jonathan Olson: what do you mean 1:1 node mapping?
[1/23/15, 12:12:45 AM] Sam Reid: for example, we had the a tree like this:
[1/23/15, 12:13:20 AM] Sam Reid: var b = new Text(); var c = new Path(); var a = new Node({children:[b,c]};
[1/23/15, 12:13:33 AM] Sam Reid: in PixiNode, I had one DisplayObjectContainer for a
[1/23/15, 12:14:00 AM] Sam Reid: when a’s transform (local) changed, I just updated the DisplayObjectContainer’s (local) transform: easy!
[1/23/15, 12:14:13 AM] Jonathan Olson: SVG basically accomplishes a very similar thing
[1/23/15, 12:14:17 AM] Sam Reid: ok
[1/23/15, 12:14:20 AM] Jonathan Olson: but it's more of a 1:1 instance mapping
[1/23/15, 12:14:25 AM] Jonathan Olson: because of duplication
[1/23/15, 12:14:40 AM] Sam Reid: ok sounds good, so I’ll resume investigation of pixiblock?
[1/23/15, 12:15:06 AM] Jonathan Olson: yes
[1/23/15, 12:15:09 AM] Sam Reid: “because of duplication” ???
[1/23/15, 12:15:19 AM] Jonathan Olson: can Pixi handle DAGs?
[1/23/15, 12:15:27 AM] Jonathan Olson: (SVG can too, but with performance losses)
[1/23/15, 12:15:28 AM] Sam Reid: it was one thing we didn’t test yet AFAIK
[1/23/15, 12:15:33 AM] Sam Reid: testing….
[1/23/15, 12:19:33 AM] Sam Reid: no DAG. Adding node to 2nd parent removes it from 1st parent.
[1/23/15, 12:20:01 AM] Jonathan Olson: so it will be very similar to SVG
[1/23/15, 12:20:16 AM] Jonathan Olson: I'd recommend reading (code reviewing?) SVGBlock / SVGGroup
[1/23/15, 12:20:47 AM] Sam Reid: grep -rl ‘svg’ ./ | xargs sed -i ’s/svg/pixi/g’
[1/23/15, 12:21:17 AM] Sam Reid:
[1/23/15, 12:21:33 AM] Sam Reid: I’ll look at SVG support

@samreid
Copy link
Member Author

samreid commented Feb 8, 2015

I added no-frills pixi rendering. Performance is very slow on iPad3 when using pixi for the entire sim (though very fast for just a few moving objects). When testing with http://localhost:8080/forces-and-motion-basics/forces-and-motion-basics_en.html?joistRenderer=pixi&screens=2 I wanted to count the number of draw calls pixi is making to see if they are batching draw calls. I instrumented each drawElements call in pixi-dev.js with a call to phetAllocation so they would be counted. Then in the console, I observed:

> x.loop.count
74
> x.drawElements.count
6364
> x.drawElements.count/74
86

This means 86 drawElements calls were being made per frame. To see if this correlated with any of the sim scene graph numbers, I used popupDebug to get information about the scene graph. I observed:

Display Summary
[427w, 543h] frame:5210 input:true cursor:default
Nodes: 205
Instances: 205
Drawables: 81
Block Summary
PixiBlock#155-commonAncestor (79 drawables)

Perhaps the number of drawElements calls is scaling with the number of drawables?

Many sources cite that minimizing the number of draw calls is one (of many) important ingredients to getting good WebGL performance, so that's why I wanted to look into this.

@englercj
Copy link

englercj commented Feb 8, 2015

We do automatic batching, and we count draw calls for you as well (renderer.drawCalls). Are you using the same base texture for the tings you are drawing? Try to make containers have children with common base textures, or group common basetexture sprites together in the scene graph.

Basically we iterate forward and when we encounter a new texture on a sprite we flush the batch and start a new one. So if your scene graphs looks like [texture1, texture2, texture1, texture2] there will basically be no batching.

@samreid
Copy link
Member Author

samreid commented Feb 10, 2015

Are you using the same base texture for the tings you are drawing?

I'm not too familiar with Base Texture, even after taking a look at this http://www.goodboydigital.com/pixijs/docs/classes/BaseTexture.html Is it a way of doing sprite sheeting? Here's a picture of one of our interactive simulations:

image

Here is the list of node types as they appear in the rendering order. Generally each Image instance mentioned is from a distinct image file. You can also see that there are many shapes/text/images interleaved and no group of related images in a single layer. Is there a preferred Pixi way for rendering a scene like this?

Rectangle#1 (MotionScreenView/Rectangle)  
Rectangle#2 (MotionScreenView/Rectangle)  
Image#3 (MotionScreenView/MovingBackgroundNode/Image) 
Image#4 (MotionScreenView/MovingBackgroundNode/Image) 
Image#5 (MotionScreenView/MovingBackgroundNode/Image) 
Image#6 (MotionScreenView/MovingBackgroundNode/Image) 
Image#7 (MotionScreenView/MovingBackgroundNode/Image) 
Image#8 (MotionScreenView/MovingBackgroundNode/Image) 
Image#156 (MotionScreenView/MovingBackgroundNode/Image) 
Rectangle#9 (MotionScreenView/Rectangle) 
Rectangle#10 (MotionScreenView/Rectangle) 
Image#13 (MotionScreenView/PusherNode/Image) 
Image#44 (MotionScreenView/Image) 
Text#45 (MotionScreenView/Text) 
Path#46 (MotionScreenView/HSlider/Path) 
Text#47 (MotionScreenView/HSlider/Text) 
Path#48 (MotionScreenView/HSlider/Path) 
Path#49 (MotionScreenView/HSlider/Path) 
Path#50 (MotionScreenView/HSlider/Path) 
Path#51 (MotionScreenView/HSlider/Path) 
Path#52 (MotionScreenView/HSlider/Path) 
Text#53 (MotionScreenView/HSlider/Text) 
Path#54 (MotionScreenView/HSlider/Path) 
Path#55 (MotionScreenView/HSlider/Path) 
Path#56 (MotionScreenView/HSlider/Path) 
Path#57 (MotionScreenView/HSlider/Path) 
Path#58 (MotionScreenView/HSlider/Path) 
Text#59 (MotionScreenView/HSlider/Text) 
Rectangle#60 (MotionScreenView/HSlider/Rectangle) 
Rectangle#61 (MotionScreenView/HSlider/Rectangle) 
Rectangle#63 (MotionScreenView/HSlider/SliderKnob/Rectangle) 
Circle#64 (MotionScreenView/HSlider/SliderKnob/Circle) 
Circle#65 (MotionScreenView/HSlider/SliderKnob/Circle) 
Circle#66 (MotionScreenView/HSlider/SliderKnob/Circle) 
Circle#67 (MotionScreenView/HSlider/SliderKnob/Circle) 
Circle#68 (MotionScreenView/HSlider/SliderKnob/Circle) 
Circle#69 (MotionScreenView/HSlider/SliderKnob/Circle) 
Rectangle#70 (MotionScreenView/Rectangle) 
Text#71 (MotionScreenView/Text) 
Rectangle#72 (MotionScreenView/ArrowButton/Rectangle) 
Rectangle#73 (MotionScreenView/ArrowButton/Rectangle/Rectangle) 
Path#74 (MotionScreenView/ArrowButton/Path)  
Rectangle#75 (MotionScreenView/ArrowButton/Rectangle)  
Rectangle#76 (MotionScreenView/ArrowButton/Rectangle/Rectangle)
Path#77 (MotionScreenView/ArrowButton/Path)  
Circle#78 (MotionScreenView/GaugeNode/Circle) 
Path#79 (MotionScreenView/GaugeNode/Path)  
Text#80 (MotionScreenView/GaugeNode/Text)  
Circle#81 (MotionScreenView/GaugeNode/Circle) 
Path#82 (MotionScreenView/GaugeNode/Path) 
Path#83 (MotionScreenView/GaugeNode/Path) 
Rectangle#84 (MotionScreenView/MotionControlPanel/Panel/Rectangle)  
Rectangle#85 (MotionScreenView/MotionControlPanel/Panel/VBox/VerticalCheckBoxGroup/HBox/CheckBox/Rectangle)  
Path#86 (MotionScreenView/MotionControlPanel/Panel/VBox/VerticalCheckBoxGroup/HBox/CheckBox/FontAwesomeNode)  
Path#88 (MotionScreenView/MotionControlPanel/Panel/VBox/VerticalCheckBoxGroup/HBox/CheckBox/Path) 
Text#89 (MotionScreenView/MotionControlPanel/Panel/VBox/VerticalCheckBoxGroup/HBox/CheckBox/Path/Text) 
Rectangle#90 (MotionScreenView/MotionControlPanel/Panel/VBox/VerticalCheckBoxGroup/HBox/CheckBox/Rectangle) 
Rectangle#91 (MotionScreenView/MotionControlPanel/Panel/VBox/VerticalCheckBoxGroup/HBox/CheckBox/Rectangle) 
Path#92 (MotionScreenView/MotionControlPanel/Panel/VBox/VerticalCheckBoxGroup/HBox/CheckBox/FontAwesomeNode)
Path#94 (MotionScreenView/MotionControlPanel/Panel/VBox/VerticalCheckBoxGroup/HBox/CheckBox/Path)   
Text#95 (MotionScreenView/MotionControlPanel/Panel/VBox/VerticalCheckBoxGroup/HBox/CheckBox/Path/Text) 
Rectangle#96 (MotionScreenView/MotionControlPanel/Panel/VBox/VerticalCheckBoxGroup/HBox/CheckBox/Rectangle) 
Rectangle#97 (MotionScreenView/MotionControlPanel/Panel/VBox/VerticalCheckBoxGroup/HBox/CheckBox/Rectangle) 
Path#98 (MotionScreenView/MotionControlPanel/Panel/VBox/VerticalCheckBoxGroup/HBox/CheckBox/FontAwesomeNode) 
Path#100 (MotionScreenView/MotionControlPanel/Panel/VBox/VerticalCheckBoxGroup/HBox/CheckBox/Path) 
Text#101 (MotionScreenView/MotionControlPanel/Panel/VBox/VerticalCheckBoxGroup/HBox/CheckBox/Path/Text)  
Rectangle#102 (MotionScreenView/MotionControlPanel/Panel/VBox/VerticalCheckBoxGroup/HBox/CheckBox/Rectangle) 
Rectangle#103 (MotionScreenView/MotionControlPanel/Panel/VBox/VerticalCheckBoxGroup/HBox/CheckBox/Rectangle) 
Path#104 (MotionScreenView/MotionControlPanel/Panel/VBox/VerticalCheckBoxGroup/HBox/CheckBox/FontAwesomeNode) 
Path#106 (MotionScreenView/MotionControlPanel/Panel/VBox/VerticalCheckBoxGroup/HBox/CheckBox/Path)  
Text#107 (MotionScreenView/MotionControlPanel/Panel/VBox/VerticalCheckBoxGroup/HBox/CheckBox/Path/Text) 
Rectangle#108 (MotionScreenView/MotionControlPanel/Panel/VBox/VerticalCheckBoxGroup/HBox/CheckBox/Rectangle) 
Circle#109 (MotionScreenView/ResetAllButton/Circle) 
Circle#110 (MotionScreenView/ResetAllButton/Circle/Circle) 
Path#111 (MotionScreenView/ResetAllButton/Path) 
Image#112 (MotionScreenView/ItemNode/Image) 
Rectangle#113 (MotionScreenView/ItemNode/Rectangle)  
Text#114 (MotionScreenView/ItemNode/Text) 
Image#115 (MotionScreenView/ItemNode/Image)  
Rectangle#116 (MotionScreenView/ItemNode/Rectangle) 
Text#117 (MotionScreenView/ItemNode/Text) 
Image#118 (MotionScreenView/ItemNode/Image)  
Rectangle#119 (MotionScreenView/ItemNode/Rectangle) 
Text#120 (MotionScreenView/ItemNode/Text)  
Image#121 (MotionScreenView/ItemNode/Image) 
Rectangle#122 (MotionScreenView/ItemNode/Rectangle) 
Text#123 (MotionScreenView/ItemNode/Text) 
Image#124 (MotionScreenView/ItemNode/Image)  
Rectangle#125 (MotionScreenView/ItemNode/Rectangle) 
Text#126 (MotionScreenView/ItemNode/Text)  
Image#127 (MotionScreenView/ItemNode/Image) 
Rectangle#128 (MotionScreenView/ItemNode/Rectangle) 
Text#129 (MotionScreenView/ItemNode/Text) 
Image#130 (MotionScreenView/ItemNode/Image) 
Rectangle#131 (MotionScreenView/ItemNode/Rectangle)  
Text#132 (MotionScreenView/ItemNode/Text) 
Rectangle#143 (NavigationBar/Rectangle)  
Image#144 (NavigationBar/PhetButton/Image) 
Path#145 (NavigationBar/PhetButton/FontAwesomeNode) 
Text#150 (NavigationBar/Text) 

@englercj
Copy link

A BaseTexture represents a single source of something we draw. Usually this is an Image or Canvas object. A Texture is the application of a frame to a BaseTexture, it describes what rectangle of the base's source is actually drawn. The Texture is then used in a Sprite which defined the spatial properties of something we draw.

When you are drawing many sprites, we can batch them into a single gl draw call if their base texture is the same. That is, if all the sprites are drawn from the same "image" (like a sprite sheet) they are batched. Otherwise you basically get one draw call per-sprite because you will get one draw call per baseTexture.

Graphics are drawn with the stencil buffer and therefore require us to draw the buffer to the viewport individually (but you can use cacheAsBitmap) to make it an image draw call instead of a stencil buffer operation.

Does that make sense?

@samreid
Copy link
Member Author

samreid commented Mar 4, 2015

Status update from PixiBlock.js

/**
 * Handles a visual Pixi layer of drawables.
 *
 * STATUS REPORT
 * March 4 2015
 * PixiBlock is in bad condition--it is a collection of prototypes and not ready for production code.  The best way to
 * see the status of PixiBlock is to launch:
 * http://localhost/forces-and-motion-basics/forces-and-motion-basics_en.html?rootRenderer=pixi&screens=1
 *
 * Completed in this version of forces and motion: basics, with pixi
 * 1. Images are rendering, and are in the correct position
 *
 * Issues in this version of forces and motion: basics, with pixi
 * 1. Z-ordering is incorrect.  When dragging a (hidden) puller, the z-order of many things changes.
 * 2. When dropping a puller, there is an exception that crashes the simulation
 * 3. Resizing is not handled
 * 4. Context loss is not handled
 * 5. Path is not generalized--just doing moveTo/lineTo
 * 6. Getting 30 fps when dragging a puller on iPad3. (Also note that transparency, curves, etc will slow this down further)
 *      Though we could take steps to optimize for pixi, such as batching images together
 *
 * @author Sam Reid
 * @author Jonathan Olson <jonathan.olson@colorado.edu>
 */

@samreid
Copy link
Member Author

samreid commented May 27, 2015

No plans to resume this investigation at the moment. If/when we do, @jonathanolson should be more heavily involved.

@samreid samreid removed their assignment May 27, 2015
zepumph pushed a commit that referenced this issue Dec 15, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants