[p5.strands]: Allow equivalent of p5 transforms

[davepagurek]

Increasing access

In normal p5, users have access to functions like rotate(), scale(), etc. In p5.strands, if one wants to position particles, currently there is nothing that looks like that, so to do a rotation, one has to do a lot of manual trig. This is a fairly large step in the learning curve that we could possibly smooth out.

Most appropriate sub-area of p5.js?

• Accessibility
• Color
• Core/Environment/Rendering
• Data
• DOM
• Events
• Image
• IO
• Math
• Typography
• Utilities
• WebGL
• Build process
• Unit testing
• Internationalization
• Friendly errors
• Other (specify if possible)

Feature enhancement details

I was working on this p5.strands sketch https://openprocessing.org/sketch/2664809 and found myself essentially manually applying a rotation matrix to points. This is fairly hard to do and I wouldn't expect this to be something most users are comfortable with. Maybe there's something we can do there?

My initial thought: we're in the process of making p5.Matrix public and making regular p5 functions have strands equivalents. Maybe we can make an equivalent of p5.Matrix that works in strands?

e.g.:

Before	After
const curveCos = cos(theta) const curveSin = sin(theta) // ... inputs.position += [ curveX * curveCos - curveY * curveSin, curveX * curveSin + curveY * curveCos, 0 ]	const rotation = createTransformMatrix().rotateZ(theta) inputs.position += rotation * [curveX, curveY, 0]

[perminder-17]

This would be a great enhancement, manually applying a rotation matrix to points would really be hard as I can see and I also wouldn't expect this to be something most users are comfortable with as well. Thanks for bringing this up @davepagurek :)

[GregStanton]

Hi @davepagurek and @perminder-17! This is great to see, and @davepagurek, thanks for mentioning this use case! I've been working on a new Transform class and coincidentally made a relevant demo yesterday, before I saw this issue. It addresses the exact case you mentioned (not p5.strands specifically, but applying transforms to individual points and avoiding manual trig). Great minds think alike! 😄

Actually, I've been working on a proposal for several related classes:

• Transform (new)
• Vector (enhanced)
• Matrix (new)
• Tensor (speculative)

I suppose I shouldn't hype it up too much, but I feel I've cooked up something special. For each class, I've written up motivation and use cases, a core feature set (with potential roadmap items), an API, and an implementation architecture. I'm entering the phase where I'm writing up all my ideas and finishing the details for a set of public proposals.

I was originally thinking of making these sub-issues of #7754, but to keep things concise and focused, I might start a new umbrella issue since the proposals are all related. I would link to all previous discussions, including this one. How does that sound?

P.S. It looks like you're using operator overloading in your demo code snippet? Is that on purpose? Don't get me wrong. I wish we had operator overloading in JavaScript, but I don't think we have that available unless we do something like paper.js has done. My operating assumption has been that this is probably off the table for p5, based on past discussions, but please let me know if that's changed!

[davepagurek]

We actually do have that in p5.strands, as a way to bridge some of the features of GLSL to our shaders-in-js API. Like paper.js, it rewrites operators between vectors into .mult(...) and similar methods, which can also be written out explicitly. Currently it's just there though and not in general.

[davepagurek]

Under the hood in strands, variables that look like numbers also aren't actual js numbers, they're objects that keep track of operations applied to them so that we can output matching structures in GLSL. The main benefit of operator overloading is to make those FEEL like they're just numbers so users aren't aware that anything different is happening, unlike some other GLSL builders we looked at. This happens to also allow us to do multiplications on matrices and vectors if we want in strands, but it's still up in the air whether or not we think it makes sense to actively encourage this, or just mention it's a thing you can do so people coming from GLSL don't feel like it's way more verbose in strands.

[GregStanton]

Oh, wow. Super interesting. Thanks @davepagurek! I'm glad I asked. (And I should've known you knew what you were doing with the overloading 😅.)

As I mentioned, I'm planning to propose a dedicated Transform class, as well as general math classes including Vector and Matrix that are decoupled from Transform. (The implementations will be different anyway, due to performance requirements, and the APIs will be totally different.)

Regarding transforms, my sense is that it'd be best to avoid operator overloading in p5.strands (except possibly for GLSL users who really know what they're doing, although likely not even for them). I think my reasoning applies in this context, but I haven't learned a lot about p5.strands yet, so I'll explain my thinking.

I'll comment briefly about overloads for general matrix operations, within p5.strands, at the end. I'd be curious to hear your thoughts about any of this.

Transform API

In the design I'm working on, transform features can be used without any knowledge of matrices or matrix multiplication. I'm treating matrices as essentially an implementation detail. Here's a sample of the current API that includes the general transform features (we'd also have built-in transforms like rotate() and some other things):

applyTransform(t)
applyToTransform(t)
applyToPoint(p)
applyToDirection(d)
applyToNormal(n)

Notes:

• We could rename the standalone applyMatrix() (alias + deprecation + 3.0 removal). I'll elaborate in the proposal.
• I'm hoping you can guess the difference between applyTransform() and applyToTransform(), based on the names.
• The applyToNormal() feature would obviate the need for a feature like getNormalMatrix() and increase cohesion.

Abstracting away the matrix operations has a few major benefits in the context of the current conversation:

1. Users don't need to know anything about matrices; they can use knowledge of the standalone transforms instead.
2. Users don't need to know how to handle points and directions in homogeneous coordinates (or risk accidental errors).
3. Users don't need to know that normals are handled by an inverse-transpose matrix when non-uniform scaling is applied.
4. Users won't need to account for differing matrix conventions, which is good because these vary widely across contexts.
5. Users will write readable, self-documenting code by default.

Since users won't need to know the transforms are implemented with matrices, they may be less likely to think to themselves, "Why is this more verbose than in GLSL?"

Confession

When I first made the Transform demo, and when I applied the same transforms using standalone features, I got strange results. It took me a while to figure out what was going on, despite being well versed in linear algebra, including graphics transformations.

My mistake was that I had assumed p5 applies transforms by pre-multiplication (as in math), and that p5 applies transforms in the order that they're specified. I only figured out what was going on when I looked carefully at the glMatrix source code. I learned that, for reasons I won't get into here, p5 and glMatrix accumulate transforms using post-multiplication by default, but they apply transforms to points using pre-multiplication.

A consequence of the conventions used by p5 is that transforms are actually applied in the reverse order to how they're specified by users. That matters because different orders can produce different results. (Unfortunately, I don't think we can do anything about it, since the native drawing context in the 2D canvas and DOMMatrix also use these conventions. The existing p5 API for standalone transforms is similar to the native APIs, so having different behavior from those could cause quite a bit of confusion.)

Abstracting all of this away definitely seems like the way to go. We just need to be sure to document the fact that the order of transforms is reversed. So, if a user wants to apply a rotation and then a translation, they need to call methods in the opposite order: translate() and then rotate().

Overloading for pure math operations

If JS ever gets operator overloading, or if we want to implement overloads ourselves (since it's already possible in p5.strands), it'd be good to consider taking advantage of them in the general math classes like Vector and Matrix. This might be really cool! Overloading for vector and matrix math is something I've always wished were possible in JS.

However, for p5.strands, I suppose we'd need a matrix backend that runs on the GPU? I think the most viable backends for us are currently designed to run on the CPU. This is assuming we might want a tensor class. I actually came up with a plan for supporting a hardware-accelerated backend in the future, but it seemed like too much of a complication for now.