Using a separate math library for handling p5.js's math operations

[RandomGamingDev]

Increasing Access

The current p5.js math component is heavily cluttered and isn't nearly as optimized as it could be which makes it hard to not only use if you're trying to understand and interact with the code, but also to contribute to, especially for a library designed to increase accessibility to beginners.
Plus, it's hard to integrate p5.js with other math libraries when you need any math that's more complex, for instance if someone wanted to do something that involved linear algebra, or calculus that isn't included in p5.js, like 4d scenes.
Not only would using another library mean a better documented and managed library with more features for p5.js, but it'll also be more performant, and we could choose a library that can utilize the GPU for especially heavy operations.

Most appropriate sub-area of p5.js?

• Accessibility
• Color
• Core/Environment/Rendering
• Data
• DOM
• Events
• Image
• IO
• Math
• Typography
• Utilities
• WebGL
• Build Process
• Unit Testing
• Internalization
• Friendly Errors
• Other (specify if possible)

Feature enhancement details

Instead of using p5.js's current math component, revamp it so that it uses another more powerful math library for its calculations.

[RandomGamingDev]

Maybe we could try using this or something similar: https://mathjs.org/

[nickmcintyre]

@RandomGamingDev thanks for starting the discussion.

The current p5.js math component is heavily cluttered and isn't nearly as optimized as it could be which makes it hard to not only use if you're trying to understand and interact with the code, but also to contribute to, especially for a library designed to increase accessibility to beginners.

I recently read the source code of src/math and didn't get the impression it was "cluttered". Quite the opposite, most of it's a thin wrapper around the native JavaScript Math object.

Plus, it's hard to integrate p5.js with other math libraries when you need any math that's more complex, for instance if someone wanted to do something that involved linear algebra, or calculus that isn't included in p5.js, like 4d scenes.
Not only would using another library mean a better documented and managed library with more features for p5.js, but it'll also be more performant, and we could choose a library that can utilize the GPU for especially heavy operations.

I wrote a simple sketch with Math.js and found it approachable. Skimming some issues, it doesn't look like Math.js uses hardware acceleration. They did mention stdlib's ndarray as a possible route forward. You'd have to ask @davepagurek how either approach might benefit internal calculations for WebGL.

As for users, I saw you mentioned #5210 in Discord. @golanlevin made a good suggestion to start with an addon library and examples. He also mentioned my library número. It might provide some inspiration for how to make matrices play nicely with p5.Vector. I actually used Math.js in an earlier iteration–happy to discuss that in a separate thread.

Personally, I'd study how three.js approaches the math section of their API for enhancements that could increase access. I used their Raycaster in a recent project and it's great. The implementation depends on their Ray object.

@limzykenneth, @ericnlchen, @ChihYungChang, @bsubbaraman, @albertomancia, @JazerUCSB, @tedkmburu, @perminder-17, @Obi-Engine10, @jeanetteandrews, and @GregStanton may have some thoughts here, too.

[RandomGamingDev]

I recently read the source code of src/math and didn't get the impression it was "cluttered". Quite the opposite, most of it's a thin wrapper around the native JavaScript Math object.

While I agree that most of the library's fine there are major parts that I do believe are cluttered and could use some improvement.

p5.Vector.js is hard to read and has a lot of points that could really use improvement in how it's coded, and isn't expandable or collapsible to smaller or larger vectors. This leads to annoying to deal with vectors, and a repetitive codebase with code segments like this.

Then there's also p5.Matrix.js which for some reason isn't in the math section, is even more cluttered, not very expandable since it only has options for mat3 & mat4, the documentation isn't exactly the best with errors like saying that it's for describing 4x4 matrices despite it supporting mat3, it doesn't have a lot of mathematical functions for certain operations like a functions for getting the ref (Row echelon form) & rref (Reduced row-echelon form) form of matrices, and has code using it using p5.Matrix.set() for every element in the matrix inside and outside of the library which led to more clutter in things like p5.Camera.js which is probably from the same system being used everywhere else in the library.

While I could see the system getting polished and fixed to remove this clutter, better organize everything, make everything more expandable, better document things, and add more math functions, it won't be very easy to maintain because of p5.js's size, and the fact that p5.js isn't a heavily math oriented library. I feel that the best solution, at least at the current moment, would be to replace most of the internals of p5.js's math that have been causing this, and to just add a light wrapper around a more advanced, reliable, better documented, and more powerful library, rather than reinventing the wheel.

Skimming some issues, it doesn't look like Math.js uses hardware acceleration.

No, they don't use hardware acceleration, but I'm not really sure p5.js needs hardware acceleration rn, and I thought that recommending math.js could make for a good starting point.

I wrote a simple sketch with Math.js and found it approachable

Yes, it can be decently approachable, but it doesn't integrate very well with p5.js if you're trying to do some manipulation of p5.js's internal systems while relying on another math library (especially for more complex things like 4d scenes) and is even harder for beginners to use, understand, and learn from. I think p5.js should prioritize making its math easy to use and powerful from from the very beginning instead of relying on outside libraries or plugins, especially since this would help speed up the development of p5.js, bug testing, feature additions & enhancement, as well as make it easier for people who are new to certain topics understand them better, since documentation would likely be better with a math dedicated library.

[davepagurek]

Just to chime in about the additional WebGL-specific aspects of the p5.Matrix class: In addition to everything else you all have mentioned, it needs to store its data in a format that we can quickly drop into a setUniform call without needing to convert. Based on the docs for sending a uniform matrix to a shader, this means:

• The underlying data format has to be a Float32Array
• Matrix data must be in column-major order

It looks like Math.js doesn't do typed arrays just yet? We might need to either benchmark how much of a slowdown it is to convert, or patch in typed array support (it seems like one commenter in that discussion has successfully patched in support for it for their own uses.)

In the past I've also used the glMatrix library, which is sort of on the other end of the spectrum, aiming to be an optimized library with WebGL compatibility. Another option could be to use something like that under the hood and add a p5 style API on top of it for easier use.

@nickmcintyre from making numero, maybe you have some more knowledge about its Tensorflow backend and whether that fits the format?

[RandomGamingDev]

Converting between formats for the small matrices used by p5.js's shaders shouldn't be too large of a problem altho yeah, using another library for performance could be good too if we want to go down the performance route, altho I don't feel like that's p5.js's main goal and that choosing an easier to use library would be better for contributors and users of the library.

[nickmcintyre]

@davepagurek here's a quick sketch with a simple example that creates a couple of Float32Arrays. It uses TensorFlow's linear algebra engine and WebGL backend. There's also an advanced example using dataToGPU that I don't really understand. Seemed interesting.

@RandomGamingDev just my two cents, but I think your critique would be more helpful if you identified specific problems with the codebase. Implementing a bunch of vector/matrix operations from scratch in JavaScript is a complex job. Style is subjective, but I think the contributors to p5.Vector and p5.Matrix did solid work. You might skim three.js' math folder for comparison. Their implementations have a similar feel. They also manage to do a lot with a few extra primitives that are implemented from scratch.

A few high-level questions come to mind: Are there specific performance issues that a third-party library would solve? Do people working on p5's core feel like any math is "missing"? Are there potential core features that are only feasible with a more powerful toolkit?

[GregStanton]

Thanks for tagging me @nickmcintyre.

I think it would help if we could answer the questions @nickmcintyre posed in terms of concrete user stories, or "contributor stories" in the case of the codebase. For now, I'll share my current thinking, although I'm not really sure about any of this!

Background

For some of my p5 sketches, I've written numerical integrators, I've used math.js to parse algebraic expressions, and I've used a different outside library for nonlinear regression:

• mass-spring simulation and phase plot
• interactive graphing calculator
• bacteria simulation with curve fitting

These types of sketches require a lot of foundational code (e.g. for a coordinate system that conforms to mathematical conventions, for axes, for drawing arrows, etc.). So, I recently started a p5 add-on library called Mathemagical.js. It will likely contain a lot of the computational power under discussion here.

Scope

"Any math that's more complex" (to quote @RandomGamingDev) is an immense scope. Exposing this kind of functionality to the p5 user would mean a big addition to the API, so it should have a compelling justification. Once something goes into the API, we're basically stuck with it.

Adding something the size of the math.js function reference would probably double the size of the API, at least. Most of it will be incomprehensible to beginners who are interested in creative coding, rather than math, and they may feel intimidated.

Dependency issues

Regarding additional math features, @RandomGamingDev noted that "it won't be very easy to maintain because of p5.js's size, and the fact that p5.js isn't a heavily math oriented library." The argument, as I understood it, is that p5.js should therefore trust another project to develop and maintain the math functionality.

Trusting another project with a large portion of the p5 API seems risky, especially right now. As far as I know, the project lead of math.js works on it in his spare time and there's not a larger organization backing it, in case he has to step out. More generally, WebGPU seems likely to shake up the technical computing landscape.

Wrapping a dependency using the facade pattern or adapter pattern may mitigate the risk somewhat, but we may still end up being tied to the structure of whatever dependency we add.

Internal improvements

Another option is to depend on, and expose, only a small subset of features from a library like math.js. The only obvious way to narrow it down is to consider the math p5 already uses, e.g. p5.Matrix. For that, it probably makes sense to borrow from a library that's more specifically geared toward the purposes of p5, such as glMatrix or the three.js math folder, which others have mentioned. Maybe those libraries could inspire performance improvements or refactoring. Here are a few other projects that may help if p5.Matrix needs improvement (some have already been mentioned):

• Stdlib
• ndarray from SciJS
• A BLAS implementation in JavaScript
• NumJS
• webgpu-blas
• número

Add-on library

It probably makes sense to relegate any large addition of math features to an add-on library like Mathemagical.js or número. For example, this may include numerical analysis, linear algebra, geometry, optimization, probability, and statistics. Since these features would be core to the purpose of the project, they'll attract users rather than intimidate them. Those same users can become contributors, so developing or maintaining extra math features will be less of a stretch.

Math visualizations are another consideration. It seems likely that users of a graphics library with advanced math features will want to make math visualizations. But in p5, even something as simple as axes with tick marks feels out of scope. If p5 goes beyond graphical primitives and starts providing domain-specific visualizations (math or otherwise), the scope becomes unbounded. (Composite objects like axes also work better with an object-based interface, rather than a function-based interface like p5's. In Mathemagical, we provide both types of interfaces. Matplotlib is an example of a library from the Python ecosystem that does the same thing. The complexity is forced by the use case, but p5 works without this.)

Edit: Made reply more concise, fixed a link.

[RandomGamingDev]

@RandomGamingDev just my two cents, but I think your critique would be more helpful if you identified specific problems with the codebase. Implementing a bunch of vector/matrix operations from scratch in JavaScript is a complex job. Style is subjective, but I think the contributors to p5.Vector and p5.Matrix did solid work. You might skim three.js' math folder for comparison. Their implementations have a similar feel. They also manage to do a lot with a few extra primitives that are implemented from scratch.

I did identify specific problems in #6527 (comment) and in the post itself, where I detailed the problems with the current math system. While I agree that the job is complex and that a lot of the job is done quite solidly, I don't believe that all of it's solid, or that a lot of the things added are a matter of just style.
For instance, while three.js's math libraries and p5.js's math libraries have a similar feel, there are a lot of points where p5.js could stand to be more like three.js. Let's show a few examples from just the Matrix libraries since they're the most tedious to deal with and the most apparent (a lot of other parts have these same issues too):

• p5.Matrix.js isn't in the math section, but rather the p5.js section

• Then there's also p5.Matrix.js which for some reason isn't in the math section, is even more cluttered, not very expandable since it only has options for mat3 & mat4, the documentation isn't exactly the best with errors like saying that it's for describing 4x4 matrices despite it supporting mat3

• .transpose()
  Three.js version: The transpose function here, although you could say that it looks cluttered, actually isn't that cluttered especially since it's pretty easy to take a look at the code and instantly tell that lines that look like this:

  tmp = te[ 1 ]; te[ 1 ] = te[ 4 ]; te[ 4 ] = tmp;

  just swap different numbers in the matrix, and then to tell that they're getting swapped over the diagonal, with clear naming, and a nice clean split between 2 groups of 3 lines of swaps to signify which parts of the matrices they're operating on. It also always works for the matrix it's operating on, which isn't true for p5.Matrix.js.

• p5.js version:
  For some reason this works only when doing it on another matrix and specifically flipping it and then copying it over to this matrix and not on just this matrix I think that just copying the data over and then swapping is cleaner, but shouldn't there be an option to do it for this specific matrix rather than just being forced to do mat.transpose(mat)? For some reason, there are variables allocated like this despite them being used nowhere else and including every other part of the writing to this.mat4 which don't use the variables which just directly use a.mat4[<foo>]:

  let a01, a02, a03, a12, a13, a23;

  Then there's also the major, and overarching issue that's constantly present in a lot of p5.js's code, which is repetition, where there absolutely shouldn't be repetition since a whole 23 lines which could just be replaced by adding something like:

  const aMat4 = a instanceof p5.Matrix ? a.mat4 : a;

  and then using the just the part after the else with the variable aMat4. The coding practices used within the math library and that's spreading out of it is I believe, unmaintainable, and even if it's usable for the user, it makes the experience of coding for the library a lot more tedious. Not only that, but this doesn't even work for all p5.Matrix.js matrices since, as it turns out, although p5.Matrix.js is documented to be "A class to describe a 4×4 matrix" it also supports mat3 for some reason, which this code cannot account for, makes a lot harder to track down as a bug, and if we continue using the current coding style for it, we'll also get a massive function, massive class, and extra checks. Then there's also the expandability and collapsibility issue I talked about again, where not only is this sorta code harder to transfer over to say a 5x5 matrix where it'd be completely unrealistic to repeat massive code segments like that. I get writing down the formula for efficiency and so that you can't easily read it, but there's a major problem when that formula is 23+ lines large, will be constantly repeated, and will be even larger in the future.

• This type of repetitive code is not only within p5.Matrix.js, but has also spread to things using p5.Matrix.js like this part in p5.Camera.js where set is used for, yet again, each individual number in the matrix. This is also despite there being this:

    if (inMatrix instanceof p5.Matrix) {
      this.mat4 = inMatrix.mat4;
      return this;

  and p5.Matrix.copy() which would work together just find for copying the values of a matrix as well as using a for loop or = [...toCopy].

This type of code is not only clearly apparent, and clearly full of bad practices rather than just being a certain style, but it's also spreading through p5.js and will likely continue to spread unless we clean up the code somehow, which is why I'm proposing that we do so by replacing the math library with a library with contributors that can add extra features in a clean and simple to use API for p5.js that can then be wrapped around with a custom p5.js API so that p5.js can safely swap between libraries and make the features of the underlying math library easily accessible to people who want to use them without having to add addons that quite likely won't work very well with p5.js's internals

A few high-level questions come to mind: Are there specific performance issues that a third-party library would solve? Do people working on p5's core feel like any math is "missing"? Are there potential core features that are only feasible with a more powerful toolkit?

I probably should've clarified, but by optimization I mainly meant optimization of the coding experience and workflow as well as optimization in terms of performance since I don't believe that performance is currently p5.js's main concern. However, I believe that there will be a significant performance increase from using a system that doesn't have the same problems as shown before with the p5.js codes and things like tons of large type checks and other checks. While I don't believe any truly core math is missing, I believe that we should make and effort to expand past that and provide the best experience possible instead of just the core because the core of p5.js is making things beginner friendly and easily accessible, which means that we should be making these more accessible. For instance, differently sized matrices and better more intuitive matrix types and operations (there are a lot of functions that apply only to mat4 when mat3 appears to be supported at least partially). While you could argue that these aren't that bad and that all of the core features needed to make programs are there, for which you'd be right, implementing a lot of these features without another library which again would likely have terrible compatibility with p5.js's current system (the compatibility would also have to have you have a decently deep understanding of p5.js's math classes,) would require a lot of work, work that I don't believe would be very easy or even possible for beginners.

"Any math that's more complex" (to quote @RandomGamingDev) is an immense scope. Exposing this kind of functionality to the p5 user would mean a big addition to the API, so it should have a compelling justification. Once something goes into the API, we're basically stuck with it.
Adding something the size of the math.js function reference would probably double the size of the API, at least. Most of it will be incomprehensible to beginners who are interested in creative coding, rather than math, and they may feel intimidated.

Another option is to depend on, and expose, only a small subset of features from a library like math.js. The only obvious way to narrow it down is to consider the math p5 already uses, e.g. p5.Matrix. For that, it probably makes sense to borrow from a library that's more specifically geared toward the purposes of p5, such as glMatrix or the three.js math folder, which others have mentioned. Maybe those libraries could inspire performance improvements or refactoring. Here are a few other projects that may help if p5.Matrix needs improvement (some have already been mentioned):

It probably makes sense to relegate any large addition of math features to an add-on library like Mathemagical.js or número. For example, this may include numerical analysis, linear algebra, geometry, optimization, probability, and statistics. Since these features would be core to the purpose of the project, they'll attract users rather than intimidate them. Those same users can become contributors, so developing or maintaining extra math features will be less of a stretch.

First off, I believe that you took my quote out of context since the full quote's "Plus, it's hard to integrate p5.js with other math libraries when you need any math that's more complex, for instance if someone wanted to do something that involved linear algebra, or calculus that isn't included in p5.js, like 4d scenes." Also, I wasn't saying to add every other bit of math that could possibly be needed to the p5.js's math API. To be more specific, I think that we should have a system where p5.js's current wrapper is kept (like what you're suggesting), but the internals are replaced with a better math library in a way that makes it a lot easier to switch between math libraries if needed, and that makes it a lot easier to expand by wrapping over the math library's function's with p5.js style coding. Also, even if we were to expand the API to wrap over the entire math library, I think that those who are interested in creative coding would either find it interesting, useful, encouragement to use this library over others that don't provide these things, or simply skip over them rather than being scared away from the intuitive API of p5.js by a few math functions grouped under a math part of the library. Plus, I feel that adding more math functions to the API to make it easier for people fits perfectly with p5.js's goals for ease of use, especially for beginners.

Regarding additional math features, @RandomGamingDev noted that "it won't be very easy to maintain because of p5.js's size, and the fact that p5.js isn't a heavily math oriented library." The argument, as I understood it, is that p5.js should therefore trust another project to develop and maintain the math functionality.

Yes, the argument is that p5.js should trust another project's math functionality and use it within itself.

Trusting another project with a large portion of the p5 API seems risky, especially right now. As far as I know, the project lead of math.js works on it in his spare time and there's not a larger organization backing it, in case he has to step out. More generally, WebGPU seems likely to shake up the technical computing landscape.
Wrapping a dependency using the facade pattern or adapter pattern may mitigate the risk somewhat, but we may still end up being tied to the structure of whatever dependency we add.

Wrapping it should mitigate the risk enough for it to be worthwhile considering how much easier it'll be to switch between math libraries and the potential benefits.

[RandomGamingDev]

Just another note: There are also multiple instances of every index of an array being manually indexed. One example's here where every index of the arguments is manually indexed to be written to manually indexed indexes in the matrix for 16 whole lines instead of just using a for loop like this:

for (let i = 0; i < 16; i++)
  this.mat4[i] = arguments[i];

or the even simpler approach:

this.mat4 = [...arguments];

[lindapaiste]

@RandomGamingDev has raised a lot of excellent points in this thread. IMO we should rethink the design of the Matrix object, regardless of whether we choose to implement the core mathematics ourselves or rely on a third party.

The current Matrix class is a collection of specific use-case functions. There is not a lot of thought into what a Matrix object is and what methods and properties it provides. This leads to confusing and overly-complex code upstream in classes like Camera which use Matrix instances. I would love to see a clearer interface.

Most of the methods on the Matrix class are only applicable to some Matrix objects, which is a sign of bad design. A good place to start detangling would be to separate this into three objects: a core Matrix (or SquareMatrix, since our internal uses only need to support nxn shapes) and a Matrix4x4 and Matrix3x3 class which extend the common core and add their own specific methods.

When looking at the current functions of the Matrix class, we need to ask:

• Does/should this function modify the existing matrix, or return a new matrix instance based on the current matrix? Making a clear distinction here can resolve a lot of upstream messiness.
• What data does this function use from this? (ortho and perspective do not use any and therefore should be static methods or external utility functions)
• Does this function apply to 3x3, 4x4, or both?
• Can this function be generalized to support any nxn matrix size? That is, can we convert hard-coded indices by using i inside of a loop? The diagonal method is a good candidate for this.

A note on performance regarding that last point. The hard-coded approach is faster at runtime since there's no for loops or i+2 math for the browser to execute. However it leads to a less maintainable codebase (and a larger bundle size) so that's something that we need to weigh the pros and cons of. My suggestions prioritize clean code over runtime performance.

Looking at the current Matrix class, here's how the methods break down:

4x4 only

• set [1]
• get [2]
• static identity
• transpose [7]
• invert
• determinant [3]
• mult
• scale
• rotate (and rotateX, rotateY, rotateZ)
• translate
• perspective [4]
• ortho [4]
• multiplyVec4
• multiplyPoint
• multiplyAndNormalizePoint
• multiplyDirection
• createSubMatrix3x3

3x3 only

• invert3x3
• transpose3x3
• inverseTranspose [5]
• mult3x3
• column [6]
• row [6]
• multiplyVec3

both

• copy [2]
• diagonal

[1] The entire matrix is replaced by set, so this should be a static method.
[2] get and copy both do the same thing.
[3] determinant appears to be unused. There an internal calculation of a determinant inside invert, but it doesn't use this.
[4] As previously mentioned, perspective and ortho are factory functions which create a new 4x4 matrix and don't make sense as instance methods.
[5] inverseTranspose is a method of a 3x3 matrix which takes a 4x4 matrix as an argument. It seems like it doesn't use the original 3x3 matrix at all, and should instead be a method on the 4x4 instance with no arguments. That is, instead of calling this.renderer.uNMatrix.inverseTranspose(this.renderer.uMVMatrix); it would be this.renderer.uNMatrix = this.renderer.uMVMatrix.inverseTranspose();.
[6] I would love to see column and row be generalized, as they can have a lot of uses in the internal logic of other methods.
[7] Why does transpose take an argument? It should be an operation that creates and returns the transpose of the current matrix.

[davepagurek]

Thanks for taking an interest in this @lindapaiste!

I think the first big decision to make is whether we default to having all methods mutate the current matrix or whether all methods should all return new objects by default.

Points for mutation:

• it's consistent with p5.Vector, so that could be good for newcomers
• it's potentially faster?

Points for returning new objects:

• it's less awkward for some of the methods you've identified

[1] The entire matrix is replaced by set, so this should be a static method.

I think this could still make sense as an instance method if we choose to keep matrix methods mutating the current object, similar to p5.Vector.prototype.set.

[5] inverseTranspose is a method of a 3x3 matrix which takes a 4x4 matrix as an argument. It seems like it doesn't use the original 3x3 matrix at all, and should instead be a method on the 4x4 instance with no arguments. That is, instead of calling this.renderer.uNMatrix.inverseTranspose(this.renderer.uMVMatrix); it would be this.renderer.uNMatrix = this.renderer.uMVMatrix.inverseTranspose();.

I think this is an unfortunate consequence of the above point where it tries to use a mutate-first API like p5.Vector. The problem is that it goes from a 4x4 matrix to a 3x3 matrix, which would change the type of the current matrix. Instead, it ends up getting an argument, so that it can be a mutating method of a 3x3 matrix.

If we decide to stick with an API based on mutation, maybe we can introduce a naming convention for methods that return new objects, like makeInverseTranspose(), that returns a brand new 3x3 matrix?

Alternatively, we could make all methods return new objects, similar to the native DOMMatrixReadOnly class. That might make more memory pressure when chaining together a few operations, so maaaaybe it's a bit slower? I'm not sure that that's a big enough issue for it to affect our decision though.