Performance Issue with Large Flows

[ColtonMcInroy]

There appears to be a major performance issues once a flow becomes larger.
The following image I actually sent to @petersirka already in private chat, but now I think it is best to be covered here.

This image shows firefox profile of about 8 seconds. During this time I had a flow with about 800 nodes and just dragged to move around within the flow. Every time I move the flow, it appears to hang while it performs lots of Element.querySelectorAll calls. This becomes especially painful if there are "Print JSON" components displaying lots of information.
Once the flow becomes large, this type of behaviour becomes a huge problem. If you drag and move, then try to do it again, sometimes the pointer reference hasn't been updated correctly yet, so it will actually start dragging components you have not selected. Attempting to open a components settings becomes very slow, and the "Code" component editor in the settings sometimes has issues initiating due to this performance problem.

I have not done dived into the TotalJS/Flow code too much yet, but based on the call tree it would appear to me that something in the repositioning process is searching the DOM tree rather than using references.

[ColtonMcInroy]

Here is a flow that shows the issue. It is encrypted using the same password I sent you for the other issue.
This flow is actually a "Module" for part of a larger application, and will not work properly without the other flows. That shouldn't matter for this issue though, so you can ignore any errors.
Schema Builder.zip

[petersirka]

@ColtonMcInroy I tested your example and it worked perfectly. No performance issues when rendering, but I'm using a Macbook Air M1. I found a problem with the Teleport component and your Function component. Just update the source-code by adding 2 lines of code: https://github.com/totaljs/flowstreamcomponents/blob/main/components/teleport.html#L161

[ColtonMcInroy]

On three different systems this flow has been tested and experiences the same problem. I myself have tested both Firefox and Brave (Chrome based) on two different linux computers (6th gen and 11th gen Intel i7) with the same issue and another person tried it on their computer with the same issues.
I am using a mouse not a touchpad, but that shouldn't make much of a difference.

Scrolling shows the issue, especially scrolling fast up/down or clicking and dragging. Scrolling up/down fast results in part of the screen going completely grey (Dark mode theme) without a grid and after a second or two the redraw finishes and I can see the flow properly. If I drag to scroll, let go, and then click to drag again, often the pointer reference and redraw has not completed, so the pointer actually grabs things not where I have clicked to drag.
Opening any components settings takes a while. If the component has an editor like the Code/Multiplexer component sometimes it does not render correctly.
Dragging components around is also very slow to initiate. At points clicking on a component and dragging it to move it's position takes 2+ seconds to even start moving, then it jumps to the cursor position and I can drag it around normally.
Selecting a component also appears to have the same effect, taking sometimes 2+ seconds before the component that was clicked on becomes selected.

Thanks for the info about Teleport/Function components. I updated the Teleport component, and made adjustments to my Function component as per recommendation.

[ColtonMcInroy]

Also, when actually working within the code editor of the Code component, typing performance is effected as well. At times the cursor hangs, then all of a sudden outputs the missing typed characters. Same thing with trying to move the cursor around by clicking or but pressing arrows.

[petersirka]

I will test it on various devices on Tuesday/Wednesday.

[petersirka]

@ColtonMcInroy I'm analysing your Flows. You uploaded two files and two issues, which I missed. I can confirm the poor performance. I will try to improve performance in your case, but you have a very specific case. I will analyse the Flow designer/editor more deeply this week.

[ColtonMcInroy]

Hey @petersirka I also noticed another issue now. I stopped TotalJS Flow and then restarted it and it takes forever for all the flows to start up. It appears that the flows get initiated in the order which they were added to TotalJS Flow and it took probably more than 30 minutes to start all the flows. I started TotalJS Flow waited like 5 minutes and I could see that 2 flows started, and the rest seemed to be doing nothing. I can tell if flows start because one of the main components in the flows when initiated outputs some log information.
After about 5 or so minutes, I thought that perhaps I had done something wrong in the flow and messed it up (which shouldn't be the case considering it was running). So I tried clearing out memory and stuff to ensure nothing was slowing it down on the system. I then started the flow again and still the same thing happens.
I decided to just wait and give it some more time. After about 10+ minutes and still seeing that only the 2 flows got started. Finally I left for about 1-2 hours to enjoy some of the day and when I got back, I see that all of the flows are up and running.
I am not sure how to go about debugging this. I tried the standard DEBUG=* before starting TotalJS Flow but that appears to do nothing.
If there anything I can do on my side to debug this. I am not sure if this should be considered a separate issue or part of this one.

[petersirka]

Hey @petersirka I also noticed another issue now. I stopped TotalJS Flow and then restarted it and it takes forever for all the flows to start up. It appears that the flows get initiated in the order which they were added to TotalJS Flow and it took probably more than 30 minutes to start all the flows. I started TotalJS Flow waited like 5 minutes and I could see that 2 flows started, and the rest seemed to be doing nothing. I can tell if flows start because one of the main components in the flows when initiated outputs some log information. After about 5 or so minutes, I thought that perhaps I had done something wrong in the flow and messed it up (which shouldn't be the case considering it was running). So I tried clearing out memory and stuff to ensure nothing was slowing it down on the system. I then started the flow again and still the same thing happens. I decided to just wait and give it some more time. After about 10+ minutes and still seeing that only the 2 flows got started. Finally I left for about 1-2 hours to enjoy some of the day and when I got back, I see that all of the flows are up and running. I am not sure how to go about debugging this. I tried the standard DEBUG=* before starting TotalJS Flow but that appears to do nothing. If there anything I can do on my side to debug this. I am not sure if this should be considered a separate issue or part of this one.

@ColtonMcInroy I know about this bug and it was fixed in the Total.js framework. Please update the Total.js framework via NPM or manually.

• https://github.com/totaljs/framework4/blob/master/changelog.txt#L60

[ColtonMcInroy]

I am currently using version is 0.0.87. According to the link you provided the fix was applied on version 0.0.85 so I should already be using a fixed version.

[petersirka]

@ColtonMcInroy and does the problem still exist?

[ColtonMcInroy]

Affirmative

[petersirka]

@ColtonMcInroy please check if the Total.js version is correct in the workers. With the exception of you, we don't have any loading issues.

[petersirka]

Maybe we can prepare a special Flow version with debugging tools. Then we will see where it stops.

[petersirka]

@ColtonMcInroy I have improved performance in the 'j-Flow' component. You can test it by updating the source-code or a bundle from GitHub.

[ColtonMcInroy]

Thanks, I will take a look when I have a moment and let you know

[ColtonMcInroy]

I did a git pull, updated total4, and restarted flow today. So far the UI seems considerably better. Checking profiling in FF shows that the massive lag caused by Document.querySelectorAll no longer appears to be occurring and the UI is much more responsive.