Incorrect timing

[yisar]

https://github.com/yisar/js-framework-benchmark/tree/master/frameworks/keyed/fre

When I run the benchmark of fre, the time of select rows exceeds 1s. There must be a problem here, because when I operate in chrome, select rows only takes 20ms.
My chrome:

The benchmark:

There must be an error. It can't be so slow unless it's blocked.

In addition, fre is a framework with Time slicing. It seems that the current benchmark is not friendly, maybe.

frejs/fre#271 (comment)
#498 (comment)

Can you take a look?

[lxsmnsyc]

20ms on normal run (with no CPU throttle) is slow (in comparison, Solid and compostate-jsx only takes <1ms). Most of the VDOM libraries fail on this test since it requires rebuilding the entire VDOM of the table rather than updating the selected row.

[yisar]

@lxsmnsyc You're right. In this case, the fine-grained update of solidjs with in-place mutation can be completed in 1ms, but I mean, whether it's 20ms or 1ms, humans can't perceive it, and chasing 1ms is of little significance.

Fre also has the defect of VDOM, that is, it needs to traverse the whole list, which is a problem of all VDOM frameworks, because the mental model of immediate mode is like this.

[lxsmnsyc]

whether it's 20ms or 1ms, humans can't perceive it

Well the thing is, the benchmarks are for stress testing and not for normal user interaction. Almost every framework/libraries in the benchmarks have no noticeable differences in performance, well at least on human perception.

IMO it would better if you can compare the benchmarks against similar libraries (for instance, Preact and React).

Another thing is that the select row test has been in many discussions before. IIRC the final decision was to run the test on x16 throttling. Otherwise, if it is indeed slower than expected, then let's wait for resolution.

[ryansolid]

@yisar are you running on M1 apple chips? They have an issue with applied slowdowns when running in the chrome driver. I need to turn off slowdown on my main development computer since it is so absurdly long. Do other implementations have the same issue on your computer?

[yisar]

@ryansolid No, other frameworks work well.
I'm not sure what I missed. It's also possible that the test program deliberately magnified fre's defects.
As long as the reason can be found, there is a way to optimize the test case.

[yisar]

IIRC the final decision was to run the test on x16 throttling.

It is possible to amplify fre's defects here because fre does not use memo technology.

[lxsmnsyc]

I suggest that you look into the diff/patch implementation of fre

[ryansolid]

Ok, well 20ms with no slowdown I think is still indicator of something. Can try to turn off slowdowns to confirm that. I comment out https://github.com/krausest/js-framework-benchmark/blob/master/webdriver-ts/src/benchmarks.ts#L92 and rebuild to run with no throttling.

[yisar]

@ryansolid You're right. It's really caused by slowdown. If so, I must use memo technology.

[yisar]

@ryansolid @lxsmnsyc

You're right. It's true that the slowdown magnifies fre's defects, but even if there is no slowdown, fre's update is still a bit slow... Of course, Time slicing itself will slow down as a whole, but at least the current result is expected.

[lxsmnsyc]

It seems that the partial update is slower than React, which usually indicates that the patch algo is slower. If you can optimize that part, I'm sure that the rest of patch-related tests would perform well.

[yisar]

@lxsmnsyc Yes, yes, this part can be optimized. It needs to diff props in the reconcile stage. I didn't do this now 😭 :

[krausest]

I suggest you create a PR (we can still decide when to merge this PR) and then we can take a look at it together. The implementation on https://github.com/yisar/js-framework-benchmark/tree/master/frameworks/keyed/fre seems to have no remove button and thus my build process fails.

[yisar]

@krausest pr is here #933