First cut for Miso (GHCJS) benchmark #223
Conversation
Benchmark is working, but is not hooked-up to the infra properly.
|
Could you please help me debug the following error: When I'm appending 1,000 rows after creating 10,000 rows manually (i.e. outside the webdriver benchmark) it seems to be working. However, this is the only benchmark that is consistently failing when run with webdriver. I have added the compiled JS files in |
|
Author here, there needs to be some updates to this PR before I'd say it should be cleared for merge. Also, I'm not sure how @krausest feels about adding cc @krausest |
|
I feel really excited that Haskell libraries are getting more diffussion, although I've looked at the source code of the benchmark and it uses the Also, I think it is better to provide |
Create 1,000 rows - Miso 1.9x slower than Elm Replace 1,000 rows - Miso 1.4x FASTER than Elm Update every 10th row - Miso 3.7x slower than Elm Select row - Miso 4.8x slower than Elm Swap row - Miso 2.9x slower than Elm Remove row - Miso 1.9x slower than Elm Create 10,000 rows - Miso 1.5x slower than Elm Append rows - Miso 2.8x slower than Elm Clear 10,000 rows - Miso at-par with Elm Startup time - Miso 2.6x slower than Elm
This reverts commit b2b1566.
Create 1,000 rows - Miso 1.9x slower than Elm (as opposed to 1.9x slower) Create 10,000 rows - Miso 1.6x slower than Elm (as opposed to 1.5x slower) Append rows - Miso 2.8x slower than Elm (as opposed to 2.8x slower)
This reverts commit dd08f0b.
|
I'm pretty concerned with adding more prerequisites besides java (and scala, wich is already very heavy, but much lighter than installing the haskell tool stack which installed ~ 4GB in my home directory). I could imagine adding the compiled javascript files for languages that have such a heavy compilation process and make the build in such cases optional (something like a build-disabled-by-default script task in package.json). Would you be willing to create pull requests when major updates happen to miso or ghcjs? I think I'd have to create a deprecation policy for "binary-only" frameworks in case no updates happen - because I'm afraid I won't be able to maintain those non-javascript-frameworks for non trivial updates. Could you please rename the miso-ghcjs folder to include a version number? @saurabhnanda |
|
@krausest @saurabhnanda, I'd rather not commit these benchmarks until I've finished implementing the child keys patch, since they won't be accurate, and will give people a faulty understanding of this framework's abilities. Can we please close this PR? I think writing the benchmark code would be better left up to the author. @saurabhnanda did not ask for my permission or coordinate with me on this. |
don't feel offended. there are plenty of impls here not written by framework authors but by users without any blessings. if the impl is sub-optimal, then submit a faster or more idiomatic impl. if the framework is outdated, then submit an update. fyi, no matter how fast the impl ends up being, if it requires parsing 1.5MB of GHCJS first, it's not going to do well here.
if only the author can write fast code in the framework, won't that give average users a faulty expectation of their code's performance? ;p...generally this is true, though. |
|
Benchmark code is typically contrived to be optimal for a specific set of assumptions. Also, creating optimal Haskell is not always trivial. But very basic things are missing or lacking entirely in this example. You guys do what you want. I'll just come back to it later |
sure, i wasn't suggesting that this should be merged. |
@krausest wouldn't this be the case for regular JS frameworks as well? Aren't the versions of the library and all its dependencies locked? Isn't there a high chance that the build would fail if newer versions of dependencies are automatically fetched (any lib could introduce a backward incompatible change!)
Sure, I'll do that.
No, it's working now. The test was timing-out earlier, but @dmjio made some suggestions which improved the spreed and its working for me now.
I have a different philosophy for benchmarking, ie user, not authors should write them. And when I'm writing benchmarking code I tend to write the most obvious thing first - right after reading a tutorial. Unless it's doing something really stupid (like using widely known performance killers, eg At the end, it's up to you @krausest Have other frameworks submitted highly tweaked code? What it your preference as the maintainer - real life vs tweaked? @krausest I can also understand and appreciate how @dmjio is feeling about this, given that he's putting a lot of hard work into a brand new framework, which obviously has a lot of things in WIP mode. Given the marketing impact these benchmarking numbers have, it definitely runs the danger of putting off early adopters. May I request you to have the ability to add notes along with each framework's results, which allow the author to explain the slowdown and link to specific Github issues addressing them. This, I believe, balances the needs of users and concerns of authors. As a user, I'd like to get some idea about the performance of a framework today (with some caveats and notes) rather than being left in the dark. Let me make an informed decision. |
First of all, this isn't an accurate benchmark because the keys patch isn't finished, yet you've incorrectly named this Furthermore, the code you've submitted is performing updates to the model in Lastly, you're not collaborative. I've asked you to join the slack and talk with the others, but you refuse, and now you're trying to submit faulty benchmarks against my better wishes. Not a great start. |
Right. I'll change that.
that's the most natural way that occurred to me after staring long and hard at the Random API and Vector API in Haskell. I tried your suggestion of using a non-IO random generator, but it has equivalent performance. If you'd prefer using the non IO version, then I already have the code available - vacationlabs@dd08f0b Let me know and I'll change it.
I concede on not using slack/gitter (most of the chat doesn't get indexed by Google, and I strongly feel a lot of these conversations should get indexed and be available for the benefit of others). But I wouldn't say this is a shoot and scoot operation that I'm attempting here, don't you think - dmjio/miso#225 ? Anyway, I'd suggest just agreeing to disagree on whether submitting benchmarks need the blessing of the framework author. I'm still available in good faith to make the benchmark more representative if you still feel if it's out of whack. Heck we can even call it miso-non-official if @krausest is fine with that. |
Use the FFI version from
You're opinion on wether or not slack gets indexed isn't an excuse to not collaborate, we also have #haskell-miso IRC.
I'd gladly work with anyone who knew no Haskell, but was willing to collaborate. As opposed to someone who knows some Haskell but won't collaborate.
That's exactly what this is. You really aren't sure about what you're doing, and you're just trying to run this through. |
I'll respond to all this with a cooler head in some time (and probably directly to @dmjio). This line & tone of conversation is not adding value to anyone. Wrt whether the benchmark should be merged, or not, I'll leave it up to @krausest I'm available to make changes that seem appropriate.
Btw, this doesn't seem appropriate to me, because this is an optimisation that GHCJS should be doing directly, IMO. If these kind of tweaks/hacks are required to speed up Miso/GHCJS for the benchmark then I'd suggest having two versions of the benchmark code |
Neither is your lack of willingness to collaborate with the others.
@saurabhnanda, regardless of the merge I'll fork and host my own as the official. I've tried, but it's obvious we can't collaborate. |
I'm sorry, but I disagree. Not having a conversation, specifically on slack, is equivalent to not being collaborative? Responding to every comment on the relevant Github issue, and trying all suggestions and posting all interim results - what would you call that? @dmjio I'm rooting for Miso, btw. I just feel strongly about benchmarks being made available ASAP and being made better with every release of the library. Over and out from my side on this particular side of the conversation. Rest I'll handle privately with @dmjio Again, I'm still available to work on this PR to get it merged upstream. It would be unfortunate if this (or some other version) didn't get merged because of a meta-discussion. |
Doesn’t seem to be making anything better: Create 1,000 rows - Miso 1.9x slower than Elm Replace 1,000 rows - Miso 1.2x FASTER than Elm Update every 10th row - Miso 3x slower than Elm Select row - Miso 5.7x slower than Elm Swap row - Miso 3.3x slower than Elm Remove row - Miso 1.4x slower than Elm Create 10,000 rows - Miso 1.6x slower than Elm Append rows - Miso 2.9x slower than Elm Clear 10,000 rows - Miso at-par with Elm Startup time - Miso 2.7x slower than Elm
This reverts commit 5717509.
|
@krausest Hopefully, my final set of changes:
Let me know if something needs to be changed to make this a "non buildable" benchmark. I hope this gets merged. We need more visibility for Haskell JS frameworks. Thanks @krausest for maintaining these benchmarks. And thanks to @dmjio for the Miso framework. |
|
You guys are giving me a bad headache ;-) @saurabhnanda Thanks for your effort! I'm not sure how I feel about miso. Seems like it was either too early for the framework or too difficult for its users to create a reasonable implementation. I implemented a few benchmarks for javascript frameworks and with one exception their frameworks authors didn't make a riot (pun intended) and I guess that's part of what a developer should be allowed to ask from a framework (small improvements are fine, but you shouldn't be far off - that's the "frame" in the framework). I'm still not sure how I'm going to handle frameworks that have such big prerequisites. I'm currently pretty happy that I have a circleci build that compiles and runs all frameworks. With a dependency like ghcjs things would be much harder. And it would harm acceptance for developers trying to build the benchmarks on their own machine - already now people are complaining about the bloat (and they didn't get those 5.8 GB in .ghcjs and .stack yet). |
Not trying to hijack the thread, but I'll leave my two cents here, since this concern is relevant to the benchmark I'm working on: reflex-dom (also Haskell by way of ghcjs). There's also at least one purescript benchmark on the way, so I'm afraid we already opened pandora's box. How about requiring that We could always take this one step further and allow choosing exactly what languages are to be installed/built/tested (default: javascript/typescript), the same way one can choose for which frameworks benchmarks are run. There's also the nuclear option: have a separate repo for Haskell frameworks that is then included into this one as a submodule. That repo could internally have its own rules (cabal/stack/nix) and its own CI, and somehow export compiled javascript (e.g. 'binary' releases, continuous delivery branch, magical dwarves), that is compatible with the main one. Usually I prefer to keep things that need to work together in the same repo, so they all go in the same CI and don't break each other, but y'know, tradeoffs. |
|
To follow on from @alexteves, I have put together benchmarks for Pux and Halogen, two of the three primary frameworks used in PureScript. There is growing interest in benchmarks for these PureScript frameworks, so I've created implementations based on your repository (thanks for the great work!). They're a little different from the Haskell ones above because PureScript is much closer to Elm. In the Haskell, Elm, and PureScript cases, each one is a functional language compiled to JavaScript, and perhaps it does make sense to have that be a subset of these benchmarks rather than mixed in. Of course, that's your decision @krausest, but I thought I'd let you know about other functional benchmarks that have been put together. Cheers, |
|
Regarding the whole idiomatic vs performant implementation: do we really have to choose? To me this seems more significant the farther away from vanillajs we go (Sufficiently Smart Compiler and all that jazz), so it might only really affect a handful of frameworks. |
personally, i'd also like there to be both perf-tuned and idiomatic impls (if they differ), though not all in the same table. a third table that compares any frameworks with both variants would help cut down the clutter. i'm not terribly thrilled that there are already a whole bunch of React impls each with a different state management lib - each slower than the bare React impl. i've had the benchmark vs stress-test discussion in other forums before. displaying 10k rows, appending 1k rows and clearing 10k rows certainly falls into the stress test category that would be suitable for perf-tuned code, because this is an extreme case you would expect users to optimize for if for some reason it was actually necessary. the other metrics are more "real world" and would work well for idiomatic impls. most complex/heavy websites are < 3,500 dom nodes in total. Gmail's [pretty bloated] inbox has 3,874. try it out in the console on your favorite bloated property: by having a single table, you're forced to choose if you want fast-but-less-clear code or most-obvious-but-slow-10k code. most authors choose the former. |
Sorry about the noise. Didn't expect that to happen.
|
|
On perf vs idiomatic, as a user I'd like to see the difference in performance vs idiomatic and have the code available to me to understand how to do performance tuning specific to that framework. |
|
@SayLU PureScript seems to be a "bit" more lightweight with ~64 MB diskspace and npm based installation. Thus I'm looking forward to getting a pull request. |
|
The question of maintenance depends on how close the implementations stay to javascript, not in terms of the language, but in terms of the ecosystem. Haskell seems to be particularly far off (both haskell based frameworks had spectacular dependencies), elm and probably PureScript too are close enough such that I can promise basic maintenance. |
|
Closing this PR. |
Homepage: http://haskell-miso.org
Github: https://github.com/dmjio/miso