get rid of warning on 'npm run bench' #998
Conversation
|
Do we want to change it to return false if we couldn't bench something? |
|
As in? |
|
like if showdown or robotskirt fails? (like they are now) or maybe set a threshold and if the marked bench is higher than that it should fail? that way we could actually use it in our automated testing and flag if a pr makes it too much slower.. |
|
Benching against other parsers is optional so I wouldn't exit 1. |
|
It seems like there is not really any point in benching then. |
|
From the product/marketing perspective: We claim to be built for speed, which raises the question, compared to what? I think Apple does this pretty well. When they release a new version of Safari, for example, they do comparitive analysis against the previous version and the other major browsers. @worker8 posted an interesting link over on #963 that does some comparitive analysis. If we don't bench, can we really make the claim honestly? From the developer perspective I think what I would like to see as a consumer of Marked would be the bench results for the supported parsers - and if one fails or times out a message saying something like: This parser failed. If you want to use this parser, please submit a GitHub issue. Maybe include the URL to the issues page. Re threshold: The issue there is device dependence, I think. @Feder1co5oave ran his benches and they were all 10 seconds longer than mine. I can't think of an elegant way to deal with that to choose a quality threshold. For 8fold Component - at least when I was still doing benchmarking - I used 3ms...no matter the scenario under test. I do appreciate the idea of a threshold because, like @UziTech said, without a target, why bench...just not sure if that is a separate issue compared to the benches throwing the error or not. My throwing copper moment on this for now. |
This fixes #977 (comment)