Really simple code benchmarking
CoffeeScript
Switch branches/tags
Nothing to show
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Failed to load latest commit information.
src
test
.gitignore
.npmignore
.travis.yml
Cakefile
README.md
package.json

README.md

Build Status

Benchit

This is a really simple benchmarking library thats return the running time of code. It supports asynchronous and synchronous code.

Usage

Synchronous code block

benchit.one ->
	...
# (will return elapsed time in milliseconds)

Asynchronous code block

benchit.one (done) ->
	...
	done() # call when finished
, (elapsed) ->
	console.log elapsed

Multiple code blocks

benchit.many
	synchronousCode: ->
		...

	asynchronousCode: (done) ->
		...
		done() # call when finished
, (name, elapsed) -> console.log "#{name} elapsed: #{elapsed}ms"

###
Output:
synchronousCode elapsed: 100ms
asynchronousCode elapsed: 200ms

(note: 2nd parameter is optional)
###

Improvements to be made

  • Adjust input size according to whether it is run on slow or fast hardware. Pseudocode:
    • Run each test case at a starting input size.
    • Increase the size until a certain amount of time has elapsed for any single test case (let's say more than a second). This sets the maximum running time so tests don't run too long.
  • Run each test a few times to return the average execution time & standard deviation for each test case. Pseudocode:
    • For each test case, run it at a certain input size repeatedly a number of times.
    • Return the average running time & standard deviation for each test case.
  • Be able to do a more comprehensive run at several input sizes and graph the input size vs. running time.

Thoughts

  • I thought of using benchmark.js first for benchmarking algorithms. It is a great library for measuring Javascript & DOM performance on various browsers, however when it comes to testing algorithms, it doesn't account for input size.
    • For example, algorithm running times can increase logarithmically, linearly, exponentially, etc. as the input gets larger (i.e. asymptotic notation). AFAICT, benchmark.js doesn't support benchmarking at various input sizes, and probably nor should it. If one were to properly test several algorithms, you'd have to run the algorithm at many input sizes to produce a graph of input size vs. running time. This would take an unacceptably long time for developers who just want to compare simple rendering operations across several browsers.
  • The function compilation feature in benchmark.js aims to make benchmarks more accurate, however as far as I know, the only extra cost in not inlining the test code into the loop is an extra function call, which is a constant time operation (as long as the same number of parameters are used for each call). I'm personally more interested in relative comparisons between code blocks, hence I'm most likely not going to implement this any time soon.

Interesting reads