better-benchmark

Statistically correct benchmarking for Ruby.

This URL has Read+Write access

Added :teardown1 and :teardown2 as options to the ComparisonPartial 
options parameter.  These options take Proc's (or objects that respond to 
call()).  :teardown1 follows block1, and :teardown2 follows block2 on each 
iteration.  This is helpful if you need to clean up state or otherwise 
execute transactional code in your benchmark.

John Trupiano (author)

Tue Oct 14 09:29:17 -0700 2008

commit 2bc4a9f29aa94cf1603031c1a588213cb8495cc5
tree b70de1e0ba5d02da81dfe5219b9428209ee6e159
parent 277a2774ae154d67bf9acc4b566f7c65bb3dadad

better-benchmark / lib / better-benchmark / better-benchmark.rb

raw normal view history

c6be2536 »	John Trupiano	2008-10-07	Turned Pistos' better-bench...	1	require 'benchmark'
				2	require 'rsruby'
				3
				4	module Benchmark
				5
				6	class ComparisonPartial
				7	def initialize( block, options )
				8	@block1 = block
				9	@options = options
				10	end
				11
				12	def with( &block2 )
				13	times1 = []
				14	times2 = []
				15
2bc4a9f2 »	John Trupiano	2008-10-14	Added :teardown1 and :teard...	16	teardown1 = @options[:teardown1]
				17	teardown2 = @options[:teardown2]
c6be2536 »	John Trupiano	2008-10-07	Turned Pistos' better-bench...	18	(1..@options[ :iterations ]).each do \|iteration\|
				19	if @options[ :verbose ]
				20	$stdout.print "."; $stdout.flush
				21	end
				22
				23	times1 << Benchmark.realtime { @block1.call( iteration ) }
2bc4a9f2 »	John Trupiano	2008-10-14	Added :teardown1 and :teard...	24	teardown1.call if teardown1.respond_to?(:call)
c6be2536 »	John Trupiano	2008-10-07	Turned Pistos' better-bench...	25	times2 << Benchmark.realtime { block2.call( iteration ) }
2bc4a9f2 »	John Trupiano	2008-10-14	Added :teardown1 and :teard...	26	teardown2.call if teardown2.respond_to?(:call)
c6be2536 »	John Trupiano	2008-10-07	Turned Pistos' better-bench...	27	end
				28
				29	r = RSRuby.instance
				30	wilcox_result = r.wilcox_test( times1, times2 )
				31
				32	{
				33	:results1 => {
				34	:times => times1,
				35	:mean => r.mean( times1 ),
				36	:stddev => r.sd( times1 ),
				37	},
				38	:results2 => {
				39	:times => times2,
				40	:mean => r.mean( times2 ),
				41	:stddev => r.sd( times2 ),
				42	},
				43	:p => wilcox_result[ 'p.value' ],
				44	:W => wilcox_result[ 'statistic' ][ 'W' ],
				45	:significant => (
				46	wilcox_result[ 'p.value' ] < @options[ :required_significance ]
				47	),
				48	}
				49	end
				50	alias to with
				51	end
				52
				53	def self.compare_realtime( options = {}, &block1 )
				54	options[ :iterations ] \|\|= 20
				55	options[ :required_significance ] \|\|= 0.01
				56
				57	ComparisonPartial.new( block1, options )
				58	end
				59
				60	def self.report_on( result )
				61	puts
				62	puts( "Set 1 mean: %.3f s" % [ result[ :results1 ][ :mean ] ] )
				63	puts( "Set 1 std dev: %.3f" % [ result[ :results1 ][ :stddev ] ] )
				64	puts( "Set 2 mean: %.3f s" % [ result[ :results2 ][ :mean ] ] )
				65	puts( "Set 2 std dev: %.3f" % [ result[ :results2 ][ :stddev ] ] )
				66	puts "p.value: #{result[ :p ]}"
				67	puts "W: #{result[ :W ]}"
				68	puts(
				69	"The difference (%+.1f%%) %s statistically significant." % [
				70	( ( result[ :results2 ][ :mean ] - result[ :results1 ][ :mean ] ) / result[ :results1 ][ :mean ] ) * 100,
				71	result[ :significant ] ? 'IS' : 'IS NOT'
				72	]
				73	)
				74	end
				75	end

jtrupiano / better-benchmark forked from Pistos/better-benchmark

Pledgie Donations