Skip to content
This repository has been archived by the owner. It is now read-only.

Introduce performance benchmarking suite #824

Merged
merged 3 commits into from Jun 14, 2018

Conversation

@patiences
Copy link
Contributor

@patiences patiences commented May 29, 2018

In the coming weeks, as part of my GSoC project, there will be changes that aim to improve the performance of the code and so we need a way to evaluate that. This is a first PR with some example tests.
This test is not intended to be run like the other unit tests, rather it can be run from the command line and produces an output like

Running test_small_integers ...
  Elapsed time:  37.10159900219878  ms
  CPU process time:  274.72000000000025  ms
Running test_booleans ...
  Elapsed time:  18.298730996320955  ms
  CPU process time:  181.10000000000014  ms
tests/utils.py Outdated
transpiler.transpile_string("test.py", main_code)

if extra_code:
for name, code in extra_code.items():
transpiler.transpile_string("%s.py" % name.replace('.', os.path.sep), adjust(code))

t1_stop = time.perf_counter()
t2_stop = time.process_time()

Copy link
Member

@freakboy3742 freakboy3742 May 30, 2018

This won't be benchmarking what you think it is (or, at least, it's not benchmarking the thing that needs to be benchmarked. These timing calls are wrapped around the transpilation process - which is the process of converting Python code to Java bytecode. While this is something that certainly can be benchmarked, it's not the thing that is a performance concern - it's runtime performance that is a problem.

So - we need the start and end timers to be executed in Java, so that we're performance testing the execution of the code, not the compilation.

The simplest version of this would be to wrap the timer calls around the call to subprocess (since that's the part that spawns the actual invocation).

Copy link
Contributor Author

@patiences patiences May 30, 2018

Ah-ha.. I see. Thanks! Done

@@ -0,0 +1,31 @@
import sys
from os import path
sys.path.append( path.dirname( path.dirname( path.abspath(__file__) ) ) )
Copy link
Member

@freakboy3742 freakboy3742 May 30, 2018

The spacing on this is a little odd.

Copy link
Contributor Author

@patiences patiences May 30, 2018

Oops, fixed. Done

tests/utils.py Outdated
return out

def runAndBenchAsJava(self, test_name, code):
Copy link
Member

@freakboy3742 freakboy3742 May 30, 2018

I can see what you're doing here, but it's probably better served as 2 lines of code in the external wrapper invoking the code, rather than something on the utility class.

Copy link
Contributor Author

@patiences patiences May 30, 2018

Done

Copy link
Member

@freakboy3742 freakboy3742 left a comment

👍

@freakboy3742 freakboy3742 merged commit a0bbc11 into beeware:master Jun 14, 2018
5 checks passed
@patiences patiences deleted the perf-benchmarking branch Jul 2, 2018
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Linked issues

Successfully merging this pull request may close these issues.

None yet

2 participants