Performance tester for Owl

This repository contains Python code to test Owl; a parser generator for visibly pushdown grammars. It is part of research by Luc Timmerman at the University of Twente.

Instructions for replication

Running these tests for yourself is simple (these instructions should work for any OS with:

1. Clone the repository;

2. Edit the variables at the top of main.py to your liking:

   • TOTAL_MEMORY_GB: the total memory in gigabytes which you'd like to dedicate to the testing;
   • TOTAL_WORKER_COUNT: the total number of threads you'd like to dedicate to the testing;
   • TYPES: the types you'd like to test for (by default, this is all of them);
   • TEST_COUNT: the number of tests;
   • STEP_SIZE: the size of the steps in size between tests;
   • REMOVE_AFTERWARDS: whether or not the generated grammars and resulting parsers should be removed after running.

   As such, when using the default values (TEST_COUNT of 1000 and STEP_SIZE of 10), 1000 tests will be run with sizes N = 0, N = 10, N = 20 up to N = 9990;

3. Install dependencies (like matplotlib and numpy) using the method of your choice;

4. Running can be done by simple executing python main.py. When testing large sample sizes, it might be beneficial to use nice (or START on Windows systems) to set a priority for the threads created by using nice -n 10 python3 main.py.

Depending on the setting of REMOVE_AFTERWARDS, this will generate a number of files:

• {grammar-type}_result.json files: these are files containing the results of the tests with an entry for every input size per grammar type including:
  • time: time to parse and generate in seconds;
  • lines: number of lines in the resulting parser.
• {grammar-type}.png: a fitted graph displaying the results for a certain grammar type;
• tests/: the grammars generated. The filenames are structured as follows: {grammar-type}_{N};
• parsers/: the parsers generated by Owl. Their filenames correspond to those of the generated grammars.