Skip to content
Set of shell scripts to run Moodle performance tests using different hardware and configurations and compare results.
Branch: master
Clone or download
Latest commit fd57c0c May 19, 2019
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
lib
logs
runs
runs_outputs
runs_samples
tests
webapp webapp: Trim instead of stripping non-numbers Oct 25, 2017
.gitignore Saving the jmeter requests outputs in runs_outputs Sep 12, 2013
CHANGELOG.md Issue #44 - Show average per users*loopcount in graph Mar 28, 2014
LICENSE.md
README.md
TROUBLESHOOTING.md Issue #40 - Adding the troubleshooting guide Jan 15, 2014
after_run_setup.sh
before_run_setup.sh
clean_data.sh Issue #47 - Removing sudo Feb 20, 2015
clean_webapp_data.sh
compare.sh Issue #20 - Adding a setting to specify compare.sh size Jan 3, 2014
config.php.template
create_site_data_file.sh
defaults.properties Issue #10 - New setting to set readonly web mode Feb 17, 2014
delete_run.php
details.php
detect_big_differences.php Fixing missing dependencies in detect_big_differences.php Feb 18, 2014
download_run.php Issue #10 - Adding delete runs & download runs capabilities Feb 17, 2014
index.php
jmeter_config.properties.dist Issue #43 - Added time elapsed by the DB engine running DB queries Mar 28, 2014
recorder.bsf Improve recorder sessionsize detection Aug 4, 2017
recorderfunctions.bsf Issue #10 - Adding delete runs & download runs capabilities Feb 17, 2014
restart_services.sh Error messages to STDERR and other changes to improve readability Jan 3, 2014
set_moodle_site.php Script to configure the moodle site with advanced features Sep 23, 2013
test_runner.sh Issue #40 - Checking that the site was properly installed Jan 15, 2014
webserver_config.properties.dist Update to Moodle 3.7 May 19, 2019

README.md

Tools to compare Moodle sites and/or branches performance.

Purpose

They can be used to compare:

Features

  • Clean site installation (from Moodle 2.5 onwards)
  • Fixed data set generation with courses, users, enrolments, module instances... (from Moodle 2.5 onwards)
  • JMeter test plan generation from course contents (from Moodle 2.5 onwards)
  • Web and database warm-up processes included in the test plan (results not collected)
  • JMeter runs gathering results about moodle performance data (database reads/writes/querytime, memory usage...)
  • Database query time value will differ depending on database and hardware used.
  • Runs results comparison

There are scripts for both the web server and the JMeter server sides.

  • In case they are both in the same server you just need to clone the project once.
  • In case they are in different servers you need to clone the project in both servers
    • test_runner.sh along with jmeter_config.properties will be used in the server hosting JMeter
    • before_run_setup.sh, after_run_setup.sh and restart_services.sh will be used in the web server

Requirements

Installation

The installation process differs depending whether you have both the web server and JMeter in the same computer or not.

Web and JMeter servers in the same computer (usually a development computer)

  • Get the code
    • cd /var/www (or any other place, but accessible through a web server, not a public one please, read security below)
    • git clone git://github.com/moodlehq/moodle-performance-comparison.git moodle-performance-comparison
    • cd moodle-performance-comparison
  • Configure the tool
    • cp webserver_config.properties.dist webserver_config.properties
    • Edit webserver_config.properties with your own values
    • cp jmeter_config.properties.dist jmeter_config.properties
    • Edit jmeter_config.properties with your own values

Web server and JMeter running from a different server

  • Get the code in the web server
    • cd /var/www (or any other place, but accessible through a web server, not a public one please, read security below)
    • git clone git://github.com/moodlehq/moodle-performance-comparison.git moodle-performance-comparison
    • cd moodle-performance-comparison
  • Get the code in the JMeter server
    • cd /wherever/you/want
    • git clone git://github.com/moodlehq/moodle-performance-comparison.git moodle-performance-comparison
    • cd moodle-performance-comparison
  • Configure the tool in the web server
    • cp webserver_config.properties.dist webserver_config.properties
    • Edit webserver_config.properties with your own values
  • Configure the tool in the JMeter server
    • cp jmeter_config.properties.dist jmeter_config.properties
    • Edit jmeter_config.properties with your own values

Usage

The simplest is to just execute compare.sh, but it will only work in development computers where jmeter is installed in the web server and when you are testing differences between different branches. For other cases the process also differs depending whether you have both web server and JMeter in the same computer or not. Here there is another alternative, you can load your sql dump instead of having a clean brand new site with a fixed dataset, so you can run the generated test plan using real site generated data.

The groupname and description arguments of test_runner.sh are useful to identify the run when comparing results, you can use it to set the branch name, the settings you used or whatever will help you identify which run is it.

Note that you can run the tests as many times as you want, you just need to run after_run_setup.sh and restart_services.sh before running test_runner.sh every time to clean up the site.

It is recommendable that you run all the scripts using the same user (there is no need to use a root user at all) you can use different users to run them (there are no restrictions about it) but be sure that the permissions are correct, it seems to be one of the more common issues when running this tool.

Web and JMeter servers in the same computer, to find performance differences between different branches (usually a development computer)

  • Run compare.sh, the browser will be automatically opened after both runs are finished
    • ./compare.sh
  • In case the browser doesn't open properly the comparison page, browse to

Web and JMeter servers in the same computer, to find performance differences changing site settings / cache stores

  • Generate the data and run the tests
    • cd /path/to/moodle-performance-comparison
    • ./before_run_setup.sh {XS|S|M|L|XL|XXL}
    • Change site settings if necessary according to what you are comparing
    • ./restart_services.sh
    • ./test_runner.sh {groupname} {descriptioname}
    • ./after_run_setup.sh
    • Change site settings if necessary according to what you are comparing
    • ./restart_services.sh
    • ./test_runner.sh {groupname} {descriptioname}
  • Check the results

Web server and JMeter running from a different server

  • Generate the data and the test plan (web server)
    • cd /path/to/moodle-performance-comparison
    • ./before_run_setup.sh {XS|S|M|L|XL|XXL}
    • Change site settings if necessary according to what you are comparing
    • ./restart_services.sh
  • Get the test plan files (jmeter server)
  • Get the $beforebranch moodle version data (jmeter server)
  • Run the before test (jmeter server)
    • cd /path/to/moodle-performance-comparison
    • ./test_runner.sh {groupname} {descriptioname} testplan.jmx testusers.csv site_data.properties
  • Restore the base state to run the after branch (web server)
    • cd /path/to/moodle-performance-comparison
    • ./after_run_setup.sh
    • Change site settings if necessary according to what you are comparing
    • ./restart_services.sh
  • Get the $afterbranch moodle version data (jmeter server)
  • Run the after test (jmeter server)
    • cd /path/to/moodle-performance-comparison
    • ./test_runner.sh {groupname} {descriptioname} testplan.jmx testusers.csv site_data.properties
  • Check the results (web server)

Using your own sql dump (Moodle 2.5 onwards)

The installation and configuration is the same, it also depends on if you use the same computer for both web server and JMeter or not, but the usage changes when you want to use your own sql dump, it is not that easy to automate, as you need to specify which course do you want to use as target course and you can not use before_run_setup.sh to generate the test plan and test_files.properties.

  • cd /webserver/path/to/moodle-performance-comparison
  • Restore your dataroot to $dataroot
  • Restore your database to $dbname in $dbhost
  • Get the moodle code
  • Upgrade the site to $beforebranch
    • cd moodle/
    • git checkout $beforebranch
    • php admin/cli/upgrade.php --allow-unstable --non-interactive
  • Generate the test plan updating users passwords. You need to provide the shortname of the course that will be tested
    • php admin/tool/generator/cli/maketestplan.php --size="THESIZEYOUWANT" --shortname="TARGETCOURSESHORTNAME" --bypasscheck --updateuserspassword
  • Generate the site_data.properties file, with the current moodle version data, in the root directory of moodle-performance-comparison
    • cd ..
    • ./create_site_data_file.sh
  • Download the test plan and the test users. The URLs are provided by maketestsite.php in the previous step, before the performance info output begins.
  • Backup dataroot and database (pg_dump or mysqldump), this backup will contain the updated passwords
  • Create moodle-performance-comparison/test_files.properties with the backups you just generated and the test plan data
    • cd ../
    • Create a new /path/to/moodle-performance-comparison/test_files.properties file with the following content:

testplanfile="/absolute/path/to/testplan.jmx"

datarootbackup="/absolute/path/to/the/dataroot/backup/directory"

testusersfile="/absolute/path/to/testusers.csv"

databasebackup="/absolute/path/to/the/database/backup.sql"

  • cd /path/to/moodle-performance-comparison and continue the normal process from restart_services.sh -> test_runner.sh -> after_run_setup.sh -> restart_services.sh -> test_runner.sh

Using your own sql dump (before Moodle 2.5)

Moodle 2.5 introduces the site and the test plan generators, so you can not use them if you are comparing previous branches. But you can:

  • Use the template included in Moodle 2.5 codebase and fill the placeholders with one of your site courses info and the test plan users, loops and ramp up period
    • The test plan template is located in admin/tool/generator/testplan.template.jmx
  • Fill a testusers.php with the target course data
    • You will need to check that the test data has enough users according to the data you provided in the test plan
  • Generate the site_data.properties file, with the current moodle version data, in the root directory of moodle-performance-comparison
    • cd ..
    • ./create_site_data_file.sh
  • Follow Using your own sql dump (Moodle 2.5 onwards) instructions

Advanced usage

  • You can overwrite the values provided by the test plan using test_runner.sh options:
    • -u=[users_number]
    • -l=[loops_number]
    • -r=[rampup_period]
    • -t=[throughput]

Security

This tool in only intended to be used in development/testing environments inside the local network, it would be insecure to expose the project root in a public accessible web server, the same only exposing moodle/ directory:

  • Database connection data and other sensitive data is stored in properties files (you can change permissions)
  • It uses default sugar passwords (you can change the defaults in webserver_config.properties)
  • Stores test users credentials in Moodle's wwwroot (you can change permissions)
  • In general all files permissions are non secure at all (you can change permissions)
  • Other things I probably forgot, to resume, don't do it unless you are sure what you are doing

Troubleshooting

  • You can find an extensive troubleshooting guide here
  • You might be interested in raising the PHP memory_limit to 512MB (apache) or something like that to 'M' or bigger when comparing results.
  • You can find JMeter logs in logs/
  • You can find runs outputs in runs_outputs/ the results in runs_samples/ and the php arrays generated from them in runs/
  • The generated .jtl files can be big. Don't hesitate to get rid of them if you don't need them for extra analytic purposes.
  • Same with $backupsdir/ contents, if you run before_run_setup.sh many time you will have a looot of hd space wasted.
  • If files with java_pid[\d]+.hprof format are generated in your project root means that jmeter is running out of resource. http://wiki.apache.org/jmeter/JMeterFAQ#JMeter_keeps_getting_.22Out_of_Memory.22_errors.__What_can_I_do.3F for more info.
You can’t perform that action at this time.