Permalink
Browse files

Bug 785129 - Create Autophone tests to run unit tests, r=mcote.

  • Loading branch information...
1 parent 6464449 commit 5d632feb44121ef01ac473ad2746d6d375d1ccf1 @bclary bclary committed Dec 20, 2012
View
@@ -0,0 +1,5 @@
+*.pyc
+*.out
+*.log
+*~
+configs/unittest_defaults.ini
View
@@ -24,9 +24,12 @@ Autophone doesn't yet support distuils, so some prerequisite Python packages
must be manually installed by pip, easy_install, or some other method: pytz,
pulsebuildmonitor, and mozprofile.
-At the moment autophone is packaged with only one test, named s1s2. It
-measures fennec load times for a couple different pages, served both remotely
-and from a local file.
+Autophone is packaged with two tests: s1s2 and unittests.
+
+s1s2
+----
+s1s2 measures fennec load times for a couple different pages,
+served both remotely and from a local file.
Put the pages to be served into autophone/configs. You will need a way to
serve them (FIXME: autophone should do this). If you're using phonedash,
@@ -58,6 +61,54 @@ from the mobile devices.
The resulturl is the URL used to POST results to the database.
+unittests
+---------
+
+Autophone requires additional Python packages in order to run the unittests:
+
+* logparser - http://hg.mozilla.org/automation/logparser
+* mozautolog - http://hg.mozilla.org/users/jgriffin_mozilla.com/mozautolog/
+
+The unittests also require a local installation of the XRE and the utility
+programs such as xpcshell. A local build of Firefox can be used.
+
+In order to process crash minidumps, you will also need a local
+installation of breakpad's minidump_stackwalk. You can build
+minidump_stack via:
+
+svn checkout http://google-breakpad.googlecode.com/svn/trunk/ google-breakpad-read-only
+cd google-breakpad-read-only
+if [[ $(uname) == "Darwin" ]]; then
+ CC=clang CXX=clang++ ./configure
+else
+ CXXFLAGS="-g -O1" ./configure
+fi
+make
+sudo make install
+
+If you wish to run a development environment, you will also need to
+set up an ElasticSearch and Autolog server. Note that you do not need
+to set up test data using testdata.py but you will need to create an
+autolog index using curl -XPUT 'http://localhost:9200/autolog/'
+once the ElasticSearch server is up and running and before
+you start the Autolog server.
+
+See
+https://wiki.mozilla.org/Auto-tools/Projects/Autolog for more details.
+
+To configure Autolog to display the results for a device, you will
+need to update the OSNames property in js/Config.js in Autolog. See
+http://hg.mozilla.org/automation/autolog/file/2a32ea0367f5/js/Config.js#l67 .
+Note that the key for the device should consist of the string 'autophone-'
+followed by the same value as used in SUTAgent.ini's HARDWARE property for
+the device.
+
+Once you have the XRE, utility programs and minidump_stack installed, change
+configs/unittest_default.ini to point to your local environment.
+
+Email notifications
+-------------------
+
If you want to get notifications indicating when Autophone has disabled
a device due to errors, you can create email.ini like so:
View
@@ -7,6 +7,14 @@ Starting Autophone
Autophone has a number of command-line options. Run "python autophone.py -h"
to see them. Some important ones are
+--test-path <testpath>: Specifies the test manifest which will load the
+ appropriate phone test. Autophone will use
+ tests/manifest.ini by default.
+
+--enable-unittests: Tells Autophone to download the appropriate tests.zip
+ for each build to use when running the unittests. This
+ is required when running the unittests.
+
--restarting: By default Autophone starts with no knowledge of any devices.
It creates a local cache as devices register with it. This
option preserves that cache when restarting.
@@ -28,6 +36,33 @@ to see them. Some important ones are
--loglevel: Log messages at or above this level. Can be set to
DEBUG (default), INFO, WARNING, or ERROR.
+Running Unit Tests
+------------------
+
+Autophone can run individual unit tests such as robocop, reftests,
+crashtests, jsreftests or mochitests for each build or it can run combinations of
+them.
+
+Before running the unit tests, you will need to copy
+configs/unittest_defaults.ini.example to configs/unittest_defaults.ini
+and edit configs/unittest_defaults.ini to change the xre_path,
+utility_path, and minidump_stackwalk values. If you wish to use a
+development version of ElasticSearch or Autolog, you will need to edit
+the es_server and rest_server values as well.
+
+You can switch from using the experimental 'new' logparser and
+logparser by changing the use_newparser value to False.
+
+For example,
+
+to run only the robocop tests:
+
+python autophone.py --enable-unittests --test-path=./tests/robocoptests_manifest.ini
+
+to run all of the unit tests specified in the configs/unittests_settings.ini file:
+
+python autophone.py --enable-unittests --test-path=./tests/unittests_manifest.ini
+
Running Tests
-------------
View
@@ -200,29 +200,10 @@ def worker_msg_loop(self):
except KeyboardInterrupt:
self.stop()
- # This runs the tests and resets the self._lasttest variable.
- # It also can install a new build, to install, set build_url to the URL of the
- # build to download and install
- def disperse_jobs(self):
- try:
- logging.debug('Asking for jobs')
- while not self._jobqueue.empty():
- job = self._jobqueue.get()
- logging.debug('Got job: %s' % job)
- for k,v in self._phonemap.iteritems():
- # TODO: Refactor so that the job can specify the test so that
- # then multiple types of test objects can be ran on one set of
- # phones.
- logging.debug('Adding job to phone: %s' % v['name'])
- ### FIXME: Need to serialize tests on each phone...
- for t in v['testobjs']:
- t.add_job(job)
- self._jobqueue.task_done()
- except:
- logging.error('Exception adding jobs: %s %s' % sys.exc_info()[:2])
-
# Start the phones for testing
def start_tests(self, job):
+ if not self.is_valid_job(job):
+ return
self.worker_lock.acquire()
for p in self.phone_workers.values():
logging.info('Starting job on phone: %s' % p.phone_cfg['phoneid'])
@@ -327,7 +308,8 @@ def register_cmd(self, data):
sutcmdport=int(data['cmdport'][0]),
machinetype=data['hardware'][0],
osver=data['os'][0],
- debug=3)
+ debug=3,
+ ipaddr=self.ipaddr)
self.register_phone(phone_cfg)
self.update_phone_cache()
else:
@@ -382,7 +364,7 @@ def read_tests(self):
def trigger_jobs(self, data):
logging.debug('trigger_jobs: data %s' % data)
job = self.build_job(self.get_build(data))
- logging.info('Adding user-specified job: %s' % job)
+ logging.info('Received user-specified job: %s' % job)
self.start_tests(job)
def reset_phones(self):
@@ -439,26 +421,62 @@ def build_job(self, cache_build_dir):
rev = cfg.get('App', 'SourceStamp')
ver = cfg.get('App', 'Version')
repo = cfg.get('App', 'SourceRepository')
- blddate = datetime.datetime.strptime(cfg.get('App', 'BuildID'),
+ buildid = cfg.get('App', 'BuildID')
+ blddate = datetime.datetime.strptime(buildid,
'%Y%m%d%H%M%S')
procname = ''
- if (repo == 'http://hg.mozilla.org/mozilla-central' or
- repo == 'http://hg.mozilla.org/integration/mozilla-inbound'):
+ if repo == 'http://hg.mozilla.org/mozilla-central':
+ tree = 'mozilla-central'
+ procname = 'org.mozilla.fennec'
+ elif repo == 'http://hg.mozilla.org/integration/mozilla-inbound':
+ tree = 'mozilla-inbound'
procname = 'org.mozilla.fennec'
elif repo == 'http://hg.mozilla.org/releases/mozilla-aurora':
+ tree = 'mozilla-aurora'
procname = 'org.mozilla.fennec_aurora'
elif repo == 'http://hg.mozilla.org/releases/mozilla-beta':
+ tree = 'mozilla-beta'
procname = 'org.mozilla.firefox'
- job = { 'cache_build_dir': cache_build_dir,
- 'blddate': math.trunc(time.mktime(blddate.timetuple())),
- 'revision': rev,
- 'androidprocname': procname,
- 'version': ver,
- 'bldtype': 'opt' }
+ job = {'cache_build_dir': cache_build_dir,
+ 'tree': tree,
+ 'blddate': math.trunc(time.mktime(blddate.timetuple())),
+ 'buildid': buildid,
+ 'revision': rev,
+ 'androidprocname': procname,
+ 'version': ver,
+ 'bldtype': 'opt'}
shutil.rmtree(tmpdir)
return job
+ def is_valid_job(self, job):
+ if job is None:
+ return False
+
+ error_list = []
+
+ if 'androidprocname' not in job:
+ error_list.append('missing androidprocname')
+
+ if 'revision' not in job:
+ error_list.append('missing revision')
+
+ if 'blddate' not in job:
+ error_list.append('missing blddate')
+
+ if 'bldtype' not in job:
+ error_list.append('missing bldtype')
+
+ if 'version' not in job:
+ error_list.append('missing version')
+
+ if len(error_list) > 0:
+ error_message = 'ERROR: Invalid job configuration: %s ' % job + ', '.join(error_list)
+ self.logger.error(error_message)
+ raise NameError(error_message)
+
+ return True
+
def stop(self):
self._stop = True
self.server.shutdown()
View
@@ -223,6 +223,31 @@ def get(self, buildurl, enable_unittests, force=False):
return None
os.rename(tmpf.name, build_path)
file(os.path.join(cache_build_dir, 'lastused'), 'w')
+ symbols_path = os.path.join(cache_build_dir, 'symbols')
+ if force or not os.path.exists(symbols_path):
+ tmpf = tempfile.NamedTemporaryFile(delete=False)
+ tmpf.close()
+ # XXX: assumes fixed buildurl-> symbols_url mapping
+ symbols_url = re.sub('.apk$', '.crashreporter-symbols.zip', buildurl)
+ try:
+ urllib.urlretrieve(symbols_url, tmpf.name)
+ symbols_zipfile = zipfile.ZipFile(tmpf.name)
+ symbols_zipfile.extractall(symbols_path)
+ symbols_zipfile.close()
+ except IOError, ioerror:
+ if '550 Failed to change directory' in ioerror.strerror.strerror.message:
+ logging.info('No symbols found: %s.' % symbols_url)
+ else:
+ logging.error('IO Error retrieving symbols: %s.' % symbols_url)
+ logging.error(traceback.format_exc())
+ except zipfile.BadZipfile:
+ logging.info('Ignoring zipfile.BadZipFile Error retrieving symbols: %s.' % symbols_url)
+ try:
+ with open(tmpf.name, 'r') as badzipfile:
+ logging.debug(badzipfile.read())
+ except:
+ pass
+ os.unlink(tmpf.name)
if enable_unittests:
tests_path = os.path.join(cache_build_dir, 'tests')
if (force or not os.path.exists(tests_path)) and enable_unittests:
@@ -0,0 +1,15 @@
+[runtests]
+# Settings related to executing runtestsremote.py
+
+# test_name is a descriptor used by runtestsremote.py to
+# determine which of the downloaded unit tests to run.
+test_name = crashtest
+
+unittest_defaults = configs/unittest_defaults.ini
+
+# How many times to run the tests per phone.
+iterations = 1
+
+# How many chunks for the test
+total_chunks = 1
+
@@ -0,0 +1,14 @@
+[runtests]
+# Settings related to executing runtestsremote.py
+
+# test_name is a descriptor used by runtestsremote.py to
+# determine which of the downloaded unit tests to run.
+test_name = jsreftest
+
+unittest_defaults = configs/unittest_defaults.ini
+
+# How many times to run the tests per phone.
+iterations = 1
+
+# How many chunks for the test
+total_chunks = 9
@@ -0,0 +1,14 @@
+[runtests]
+# Settings related to executing runtestsremote.py
+
+# test_name is a descriptor used by runtestsremote.py to
+# determine which of the downloaded unit tests to run.
+test_name = mochitest
+
+unittest_defaults = configs/unittest_defaults.ini
+
+# How many times to run the tests per phone.
+iterations = 1
+
+# How many chunks for the test
+total_chunks = 9
@@ -0,0 +1,14 @@
+[runtests]
+# Settings related to executing runtestsremote.py
+
+# test_name is a descriptor used by runtestsremote.py to
+# determine which of the downloaded unit tests to run.
+test_name = reftest
+
+unittest_defaults = configs/unittest_defaults.ini
+
+# How many times to run the tests per phone.
+iterations = 1
+
+# How many chunks for the test
+total_chunks = 9
@@ -0,0 +1,8 @@
+[runtests]
+# Settings related to executing runtestsremote.py
+
+# test_name is a descriptor used by runtestsremote.py to
+# determine which of the downloaded unit tests to run.
+test_name = robocoptest
+
+unittest_defaults = configs/unittest_defaults.ini
@@ -0,0 +1,47 @@
+[runtests]
+# Settings related to executing runtestsremote.py
+
+# xre_path is the path to the XRE (probably xulrunner)
+xre_path = path-to-firefox-bin
+
+# utility_path is the path to the necessary utility programs such as
+# xpcshell etc.
+utility_path = path-to-firefox-bin
+
+# minidump_stackwalk is the path to the breakpad utility to process
+# minidumps.
+minidump_stackwalk = path-to-minidump_stackwalk
+
+# Specify androidprocname if you are testing with a custom build which
+# does not follow the conventions in autophone.py:build_job. e.g.
+# androidprocname = org.mozilla.fennec_foobar
+
+# Debug message levels for the runtestsremote.py
+console_level = DEBUG
+file_level = DEBUG
+
+time_out = 300
+
+[autolog]
+# Settings related to submitting results to Autolog
+
+# ElasticSearch server
+# For local development use localhost:9200
+es_server = buildbot-es.metrics.scl3.mozilla.com:9200
+
+# Autolog server
+# For local development use http://localhost:8051
+rest_server = http://brasstacks.mozilla.com/autologserver/
+
+# Submit passing results to Autolog. Note: Setting include_pass to
+# True will cause a failure until RESTfulAutologTestGroup.submit
+# supports submitting results via POST.
+include_pass = False
+
+# Submit log to Autolog.
+submit_log = True
+
+# Set use_newparser to True to parse the logs using newlogparser.py
+# instead of logparser/logparser.py. Set it to False to use
+# logparser/logparser.py.
+use_newparser = True
Oops, something went wrong.

0 comments on commit 5d632fe

Please sign in to comment.