Bug 785129 - Create Autophone tests to run unit tests, r=mcote.

.gitignore

@@ -0,0 +1,5 @@
+*.pyc
+*.out
+*.log
+*~
+configs/unittest_defaults.ini

INSTALL.md

@@ -24,9 +24,12 @@ Autophone doesn't yet support distuils, so some prerequisite Python packages
 must be manually installed by pip, easy_install, or some other method: pytz,
 pulsebuildmonitor, and mozprofile.
 
-At the moment autophone is packaged with only one test, named s1s2. It
-measures fennec load times for a couple different pages, served both remotely
-and from a local file.
+Autophone is packaged with two tests: s1s2 and unittests.
+
+s1s2
+----
+s1s2 measures fennec load times for a couple different pages,
+served both remotely and from a local file.
 
 Put the pages to be served into autophone/configs. You will need a way to
 serve them (FIXME: autophone should do this). If you're using phonedash,
@@ -58,6 +61,54 @@ from the mobile devices.
 
 The resulturl is the URL used to POST results to the database.
 
+unittests
+---------
+
+Autophone requires additional Python packages in order to run the unittests:
+
+* logparser  - http://hg.mozilla.org/automation/logparser
+* mozautolog - http://hg.mozilla.org/users/jgriffin_mozilla.com/mozautolog/
+
+The unittests also require a local installation of the XRE and the utility
+programs such as xpcshell. A local build of Firefox can be used.
+
+In order to process crash minidumps, you will also need a local
+installation of breakpad's minidump_stackwalk. You can build
+minidump_stack via:
+
+svn checkout http://google-breakpad.googlecode.com/svn/trunk/ google-breakpad-read-only
+cd google-breakpad-read-only
+if [[ $(uname) == "Darwin" ]]; then
+    CC=clang CXX=clang++ ./configure
+else
+   CXXFLAGS="-g -O1" ./configure
+fi
+make
+sudo make install
+
+If you wish to run a development environment, you will also need to
+set up an ElasticSearch and Autolog server. Note that you do not need
+to set up test data using testdata.py but you will need to create an
+autolog index using curl -XPUT 'http://localhost:9200/autolog/'
+once the ElasticSearch server is up and running and before
+you start the Autolog server.
+
+See
+https://wiki.mozilla.org/Auto-tools/Projects/Autolog for more details.
+
+To configure Autolog to display the results for a device, you will
+need to update the OSNames property in js/Config.js in Autolog. See
+http://hg.mozilla.org/automation/autolog/file/2a32ea0367f5/js/Config.js#l67 .
+Note that the key for the device should consist of the string 'autophone-'
+followed by the same value as used in SUTAgent.ini's HARDWARE property for
+the device.
+
+Once you have the XRE, utility programs and minidump_stack installed, change
+configs/unittest_default.ini to point to your local environment.
+
+Email notifications
+-------------------
+
 If you want to get notifications indicating when Autophone has disabled
 a device due to errors, you can create email.ini like so:
 

USAGE.md

@@ -7,6 +7,14 @@ Starting Autophone
 Autophone has a number of command-line options. Run "python autophone.py -h"
 to see them. Some important ones are
 
+--test-path <testpath>: Specifies the test manifest which will load the
+                        appropriate phone test. Autophone will use
+                        tests/manifest.ini by default.
+
+--enable-unittests: Tells Autophone to download the appropriate tests.zip
+                    for each build to use when running the unittests. This
+                    is required when running the unittests.
+
 --restarting: By default Autophone starts with no knowledge of any devices.
               It creates a local cache as devices register with it. This
               option preserves that cache when restarting.
@@ -28,6 +36,33 @@ to see them. Some important ones are
 --loglevel: Log messages at or above this level. Can be set to
             DEBUG (default), INFO, WARNING, or ERROR.
 
+Running Unit Tests
+------------------
+
+Autophone can run individual unit tests such as robocop, reftests,
+crashtests, jsreftests or mochitests for each build or it can run combinations of
+them.
+
+Before running the unit tests, you will need to copy
+configs/unittest_defaults.ini.example to configs/unittest_defaults.ini
+and edit configs/unittest_defaults.ini to change the xre_path,
+utility_path, and minidump_stackwalk values. If you wish to use a
+development version of ElasticSearch or Autolog, you will need to edit
+the es_server and rest_server values as well.
+
+You can switch from using the experimental 'new' logparser and
+logparser by changing the use_newparser value to False.
+
+For example,
+
+to run only the robocop tests:
+
+python autophone.py --enable-unittests --test-path=./tests/robocoptests_manifest.ini
+
+to run all of the unit tests specified in the configs/unittests_settings.ini file:
+
+python autophone.py --enable-unittests --test-path=./tests/unittests_manifest.ini
+
 
 Running Tests
 -------------

autophone.py

@@ -200,29 +200,10 @@ def worker_msg_loop(self):
         except KeyboardInterrupt:
             self.stop()
 
-    # This runs the tests and resets the self._lasttest variable.
-    # It also can install a new build, to install, set build_url to the URL of the
-    # build to download and install
-    def disperse_jobs(self):
-        try:
-            logging.debug('Asking for jobs')
-            while not self._jobqueue.empty():
-                job = self._jobqueue.get()
-                logging.debug('Got job: %s' % job)
-                for k,v in self._phonemap.iteritems():
-                    # TODO: Refactor so that the job can specify the test so that
-                    # then multiple types of test objects can be ran on one set of
-                    # phones.
-                    logging.debug('Adding job to phone: %s' % v['name'])
-                    ### FIXME: Need to serialize tests on each phone...
-                    for t in v['testobjs']:
-                        t.add_job(job)
-                self._jobqueue.task_done()
-        except:
-            logging.error('Exception adding jobs: %s %s' % sys.exc_info()[:2])
-
     # Start the phones for testing
     def start_tests(self, job):
+        if not self.is_valid_job(job):
+            return
         self.worker_lock.acquire()
         for p in self.phone_workers.values():
             logging.info('Starting job on phone: %s' % p.phone_cfg['phoneid'])
@@ -327,7 +308,8 @@ def register_cmd(self, data):
                     sutcmdport=int(data['cmdport'][0]),
                     machinetype=data['hardware'][0],
                     osver=data['os'][0],
-                    debug=3)
+                    debug=3,
+                    ipaddr=self.ipaddr)
                 self.register_phone(phone_cfg)
                 self.update_phone_cache()
             else:
@@ -382,7 +364,7 @@ def read_tests(self):
     def trigger_jobs(self, data):
         logging.debug('trigger_jobs: data  %s' % data)
         job = self.build_job(self.get_build(data))
-        logging.info('Adding user-specified job: %s' % job)
+        logging.info('Received user-specified job: %s' % job)
         self.start_tests(job)
 
     def reset_phones(self):
@@ -439,26 +421,62 @@ def build_job(self, cache_build_dir):
         rev = cfg.get('App', 'SourceStamp')
         ver = cfg.get('App', 'Version')
         repo = cfg.get('App', 'SourceRepository')
-        blddate = datetime.datetime.strptime(cfg.get('App', 'BuildID'),
+        buildid = cfg.get('App', 'BuildID')
+        blddate = datetime.datetime.strptime(buildid,
                                              '%Y%m%d%H%M%S')
         procname = ''
-        if (repo == 'http://hg.mozilla.org/mozilla-central' or
-            repo == 'http://hg.mozilla.org/integration/mozilla-inbound'):
+        if repo == 'http://hg.mozilla.org/mozilla-central':
+            tree = 'mozilla-central'
+            procname = 'org.mozilla.fennec'
+        elif repo == 'http://hg.mozilla.org/integration/mozilla-inbound':
+            tree = 'mozilla-inbound'
             procname = 'org.mozilla.fennec'
         elif repo == 'http://hg.mozilla.org/releases/mozilla-aurora':
+            tree = 'mozilla-aurora'
             procname = 'org.mozilla.fennec_aurora'
         elif repo == 'http://hg.mozilla.org/releases/mozilla-beta':
+            tree = 'mozilla-beta'
             procname = 'org.mozilla.firefox'
 
-        job = { 'cache_build_dir': cache_build_dir,
-                'blddate': math.trunc(time.mktime(blddate.timetuple())),
-                'revision': rev,
-                'androidprocname': procname,
-                'version': ver,
-                'bldtype': 'opt' }
+        job = {'cache_build_dir': cache_build_dir,
+               'tree': tree,
+               'blddate': math.trunc(time.mktime(blddate.timetuple())),
+               'buildid': buildid,
+               'revision': rev,
+               'androidprocname': procname,
+               'version': ver,
+               'bldtype': 'opt'}
         shutil.rmtree(tmpdir)
         return job
 
+    def is_valid_job(self, job):
+        if job is None:
+            return False
+
+        error_list = []
+
+        if 'androidprocname' not in job:
+            error_list.append('missing androidprocname')
+
+        if 'revision' not in job:
+            error_list.append('missing revision')
+
+        if 'blddate' not in job:
+            error_list.append('missing blddate')
+
+        if 'bldtype' not in job:
+            error_list.append('missing bldtype')
+
+        if 'version' not in job:
+            error_list.append('missing version')
+
+        if len(error_list) > 0:
+            error_message = 'ERROR: Invalid job configuration: %s ' % job + ', '.join(error_list)
+            self.logger.error(error_message)
+            raise NameError(error_message)
+
+        return True
+
     def stop(self):
         self._stop = True
         self.server.shutdown()

builds.py

@@ -223,6 +223,31 @@ def get(self, buildurl, enable_unittests, force=False):
                 return None
             os.rename(tmpf.name, build_path)
         file(os.path.join(cache_build_dir, 'lastused'), 'w')
+        symbols_path = os.path.join(cache_build_dir, 'symbols')
+        if force or not os.path.exists(symbols_path):
+            tmpf = tempfile.NamedTemporaryFile(delete=False)
+            tmpf.close()
+            # XXX: assumes fixed buildurl-> symbols_url mapping
+            symbols_url = re.sub('.apk$', '.crashreporter-symbols.zip', buildurl)
+            try:
+                urllib.urlretrieve(symbols_url, tmpf.name)
+                symbols_zipfile = zipfile.ZipFile(tmpf.name)
+                symbols_zipfile.extractall(symbols_path)
+                symbols_zipfile.close()
+            except IOError, ioerror:
+                if '550 Failed to change directory' in ioerror.strerror.strerror.message:
+                    logging.info('No symbols found: %s.' % symbols_url)
+                else:
+                    logging.error('IO Error retrieving symbols: %s.' % symbols_url)
+                    logging.error(traceback.format_exc())
+            except zipfile.BadZipfile:
+                logging.info('Ignoring zipfile.BadZipFile Error retrieving symbols: %s.' % symbols_url)
+                try:
+                    with open(tmpf.name, 'r') as badzipfile:
+                        logging.debug(badzipfile.read())
+                except:
+                    pass
+            os.unlink(tmpf.name)
         if enable_unittests:
             tests_path = os.path.join(cache_build_dir, 'tests')
             if (force or not os.path.exists(tests_path)) and enable_unittests:

configs/crashtests_settings.ini

@@ -0,0 +1,15 @@
+[runtests]
+# Settings related to executing runtestsremote.py
+
+# test_name is a descriptor used by runtestsremote.py to
+# determine which of the downloaded unit tests to run.
+test_name = crashtest
+
+unittest_defaults = configs/unittest_defaults.ini
+
+# How many times to run the tests per phone.
+iterations = 1
+
+# How many chunks for the test
+total_chunks = 1
+

configs/jsreftests_settings.ini

@@ -0,0 +1,14 @@
+[runtests]
+# Settings related to executing runtestsremote.py
+
+# test_name is a descriptor used by runtestsremote.py to
+# determine which of the downloaded unit tests to run.
+test_name = jsreftest
+
+unittest_defaults = configs/unittest_defaults.ini
+
+# How many times to run the tests per phone.
+iterations = 1
+
+# How many chunks for the test
+total_chunks = 9

configs/mochitests_settings.ini

@@ -0,0 +1,14 @@
+[runtests]
+# Settings related to executing runtestsremote.py
+
+# test_name is a descriptor used by runtestsremote.py to
+# determine which of the downloaded unit tests to run.
+test_name = mochitest
+
+unittest_defaults = configs/unittest_defaults.ini
+
+# How many times to run the tests per phone.
+iterations = 1
+
+# How many chunks for the test
+total_chunks = 9

configs/reftests_settings.ini

@@ -0,0 +1,14 @@
+[runtests]
+# Settings related to executing runtestsremote.py
+
+# test_name is a descriptor used by runtestsremote.py to
+# determine which of the downloaded unit tests to run.
+test_name = reftest
+
+unittest_defaults = configs/unittest_defaults.ini
+
+# How many times to run the tests per phone.
+iterations = 1
+
+# How many chunks for the test
+total_chunks = 9

configs/robocoptests_settings.ini

@@ -0,0 +1,8 @@
+[runtests]
+# Settings related to executing runtestsremote.py
+
+# test_name is a descriptor used by runtestsremote.py to
+# determine which of the downloaded unit tests to run.
+test_name = robocoptest
+
+unittest_defaults = configs/unittest_defaults.ini

configs/unittest_defaults.ini.example

@@ -0,0 +1,47 @@
+[runtests]
+# Settings related to executing runtestsremote.py
+
+# xre_path is the path to the XRE (probably xulrunner)
+xre_path = path-to-firefox-bin
+
+# utility_path is the path to the necessary utility programs such as
+# xpcshell etc.
+utility_path = path-to-firefox-bin
+
+# minidump_stackwalk is the path to the breakpad utility to process
+# minidumps.
+minidump_stackwalk = path-to-minidump_stackwalk
+
+# Specify androidprocname if you are testing with a custom build which
+# does not follow the conventions in autophone.py:build_job. e.g.
+# androidprocname = org.mozilla.fennec_foobar
+
+# Debug message levels for the runtestsremote.py
+console_level = DEBUG
+file_level = DEBUG
+
+time_out = 300
+
+[autolog]
+# Settings related to submitting results to Autolog
+
+# ElasticSearch server
+# For local development use localhost:9200
+es_server = buildbot-es.metrics.scl3.mozilla.com:9200
+
+# Autolog server
+# For local development use http://localhost:8051
+rest_server = http://brasstacks.mozilla.com/autologserver/
+
+# Submit passing results to Autolog.  Note: Setting include_pass to
+# True will cause a failure until RESTfulAutologTestGroup.submit
+# supports submitting results via POST.
+include_pass = False
+
+# Submit log to Autolog.
+submit_log = True
+
+# Set use_newparser to True to parse the logs using newlogparser.py
+# instead of logparser/logparser.py. Set it to False to use
+# logparser/logparser.py.
+use_newparser = True