The Big Switch

And so it begins ...

Merge-in the "next" branch, that contains the new version of LISA. The
old master can be found under the "legacy" branch that has been only
taking minor tweaks for quite a while now, while hot new stuff were
merged into next.

A transition guide listing major API changes and how to port code from
LISA legacy is available here:
https://lisa-linux-integrated-system-analysis.readthedocs.io/en/master/transition_guide.html

note: this is not a merge commit proper, in order to speed up git
subtree split. The commit of "next" branch merged is:
c543fb7 Merge pull request #871 from douglas-raillard-arm/_pr133

.gitignore

@@ -1,12 +1,14 @@
 *~
 .*
 *.pyc
+__pycache__
+*.egg-info
 /results*
-*.json
 *.log
 /tools/android-sdk-linux
-/ipynb/*.pid
-/ipynb/server.url
+*.pid
+server.url
 /vagrant
-/tools/wa_venv
 /tools/wa_user_directory/dependencies
+/src/buildroot
+_build

.travis.yml

@@ -15,16 +15,23 @@
 # limitations under the License.
 #
 
+git:
+  # make sure we have all relevant commits for subtree modif detection
+  depth: 1000
+
 sudo: required
-language: python
+# Use "generic" language to avoid having a virtualenv created for Python by default
+language: generic
+
+# Ubuntu Xenial 16.04 (latest version we can have on Travis)
+# https://docs.travis-ci.com/user/reference/overview/
+dist: xenial
+
 install:
-  - pip install --upgrade trappy bart-py devlib psutil wrapt matplotlib
-    future jupyter sphinx
+  - cd "$TRAVIS_BUILD_DIR"
+  - sudo ./install_base_ubuntu.sh --install-doc-extras
+  - echo "$(python3 --version)"
+
 script:
-  - cd $TRAVIS_BUILD_DIR
-  - 'echo backend : Agg > matplotlibrc' # Otherwise it tries to use tkinter
-  - export PATH=$TRAVIS_BUILD_DIR/tools/x86_64/:$PATH # For trace-cmd
-  - source init_env && lisa-test tests/lisa/
-  # -W is like -Werror.
-  # Ideally we should use -n too, which is like -Wall, but we have too many errors for that.
-  - cd doc/ && sphinx-build -W -b html . _build/html
+  - cd "$TRAVIS_BUILD_DIR"
+  - bash ./tools/scripts/travis_tests.sh

CONTRIBUTING.md

@@ -0,0 +1,66 @@
+# Contributing to LISA
+
+First of all, if you're reading this, thanks for thinking about contributing!
+This project is maintained by us Arm folks, but we welcome contributions from
+anyone.
+
+## How to reach us
+
+If you're hitting an error/bug and need help, it's best to raise an issue on github.
+
+## Coding style
+
+As a rule of thumb, the code you write should follow the
+[PEP-8](https://www.python.org/dev/peps/pep-0008/).
+
+We strongly recommend using a code checker such as [pylint](https://www.pylint.org/),
+as it tracks unused imports/variables, informs you when you can simplify a
+statement using Python features, and overall just helps you write better code.
+
+## Documentation
+
+Docstring documentation should follow the ReST/Sphinx style.
+Classes, class attributes and public methods must be documented. If deemed
+necessary, private methods can be documented as well.
+
+All in all, it should look like this:
+
+```python
+def foo(a, b):
+	"""
+	A one liner description
+
+	:param a: A description for param a
+	:type a: int
+
+	:param b: A description for param b
+	:type b: str
+
+	Whatever extra description you might over as many lines as you need
+	(but be reasonable)
+	"""
+	pass
+```
+
+## Tests
+
+You should strive to validate as much of your code as possible through self-tests.
+It’s a nice way to showcase that your code works, and also how it works. On top
+of that, it makes sure that later changes won’t break it.
+
+Have a look at [the doc](https://lisa-linux-integrated-system-analysis.readthedocs.io/en/next/lisa_tests.html) for more info on LISA self-tests.
+
+## Commits
+
+As for the shape of the commit, nothing out of the ordinary: just follow the
+good old 50/72 rule (it's okay if you bite off a few extra chars).
+
+The header should highlight the impacted files/classes. The 'lisa' prefix can be omitted - for instance,
+if you're modifying the `lisa/wlgen/rta.py` file, we'd expect a header of the shape `wlgen/rta: ...`.
+
+When that path gets a bit verbose, it's alright to shorten it as long as there
+is no confusion as to what you're referencing. In that case, if modifying the
+`lisa/tests/kernel/scheduler/load_tracking.py` file, we'd expect a header of
+the shape `tests: load_tracking: ...`.
+
+When in doubt, have a look at the git log.

README.md

@@ -1,17 +1,3 @@
-# A word about the future
-
-A revamped and streamlined version of LISA is available under the 'next' branch
-(https://github.com/ARM-software/lisa/tree/next). Active development is moving
-towards that branch, so while we'll accept small fixes to LISA/master until we feel
-confident it is time to flip the switch, any other kind of pull requests should be
-sent against LISA/next.
-
-In short, this what you should expect to see:
-* Complete redo of our test classes (LisaTest & Executor)
-* Various clean ups of LISA internals (e.g. TestEnv)
-* Migration to Python3
-* Documentation overhaul (as in, not just bare docstrings)
-
 # Introduction
 
 The LISA project provides a toolkit that supports regression testing and
@@ -48,7 +34,7 @@ There are two "entry points" for running LISA:
   automated pass/fail regression tests for kernel behaviour. The
   [BART](https://github.com/ARM-software/trappy) toolkit provides additional
   domain-specific test assertions for this use-case. LISA provides some
-  ready-made automated tests under the `tests/` directory.
+  ready-made automated tests under the `lisa/tests/` directory.
 
 # Motivations
 
@@ -66,13 +52,13 @@ The main goals of LISA are:
     * **enables** kernel developers to easily post process data to produce
       statistics and plots
 
-# Documentation
-
-* [Wiki Home page](https://github.com/ARM-software/lisa/wiki)
-* [Installation](https://github.com/ARM-software/lisa/wiki/Installation)
-* [Quickstart Tutorial](https://github.com/ARM-software/lisa/wiki/Quickstart-tutorial)
+# Documentation [![Documentation Status](https://readthedocs.org/projects/lisa-linux-integrated-system-analysis/badge/?version=next)](https://lisa-linux-integrated-system-analysis.readthedocs.io/en/next/?badge=next)
 
-* [API Documentation](http://lisa-linux-integrated-system-analysis.readthedocs.io/en/latest/)
+You should find everything on [ReadTheDocs](https://lisa-linux-integrated-system-analysis.readthedocs.io/en/next/).
+Here are some noteworthy sections:
+* [Installation](https://lisa-linux-integrated-system-analysis.readthedocs.io/en/next/users_guide.html#installation)
+* [Self-tests](https://lisa-linux-integrated-system-analysis.readthedocs.io/en/next/lisa_tests.html)
+* [Kernel tests](https://lisa-linux-integrated-system-analysis.readthedocs.io/en/next/kernel_tests.html)
 
 # External Links
 * Linux Integrated System Analysis (LISA) & Friends

Vagrantfile

@@ -3,7 +3,7 @@
 
 
 Vagrant.configure(2) do |config|
-  config.vm.box = "ubuntu/trusty64"
+  config.vm.box = "ubuntu/bionic64"
 
   # Compiling pandas requires 1Gb of memory
   config.vm.provider "virtualbox" do |v|
@@ -21,18 +21,28 @@ Vagrant.configure(2) do |config|
     fi
 
     cd /home/vagrant/lisa
+    # Install required packages
     ./install_base_ubuntu.sh --install-android-sdk
 
-    chown vagrant.vagrant /home/vagrant/lisa
-    echo cd /home/vagrant/lisa >> /home/vagrant/.bashrc
+    chown -R vagrant.vagrant /home/vagrant/lisa
+
+    # Let' use a venv local to vagrant so that we don't pollute the host one.
+    # This allows to use LISA both from the host and the VM.
+    export LISA_VENV_PATH=/home/vagrant/venv
+
+    # .bashrc setup
+    echo "cd /home/vagrant/lisa" >> /home/vagrant/.bashrc
     for LC in $(locale | cut -d= -f1);
     do
         echo unset $LC  >> /home/vagrant/.bashrc
     done
-    echo "export ANDROID_HOME=/vagrant/tools/android-sdk-linux" >> /home/vagrant/.bashrc
-    echo 'export PATH=\$ANDROID_HOME/platform-tools:\$ANDROID_HOME/tools:\$PATH' >> /home/vagrant/.bashrc
-    echo source init_env >> /home/vagrant/.bashrc
+    echo 'export LISA_VENV_PATH=$LISA_VENV_PATH' >> /home/vagrant/.bashrc
+    echo 'source init_env' >> /home/vagrant/.bashrc
+
+    # Trigger the creation of a venv
+    su vagrant bash -c 'source ./init_env'
 
+    # We're all done!
     echo "Virtual Machine Installation completed successfully!                "
     echo "                                                                    "
     echo "You can now access and use the virtual machine by running:          "
@@ -44,5 +54,20 @@ Vagrant.configure(2) do |config|
     echo "                                                                    "
     echo "    $ vagrant suspend                                               "
     echo "                                                                    "
+    echo " To destroy it, use:                                                "
+    echo "                                                                    "
+    echo "    $ vagrant destroy                                               "
+    echo "                                                                    "
   SHELL
+
+  # TODO: Run self tests to explode sooner rather than later
+  # config.trigger.after :up do |trigger|
+  #   trigger.info = "Verifying LISA installation"
+  #   trigger.name = "LISA install verification"
+  #   trigger.run_remote = {inline: "
+  #     cd /home/vagrant/lisa
+  #     source init_env
+  #     python3 -m nose
+  #   "}
+  # end
 end

devmode_requirements.txt

@@ -0,0 +1,18 @@
+# Must be ordered to satisfy the dependencies without pulling from PyPI.
+# Once they are found by pip in editable mode as specified here, they will be
+# used and not looked up on PyPI.
+-e ./tools/exekall
+
+# devlib before WA and LISA
+-e ./external/devlib/
+
+# TRAPpy before BART and LISA
+-e ./external/trappy/
+# BART before LISA
+-e ./external/bart/
+
+# WA before LISA
+-e ./external/workload-automation/
+-e ./[notebook,doc,test]
+
+-e ./tools/bisector[dbus]