Skip to content

Commit

Permalink
Merge fe43b20 into bf90c3b
Browse files Browse the repository at this point in the history
  • Loading branch information
Maikerupitazu committed Jul 22, 2020
2 parents bf90c3b + fe43b20 commit 2cd49fa
Show file tree
Hide file tree
Showing 9 changed files with 145 additions and 48 deletions.
29 changes: 29 additions & 0 deletions AndroidRunner/Plugins/android/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,29 @@
# Android Plugin
This plugin collects memory and CPU usage via the `cpuinfo` and `meminfo` Android utilities.

## Configuration
Below, an example configuration can be found.
```json
"profilers": {
"android": {
"sample_interval": 100,
"data_points": ["cpu", "mem"],
"subject_aggregation": "user_subject_aggregation.py",
"experiment_aggregation": "user_experiment_aggregation.py"
}
}
```

**sample_interval** *int*
The sample interval in which the ADB commands are executed and the data points are gathered.

**data_points** *Array<string>*
The types of data that should be measured defined in an array of string enums. Possible options are:
- `cpu` - collects the CPU usage as a percentage of the device's total CPU capacity at a given point in time.
- `mem` - collects the memory usage in KB at a given point in time.

**subject_aggregation** *string*
TODO: default subject aggregation

**experiment_aggregation** *string*
TODO: default experiment aggregation
28 changes: 28 additions & 0 deletions AndroidRunner/Plugins/batterystats/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,28 @@
# Batterystats Plugin
This plugin uses the `batterystats` utility and estimates energy consumption via the algorithm proposed in [this article](https://ieeexplore.ieee.org/stamp/stamp.jsp?arnumber=7884613&casa_token=oEEnY7XOip8AAAAA:AyRZxwboUh55-n9vmW5NGT62mL_hv85T4wPGWlDQGJ36VpF3bcAV1ufvYBhsYxlB0lIMOYJ_Hc-O&tag=1).

## Configuration
Below, an example configuration can be found.
```json
"profilers": {
"batterystats": {
"cleanup": true,
"subject_aggregation": "default",
"experiment_aggregation": "default",
"enable_systrace_parsing": true,
"python2_path": "python2"
}
}
```

**enable_systrace_parsing** *boolean*
The Batterystats profiler uses the profiling tool Systrace internally to measure CPU specific activity and energy consumption on the mobile device. For some devices the parsing of the output of Systrace fails, causing the experiment run to fail. You can safely disable the Systrace parsing when you encounter Systrace parsing errors given that your experiment does not need rely on CPU specific information, but rather on the overall energy consumption of the mobile device. The overall energy consumption is not affected by the Systrace logs since it is tracked using another tool. The default is *true*.

**python2_path** *string*
The path to python 2 that is used to launch Systrace. The default is *python2*.

**subject_aggregation** *string*
TODO: default subject aggregation

**experiment_aggregation** *string*
TODO: default experiment aggregation
23 changes: 23 additions & 0 deletions AndroidRunner/Plugins/frametimes/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,23 @@
# Frametimes Plugin
The frame times plugin gathers unique frame rendering durations (in nanoseconds) by utilizing `dumpsys gfxinfo framestats` and counts the amount of delayed frames that occurred following the 16ms threshold [defined by Google](https://developer.android.com/training/testing/performance).

## Configuration
Below an example of the configuration options is found:
```json
"profilers": {
"Frametimes": {
"subject_aggregation" : "default",
"sample_interval": 1000
}
}
```

**sample_interval** *int*
The sample interval is configurable but advised to keep under 120 seconds as the framestats command returns only data from frames rendered in the past 120 seconds as described [here](https://developer.android.com/training/testing/performance).
Shorter sample intervals will not cause duplication in the frames gathered as only unique frames are kept.

**subject_aggregation** *string*
The default subject aggregation consists of combining both the frametimes as the delayed frames count in single files for easy further processing.

**experiment_aggregation** *string*
This plugin contains no default experiment aggregation.
18 changes: 18 additions & 0 deletions AndroidRunner/Plugins/garbagecollection/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,18 @@
# Garbage Collection Plugin
The garbage collection (GC) plugin gathers and counts GC log statements by searching in adb's logcat for logs that meet the format of a GC call as described [here](https://dzone.com/articles/understanding-android-gc-logs).

## Configuration
Below an example configuration is found:
```json
"profilers": {
"Garbagecollection": {
"subject_aggregation" : "default"
}
}
```

**subject_aggregation** *string*
The default configuration for this plugin is the subject aggregation which lists the counted GC calls in a single file for easy further processing.

**experiment_aggregation** *string*
This plugin contains no default experiment aggregation.
File renamed without changes.
26 changes: 26 additions & 0 deletions AndroidRunner/Plugins/trepn/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,26 @@
# Trepn Plugin
This plugin collects data via the Trepn profiler, e.g., power consumption, battery temperature, CPUs frequency.
The plugin depends on the [Trepn Android app](./com.quicinc.trepn.apk) being installed on the device.

## Configuration
Below an example configuration is found:
```json
"profilers": {
"trepn": {
"sample_interval": 100,
"data_points": ["battery_power", "mem_usage"]
}
}
```

**sample_interval** *int*
The sample interval in which the data points are gathered.

**data_points** *Array<string>*
The types of data that should be measured defined in an array of string enums. Possible options are listed in [data_points.json](./data_points.json).

**subject_aggregation** *string*
TODO: default subject aggregation

**experiment_aggregation** *string*
TODO: default experiment aggregation
File renamed without changes.
69 changes: 21 additions & 48 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,15 @@
[![Build Status](https://travis-ci.org/S2-group/android-runner.svg?branch=master)](https://travis-ci.org/S2-group/android-runner)
[![Coverage Status](https://coveralls.io/repos/github/S2-group/android-runner/badge.svg?branch=master)](https://coveralls.io/github/S2-group/android-runner?branch=master&service=github)
# Android Runner
Automated experiment execution on Android devices
Android Runner (AR) is a tool for automatically executing measurement-based experiments on native and web apps running on Android devices.
As visualized below, it consists of the following components:
- **Experiment orchestrator**: Is in charge of executing the whole experiment according to the experiment configuration provided by the user.
- **Devices manager**: Is responsible for providing a layer of abstraction on the low-level operations involving the Android devices.
- **Progress manager**: Keeps track of the execution of each run of the experiment.
- **Plugin handler**: Provides a set of facilities for managing the profilers and an extension point that third-party developers can use for integrating their own measurement tools into AR.

![Overview of Android Runner](./documentation/overview.jpg)


## Install
This tool is only tested on Ubuntu, but it should work in other linux distributions.
Expand Down Expand Up @@ -143,18 +151,13 @@ The package names of the apps to test when the apps are already installed on the
The names of browser(s) to use. Currently supported values are `chrome`.

**profilers** *JSON*
A JSON object to describe the profilers to be used and their arguments. Below are several examples:
```js
A JSON object to describe the profiler plugins to be used and their arguments. Below, an example is found:
```json
"profilers": {
"trepn": {
"sample_interval": 100,
"data_points": ["battery_power", "mem_usage"]
}
}
```

```js
"profilers": {
},
"android": {
"sample_interval": 100,
"data_points": ["cpu", "mem"],
Expand All @@ -163,40 +166,16 @@ A JSON object to describe the profilers to be used and their arguments. Below ar
}
}
```
Out of the box, AR contains the plugins listed below which can immediately be used as a profiler for an experiment.

```js
"profilers": {
"batterystats": {
"cleanup": true,
"subject_aggregation": "default",
"experiment_aggregation": "default",
"enable_systrace_parsing": true,
"python2_path": "python2"
}
}
```

```json
"profilers": {
"Garbagecollection": {
"subject_aggregation" : "default"
}
}
```
The garbage collection (GC) plugin gathers and counts GC log statements by searching in adb's logcat for logs that meet the format of a GC call as described [here](https://dzone.com/articles/understanding-android-gc-logs).
The default subject aggregation lists the counted GC calls in a single file for easy further processing.
```json
"profilers": {
"Frametimes": {
"subject_aggregation" : "default",
"sample_interval": 1000
}
}
```
The frame times plugin gathers unique frame rendering durations (in nanoseconds) by utilizing `dumpsys gfxinfo framestats` and counts the amount of delayed frames that occurred following the 16ms threshold [defined by Google](https://developer.android.com/training/testing/performance).
The sample interval is configurable but advised to keep under 120 seconds as the framestats command returns only data from frames rendered in the past 120 seconds as described [here](https://developer.android.com/training/testing/performance).
Shorter sample intervals will not cause duplication in the frames gathered as only unique frames are kept.
The default subject aggregation consists of combining both the frametimes as the delayed frames count in single files for easy further processing.
| Name (quality attribute) | Description |
|--------------------------------------------------------------------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| [batterystats](./AndroidRunner/Plugins/batterystats/README.md) (Energy) | Uses the `batterystats` utility and estimates energy consumption via the algorithm proposed in [this article](https://ieeexplore.ieee.org/stamp/stamp.jsp?arnumber=7884613&casa_token=oEEnY7XOip8AAAAA:AyRZxwboUh55-n9vmW5NGT62mL_hv85T4wPGWlDQGJ36VpF3bcAV1ufvYBhsYxlB0lIMOYJ_Hc-O&tag=1). |
| [monsoon](./AndroidRunner/Plugins/monsoon/README.md) (Energy) | Collects energy consumption via the Monsoon hard-ware profiler and the [Physalia tool](https://github.com/TQRG/physalia). |
| [trepn](./AndroidRunner/Plugins/trepn/README.md) (mixed) | Collects data via the Trepn profiler, e.g., power consumption, battery temperature, CPUs frequency. |
| [mem-CPU](./AndroidRunner/Plugins/android/README.md) (Performance) | Collects memory and CPU usage via the `cpuinfo` and `meminfo` Android utilities found in ADB's [dumpsys](https://developer.android.com/studio/command-line/dumpsys). |
| [frametimes](./AndroidRunner/Plugins/frametimes/README.md) (Performance) | Collects frame rendering durations and the number of delayed frames with the technique used in [this article](https://dl.acm.org/doi/pdf/10.1145/2897073.2897100?casa_token=jD3bYLV001kAAAAA:OZiAzZFwtvSO-uK3hgWlz6iNVcTt6uYoT1UWroDEGhDHrEBvLbsIl4E13RhAtRK4IaEPd6putLTzzZw). |
| [gc](./AndroidRunner/Plugins/trepn/README.md) (Performance) | Collects the number of garbage collections as in [this article](https://dl.acm.org/doi/pdf/10.1145/2897073.2897100?casa_token=jD3bYLV001kAAAAA:OZiAzZFwtvSO-uK3hgWlz6iNVcTt6uYoT1UWroDEGhDHrEBvLbsIl4E13RhAtRK4IaEPd6putLTzzZw). |

**subject_aggregation** *string*
Specify which subject aggregation to use. The default is the subject aggregation provided by the profiler. If a user specified aggregation script is used then the script should contain a ```bash main(dummy, data_dir)``` method, as this method is used as the entry point to the script.
Expand All @@ -207,12 +186,6 @@ Specify which experiment aggregation to use. The default is the experiment aggre
**cleanup** *boolean*
Delete log files required by Batterystats after completion of the experiment. The default is *true*.

**enable_systrace_parsing** *boolean*
The Batterystats profiler uses the profiling tool Systrace internally to measure CPU specific activity and energy consumption on the mobile device. For some devices the parsing of the output of Systrace fails, causing the experiment run to fail. You can safely disable the Systrace parsing when you encounter Systrace parsing errors given that your experiment does not need rely on CPU specific information, but rather on the overall energy consumption of the mobile device. The overall energy consumption is not affected by the Systrace logs since it is tracked using another tool. The default is *true*.

**python2_path** *string*
The path to python 2 that is used to launch Systrace. The default is *python2*.

**scripts** *JSON*
A JSON list of types and paths of scripts to run. Below is an example:
```js
Expand Down
Binary file added documentation/overview.jpg
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.

0 comments on commit 2cd49fa

Please sign in to comment.