Detected non-deterministic results under various configurations

[AnnabellaM]

Hi, I have recently been using Amandroid for an empirical study to detect non-deterministic behaviors in static analyzers. The experiments resulted in discovering some nondeterministic analysis results across multiple runs under various configurations of Amandroid.

The details of the experimental setup are as below:

• The experiments were conducted on the micro-benchmark DroidBench and a real-world benchmark FossDroid.

• The experiments were conducted under 25 sample configurations which were generated using a 2-way covering array from the configuration space.

• The timeout set for Amandroid running on each DroidBench program was 5 minutes. For running on each FossDroid program, the timeout was set to 1 hours.

• We ran Amandroid on each program-configuration combination 5 times and compared the results across 5 runs for detecting non-deterministic behaviors.

• All experiments were conducted in docker containers. The hardware environment is a server with 128GB of RAM and 24 Intel Xeon Silver 4116 CPUs@2.10GHz running Ubuntu 16.04.

In the end, the experiments detected non-deterministic results on 2 programs. None of these programs were from the DroidBench, and all programs were from the FossDroid. These results were detected under 2 configurations out of a total of 25 sample configurations. The two configurations both set the module option to DATA_LEAKAGE and the approach option to COMPONENT_BASED

The attached data is the detected nondeterministic results from DroidBench and FossDroid and configuration files
(note1: the configurations are hash-coded in the detected results, but the actual configuration options and values that each hash code stands for can be found in the attached configuration files.)