Seenomaly

Here we show all the GIF examples in our paper

Figure 1 (a)

Figure 1 (b)

Figure 5

Figure 6

Figure 7

Synthetic examples

Here are real-world examples

We can see that synthetic data has the same visual effect complexity with real-world data, and it is actually not easy data, which means synthetic data can evaluate our model.

How to use

Note: This software is tested on tensorflow 1.13 with cuda 9.0 on Titan Xp GPU.

Data

1. Download Rico animation dataset and Rico interaction traces dataset
2. cd datasets.
3. Run python3 get_data.py to extract single frame from GIF, which also contains image processing like cleaning static images.
4. Run python3 get_class.py to get label file, that is label.txt.
5. Run python3 convert_data_to_tfrecord.py which read label.txt and generate tfrecord. Note: please change the file directory accoring to your environment.

Training

Change your own configuration and run finetune.sh.

Extract features

1. Download the model Here.
2. Download the typical examples Here. Note that this is synthetic dataset and only for evaluation.
3. Load the model and run extract_features.py to map typical examples to feature space. After running this, you will obtain features.p file.
4. Run evaluation.py to evalute the model. The KNN search algorithm is also in this file. You can refer to this method to create your applications.

Demo

Here are some GIFs selected randomly from our dataset. Apparently, they distribute at random space.

We extract features using our model

And map the features to a low dimension space

Results

Here shows how the GIFs distribute after clustering. We can see that GIFs with different visual effects distribut at different spaces.

Given a linted GUI, We can use KNN to find the most possible category in this space.

Our model does not simply classify different animation. It can actually distinguish GUIs with the same type of animation but different visual effects, which is demonstrated in this figure:

The two red circles show the same type of animation (pop-up window) but with different visual effects. One is with black background or shadow (1), which means that the pop-up window can be distinguished clearly from the background and thus they are treated as normal examples. The other one contains pop-up windows over a non-shadow background (2), and thus are treated as violation examples.

The two green circles show another example - the animation of bottom sheet over a background with/without shadow. Samples with shadow or over black background (3) are treated as normal examples. But those without shadow or black background (4) violate the Material-Design don't guideline regarding the shadow, and thus are treated as violation examples.

This figure illustrates that our model is able to to distinguish normal and violation GUI animations in the feature space. Note that this visualization is only a 2-dimensional visualization of a 2048-dimensional feature space. In this feature space, if a to-be-linited GUI animation has certain violation, it will be place close to the similar animation vioaltion examples. As a result, k-nearest-neighbors vote can idenitfy the likely violation in the to-be-linited animation.