Beginner's friendly of PyTorch Implementation for 'Learning Visual Importance for Graphic Designs and Data Visualizations'
Knowing where people look and click on visual designs can provide clues about how the designs are perceived, and where the most important or relevant content lies. The most important content of a visual design can be used for effective summarization or to facilitate retrieval from a database. We present automated models that predict the relative importance of different elements in data visualizations and graphic designs. Our models are neural networks trained on human clicks and importance annotations on hundreds of designs. We collected a new dataset of crowdsourced importance, and analyzed the predictions of our models with respect to ground truth importance and human eye movements. We demonstrate how such predictions of importance can be used for automatic design retargeting and thumbnailing. User studies with hundreds of MTurk participants validate that, with limited post-processing, our importance-driven applications are on par with, or outperform, current state-of-the-art methods, including natural image saliency. We also provide a demonstration of how our importance predictions can be built into interactive design tools to offer immediate feedback during the design process. Article link
Easy way to try
- Download pretrained models and put in:
(all models in one zip file, ~2Gb:
- Check libraries:
- Paste PNG/JPG file (only RGB) in:
- Get the result in:
If you take the png with the alpha channel (RGBA), then an error will occur.
Solution: convert PNG to JPG or make a flatten PNG.
The best result gives
- need a python conversion module for png RGBA to RGB flatten
- help to add Python code to modify the asset bitmap files and not one by one.
- help writing a python module in order to use GPU (nvidia) resources and not CPU
- tool and algorithm for markup of graphic design samples, GUI and data visualisation for train FCN
- way to train FCN in two datasets of good and bad visual design.
- way to compare the usability test results and the visual importance prediction results to ensure the quality of the prediction and understand the discrepancies with reality
- big goal is to teach a neural network to predict good and bad design by making it a plug-in for prototyping tools such as a figma or sketch
Thanks a lot to for my mentor Lena Sukmanova for helping to launch the algorithm.
Machine Learning for Everyone. Link
Learning Visual Importance for Graphic Designs and Data Visualizations. Link
Learning Layouts for Single-Page Graphic Designs. Link
Predicting Attention for Design Feedback. Link
19 Things We Can Learn From Numerous Heatmap Tests. Link
What are Web Heatmaps, Really? (And the 4 Major Problems of Heat Mapping Tools). Link
What We Can Learn From Heat Map Studies to Improve Our Calls-to-Action. Link
Машинное обучение для людей. Link
Карта внимания пользователей для дизайнеров (разные точки зрения в комментариях). Link
19 уроков на основании анализа тепловых карт и айтрекинга. Link
Тепловые карты в контексте веб-дизайна. Link
Как улучшить Calls-to-Action с помощью тепловых карт. Link
Original source. Below README from original source:
PyTorch Implementation for 'Learning Visual Importance for Graphic Designs and Data Visualizations'
This repo implements training and testing models for  by PyTorch, and based on Caffe codes authors of  provide.
To download datasets, GDI (Graphic Design Importance) Dataset and Massvis (Visualization) Dataset, see authors' repo.
python visimportance.py --dataset <dataset_name> # name of dataset, gdi or massvis (default: gdi) --dataset_dir <directory_path> # dataset directory --fcn_type <fcn_type> # FCN type, fcn32 or fcn16 (default: gdi) --overlaid_img_dir <directory_path> # output directory path for images with heatpmap overlaid onto input images --pretrained_model <file_path> # pretrained model converted from Caffe models --gpu <device_id> # GPU id (default: 0) --eval_only # evaluation only
The followings are PyTorch pretrained models. Specify the file path to
gdi_fcn32.pthFCN-32s model for GDI (Graphic Design Importance) Dataset, link
gdi_fcn16.pthFCN-16s model for GDI (Graphic Design Importance) Dataset, link
massvis_fcn32.pthFCN-32s model for Massvis (Visualization) Dataset, link
These models are converted from Caffe models authors of  provide.
python visimportance.py --dataset <dataset_name> # name of dataset, gdi or massvis (default: gdi) --dataset_dir <directory_path> # dataset directory --fcn_type <fcn_type> # FCN type, fcn32 or fcn16 (default: gdi) --overlaid_img_dir <directory_path> # output directory path for images with heatpmap overlaid onto input images --pretrained_model <file_path> # pretrained model converted from Caffe models --gpu <device_id> # GPU id (default: 0) --resume <file_path> # checkpoint file to be loaded when retraining models --checkpoint_dir <directory_path> # checkpoint file to be saved in each epoch --config <configuration_id> # configuration for training where several hyperparameters are defined
The followings are PyTorch VOC FCN-32s pretrained model.
Specify the file path to
--pretrained_model option when you newly start to train models for GDI/Massvis datasets.
voc_fcn32.pthFCN-32s model for GDI (Graphic Design Importance) Dataset, link
This model is converted from Caffe models authors of  provide.
- Ground truth
- Predicted by FCN-16s
- Ground truth
- Predicted by FCN-32s
- Zoya Bylinskii and Nam Wook Kim and Peter O'Donovan and Sami Alsheikh and Spandan Madan and Hanspeter Pfister and Fredo Durand and Bryan Russell and Aaron Hertzmann, Learning Visual Importance for Graphic Designs and Data Visualizations, Proceedings of the 30th Annual ACM Symposium on User Interface Software & Technology, 2017. authors' site, arXiv