Skip to content
Automatic art classification with deep learning and knowledge graphs.
Python
Branch: master
Clone or download

Latest commit

Fetching latest commit…
Cannot retrieve the latest commit at this time.

Files

Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
Data
examples
Readme.md
attributes.py
dataloader_kgm.py
dataloader_mtl.py
main.py
model_kgm.py
model_mtl.py
params.py
test.py
train.py
utils.py

Readme.md

Context Embeddings for Art Classification

Pytorch code for the classification part of our ICMR 2019 paper Context-Aware Embeddings for Automatic Art Analysis. For the retrieval part, check this other repository.

Setup

  1. Download dataset from here.

  2. Clone the repository:

    git clone https://github.com/noagarcia/context-art-classification.git

  3. Install dependencies:

    • Python 2.7
    • pytorch (conda install pytorch=0.4.1 cuda90 -c pytorch)
    • torchvision (conda install torchvision)
    • visdom (check tutorial here)
    • pandas (conda install -c anaconda pandas)
    • gensim (conda install -c anaconda gensim)
  4. For the KGM model, download the pre-computed graph embeddings from here, and save the file into the Data/ directory.

Train

  • To train MTL multi-classifier run:

    python main.py --mode train --model mtl --dir_dataset $semart

  • To train KGM classifier run:

    python main.py --mode train --model kgm --att $attribute --dir_dataset $semart

Where $semart is the path to SemArt dataset and $attribute is the classifier type (i.e. type, school, time, or author).

Test

  • To test MTL multi-classifier run:

    python main.py --mode test --model mtl --dir_dataset $semart

  • To test KGM classifier run:

    python main.py --mode test --model kgm --att $attribute --dir_dataset $semart --model_path $model-file

Where $semart is the path to SemArt dataset, $attribute is the classifier type (i.e. type, school, time, or author), and $model-file is the path to the trained model.

You can download our pre-trained models from:

Results

Classification results on SemArt:

Model Type School Timeframe Author
VGG16 pre-trained 0.706 0.502 0.418 0.482
ResNet50 pre-trained 0.726 0.557 0.456 0.500
ResNet152 pre-trained 0.740 0.540 0.454 0.489
VGG16 fine-tuned 0.768 0.616 0.559 0.520
ResNet50 fine-tuned 0.765 0.655 0.604 0.515
ResNet152 fine-tuned 0.790 0.653 0.598 0.573
ResNet50+Attributes 0.785 0.667 0.599 0.561
ResNet50+Captions 0.799 0.649 0.598 0.607
MTL context-aware 0.791 0.691 0.632 0.603
KGM context-aware 0.815 0.671 0.613 0.615

Examples

Paintings with the highest scores for each class:

example example

Citation

@InProceedings{Garcia2017Context,
   author    = {Noa Garcia and Benjamin Renoust and Yuta Nakashima},
   title     = {Context-Aware Embeddings for Automatic Art Analysis},
   booktitle = {Proceedings of the ACM International Conference on Multimedia Retrieval},
   year      = {2019},
}
You can’t perform that action at this time.