DDT: Deep Descriptor Transforming for Image Retrieval


DESCRIPTIONS
    This repository includes a Python implementation of DDT, which is a simple
    yet effective unsupervised approach, for image retrieval. A full
    description of this work can be found in the paper.

    The idea of DDT is that, the convolutional activations could act as a
    detector for the common object. DDT evaluates the correlations of
    descriptors and then obtaining the category-consistent regions, which
    accurately locate the common object in a set of images.

    Different from DDT, SCDA assumes only an object of interest in each image,
    and without objects from other categories. SCDA's strategy is based on the
    activation tensor of a single image. Apparently, SCDA cannot work well for
    images containing diverse objects, and also cannot handle data noise.

    What distinguishes DDT from SCDA is that we can leverage the correlations
    beneath the whole image set, instead of a single image. DDT utilizes PCA
    as projection directions for transforming column descriptors to evaluate
    their correlations.

    In DDT, we compute PC1 for all the column descriptors in the dataset. The
    PC1 projections reflect the correlations of these column descriptors. The
    positive correlation indicate the common characteristic throughout the
    dataset. Specifically, the corresponding positive correlation indicates
    indeed the common object inside these images; the negative correlation
    presents background objects rarely appear.

    Note that we do not collect the largest connected component, which is
    different from the DDT paper. We use the PC1 projections as spatial weights
    for weighted sum-pooling.

    We learn the PCA whitening parameters from a separate set of images. To be
    comparable with related works, we learn the PCA whitening parameters on
    Oxford when testing on Paris and vice versa, as accustomed. We use the
    Oxford100k dataset for whitening on the Holidays.

    We can use cropped images or full images for query. When we use cropped
    queries to evaluate, we regard the cropped region as input for the CNN and
    extract featrures.


REFERENCE
    X.-S. Wei, C.-L. Zhang, Y. Li, C.-W. Xie, J. Wu, C. Shen, and Z.-H. Zhou.
    Deep Descriptor Transforming for Image Co-Localization. In Proceedings of
    the International Joint Conference on Artificial Intelligence, pages
    3048--3054, 2017.


PREREQUIREMENTS
    Caffe with Python interface supported
    Python3.6 with Numpy, PIL, sklearn supported


LAYOUT
   Data
    ./data/oxford5k/
        ./data/oxford5k/conv/            # Convolution feature maps
            ./data/oxford5k/conv/vgg16/all/
            ./data/oxford5k/conv/vgg16/crop/
            ./data/oxford5k/conv/vgg19/all/
            ./data/oxford5k/conv/vgg19/crop/
        ./data/oxford5k/groundtruth/     # 55*4=220 query groundtruth
        ./data/oxford5k/image/           # Raw .jpg images
            ./data/oxford5k/image/all/   # 5063 database images
            ./data/oxford5k/image/crop/  # 55 crop query images
        ./data/oxford5k/descriptor_model.pkl  # PC1 model for DDT
    ./data/paris6k/
        # Similar to oxford5k, with 6392 database and 55 query images



USAGE
    Step 1. Extract crop query images and extract relu5-4 features.
        $ ./src/get/get_query.py --dataset oxford5k
        $ ./src/get/get_query.py --dataset paris6k

        # Extract pool5 features.
        $ ./src/get/get_conv.py --dataset oxford5k --model vgg19 --gpu 0
        $ ./src/get/get_conv.py --dataset paris6k --model vgg19 --gpu 0

    Step 2. Compute DDT descriptors and evaluate.
        # Compute DDT weights.
        $ ./src/ddt/ddt.py --dataset oxford5k --model vgg19
        $ ./src/ddt/ddt.py --dataset paris6k --model vgg19

        # Compute mAP.
        $ ./evaluate.py --test oxford5k --via paris6k --model vgg19
        $ ./evaluate.py --test paris6k --via oxford5k --model vgg19


AUTHOR
    Hao Zhang: zhangh0214@gmail.com


COPYRIGHT
    This repository is free for academic usage. You can run it at your own
    risk.


NOTE
    20 images out of 6412 Paris6k images are broken. As a common practice,
    we manually removed them:
        paris_louvre_000136.jpg
        paris_louvre_000146.jpg
        paris_moulinrouge_000422.jpg
        paris_museedorsay_001059.jpg
        paris_notredame_000188.jpg
        paris_pantheon_000284.jpg
        paris_pantheon_000960.jpg
        paris_pantheon_000974.jpg
        paris_pompidou_000195.jpg
        paris_pompidou_000196.jpg
        paris_pompidou_000201.jpg
        paris_pompidou_000467.jpg
        paris_pompidou_000640.jpg
        paris_sacrecoeur_000299.jpg
        paris_sacrecoeur_000330.jpg
        paris_sacrecoeur_000353.jpg
        paris_triomphe_000662.jpg
        paris_triomphe_000833.jpg
        paris_triomphe_000863.jpg
        paris_triomphe_000867.jpg