Bags of Local Convolutional Features for Scalable Instance Search

[chullhwan-song]

• https://arxiv.org/abs/1604.04653
• ICMR 2016
• https://github.com/imatge-upc/retrieval-2016-icmr

[chullhwan-song]

• Abstract
  • cnn feature를 bag of word 형태로 만드는 방법
  • assignment map
    • fast spatial re-ranking
    • obtaining object localizations that are used for query expansion
  • We demonstrate the suitability of the BoW representation based on local CNN features for instance retrieval
    
• BAG OF WORDS FRAMEWORK
  • pre-trained CNN as local feature
  • discard fc & softmax - conv feature - keeping aspect ratio
  • 1 channel -> a vector -> Bag of Word -> Inverted indexed
  • Bag of word
    • using K-means
    • making assignment map
• IMAGE RETRIEVAL : a spatial re-ranking, and query expansion.
  • initial search - 후보군 서치
    • cosine similarity between the BoW vector of the query image and the BoW vectors of the full images in the database.
      • use a sparse matrix based inverted index, GPU..
    • two types - bounding box의 유무
      • Global search (GS)
      • Local search (LS)
        
  • Local Re-Ranking
    • Top T images 가 대상
    • localization score 기반으로 re-ranking
      • re-ranking을 위한 ROI 구성
        • w= { W, W/2, W/4 } , h = {H, H/2, W/4} : W, H of assignment map 기준으로,
        • sliding window strategy으로 가능한 모든 조합의 크기로 구성
        • assignment map 과 overlap 비율이 50%넘어야함.
        • query의 aspect ratio와 비슷한 것만 선택
      • 구성된 ROI에 대해
        • BoW vector representation 구성하고
        • query's BoW vector representation cosine similarity
        • BoW vector representation
          • spatial pyramid matching
            • vector 값에 대한 loss 감쇠, locality에 대한 감쇠
            • level = 2
              
  • Query Expansion (?)
    • Global query expansion (GQE): The BoW vectors of the N images at the top of the ranking are averaged together with the BoW of the query to form the new representation for the query.
    • Local query expansion (LQE): Locations obtained in the local reranking step are used to mask out the background and build the BoW descriptor of only the region of interest of the N-top images in the ranking. These BoW vectors are averaged together with the BoW of the query bounding box to perform a second search.
    • 다른 논문 읽다가 비슷한 용어가 나오길래.. 다름 아닌 MAC 논문임.
      • "Query expansion (QE). Re-ranking brings positive images at the very top ranked positions. Then, we collect the 5 top-ranked images, merge them with the query vector, and compute their mean. Finally, the similarity to this mean vector is adopted to re-rank once more the top N images."
      • 즉, 일반 결과중에 상위 결과 Top-K 결과가 어느정도 타당성을 가지고 있다면, query+Top-K vector들 평균값을 취해, 새로운 query vector로 사용한다는 의미이다.
      • 참조 paper : "PARTICULAR OBJECT RETRIEVAL WITH INTEGRAL MAX-POOLING OF CNN ACTIVATIONS"
      • 이외 query expansion 알고리즘을 다양하게 존재 - ex) manifold learning
• Experiments
  • instance dataset - landmark data : oxford building, paris building,,,
  • caffe - vgg16: conv5_1, conv5_2, conv5_3 > best : conv5_1
  • PCA - whitening
  • dictionary - 25K