Skip to content


Repository files navigation


Position Focused Attention Network for Image-Text Matching, which is published in ijcai-2019. The paper can be downloaded from arXiv.


This is the source code of Position Focused Attention Network, an approch for Image-Text Matching based on position attention from Tencent. It is built on top of the SCAN (by Kuang-Huei Lee) in PyTorch.

Requirements and Installation

We recommended the following dependencies.

The workflow of PFAN


Position attention network in PFAN

position attention

Download data

Download the dataset files. We use the dataset files created by SCAN Kuang-Huei Lee. The position information of images can be downloaded from here (for Flickr30K) and here (for MS-COCO). Noting that we only upload the position information and caption in MS-COCO dataset, while the image feature is not uploaded because of its huge storage. The original image feature can be downloaded from SCAN. When using the original image features, we should reorder these samples from the sample ids or sample captions. The Tencent-News dataset files can be downloaded from here and here.

#For Flickr30K dataset
#For MS-COCO dataset
#For Tencent-News training dataset
#For Tencent-News testing dataset

Training new models

To train Flickr30K and MS-COCO models:


In order to further improve the performance of PFAN on Tencent-News dataset, the whole image feautre is also considered. The details are shown in Tencent_PFAN code:

sh Tencent_PFAN/

Arguments used to train Flickr30K models and MS-COCO models are as same as those of SCAN:

For Flickr30K:

Method Arguments
t-i --max_violation --bi_gru --agg_func=Mean --cross_attn=t2i --lambda_softmax=9 --num_epoches=30 --lr_update=15 --learning_rate=.0002 --embed_size=1024 --batch_size=128
i-t --max_violation --bi_gru --agg_func=Mean --cross_attn=i2t --lambda_softmax=4 --num_epoches=30 --lr_update=15 --learning_rate=.0002 --embed_size=1024 --batch_size=128


Method Arguments
t-i --max_violation --bi_gru --agg_func=Mean --cross_attn=t2i --lambda_softmax=9 --num_epoches=30 --lr_update=15 --learning_rate=.0005 --embed_size=1024 --batch_size=128
i-t --max_violation --bi_gru --agg_func=Mean --cross_attn=i2t --lambda_softmax=4 --num_epoches=30 --lr_update=15 --learning_rate=.0005 --embed_size=1024 --batch_size=128

For Tencent-News:

Method Arguments
t-i --max_violation --bi_gru --agg_func=Mean --cross_attn=t2i --lambda_softmax=9 --num_epoches=30 --lr_update=15 --learning_rate=.0002 --embed_size=512 --batch_size=128 --lambda_whole=2
i-t --max_violation --bi_gru --agg_func=Mean --cross_attn=i2t --lambda_softmax=4 --num_epoches=30 --lr_update=15 --learning_rate=.0002 --embed_size=512 --batch_size=128 --lambda_whole=2

The models on Tencent-News can be downloaded from here.

Evaluate trained models on Flickr30K and MS-COCO

from vocab import Vocabulary
import evaluation
evaluation.evalrank("$RUN_PATH/f30k_precomp/model_best.pth.tar", data_path="$DATA_PATH", split="test")

Evaluate position-attention (PFAN-A) and position-only (PFAN-P) models

i2t-1 i2t-5 i2t-10 t2i-1 t2i-5 t2i-10
PFAN-A 70.0 91.8 95.0 50.4 78.7 86.1
PFAN-P 66.0 89.4 94.1 48.6 76.9 85.1

Evaluate trained models on Tencent-News

First, start the server to process requests

sh # port 5091 is sentence model and port 5092 is tag model

Then, send requests to get results from the server

cd test_server
python dist_sentence_t2i.json sentence 5091 # to get the results using sentence model and sentence data
python dist_tag_t2i.json tag 5091 # to get the results using sentence model and tag data
python dist_tag_new_t2i.json tag 5092 # to get the results using tag model and tag data

Finally, get the MAP@1-3 and A@1-3

cd test_server


Position Focused Attention Network for Image-Text Matching






No releases published


No packages published