Questions about annotations and annotation parser.

[yubinnng]

Thanks for your innovative works!

I had some questions while reading the code, and wish for your help.

---

I will use the image(ID is 32367) and its annotations (from Trainval_GT_all_part.pkl and Trainval_Neg_all_part.pkl) as an example.

Data/hico_20160224_det/images/train2015/HICO_train2015_00032367.jpg:

Q1: HOI does not match the object class

The image has one GT annotation in effect, others are augmented to Pos_augment(default 15).
Its GT HOI ID is : 201, detail: (201, apple, eat) (refer to list of HOIs)

However, Its GT object80 ID is 47, detail: (047 orange) (refer to list of objects)

For example in trainval_GT[32367][0]:

Q2: What the meaning of using 57(corresponding to move) and using Neg[4]-1 in annotation parser of Neg.

HAKE-Action/lib/ult/ult_HICO_DET.py

Line 352 in 1771c22

action_verb.append(Generate_action_PVP(57, 117))

HAKE-Action/lib/ult/ult_HICO_DET.py

Line 360 in 1771c22

gt_object.append(Generate_action_object(Neg[4] - 1, 80))

Q3: I am still confused about negative examples.

After reading the explanation, I'am not sure whether my comprehension is correct:

GT samples is the samples with interaction and Neg without interaction.
Neg samples gave some wrong HOIs and using Generate_action_PVP(-1, N) to mark these HOIs are wrong.

For example in Trainval_Neg[32367]:

The women is reading the book(corresponding to HOI (251, book, read)), but GT doesn't has the HOI and Neg has(in Trainval_Neg[32367][2])

Why marked this HOI to Neg instead of GT?

Q4: Annotation format

Although I can speculate most of the format of Trainval_Neg and trainval_GT, I wonder if I could learn the official definition about the formats of these annotation files in some where.

[Foruck]

For Q1, note that we use a different object index from that provided in the original HICO-DET dataset and in the GT file, and we use 0-based index in GT file. Therefore, your could refer to this file where the index is 1-based, and the ambiguity could be solved.
For Q2, we use 0-based index for verb, thus 57 here is actually 058 no_interaction in the official verb list. And for Neg[4]-1, this is because we use 0-based object index in GT and 1-based object index in Neg, and here the 1-based index is transformed to 0-based index.
For Q3, we also use 0-based index for HOI. Therefore, 251 here actually corresponds to 252 book no_interaction in the official hoi list. And this pair is marked as Neg because it is not provided in the official annotation. In other word, this is a missed annotation. Due to the mechanism we and previous works used to generate the negative samples, it is hard to distinguish the 'real' negative pairs from missed annotated pairs in an automatic way. Therefore, currently we choose to omit the potential noise introduced by negative sample generation.
For Q4, we will add description about the annotation format asap.

[yubinnng]

Thanks for your answer! It was truly my fault that I didn't aware the obvious cause, I will analyze carefully before I post the issue in the future. Sorry for bother you.