Support for more question types

[albertgong1]

Here are the question templates we currently support (generated from our QA_GRAMMAR_STRING at depth=10):

#0: Who is the <relation>_3 of the <relation>_5 of the <relation>_7 of the person whose <attribute_name>_9 is <attribute_value>_9 ?
#1: Who is the <relation>_3 of the <relation>_5 of the <relation>_7 of <name>_8 ?
#2: Who is the <relation>_3 of the <relation>_5 of the person whose <attribute_name>_7 is <attribute_value>_7 ?
#3: Who is the <relation>_3 of the <relation>_5 of <name>_6 ?
#4: Who is the <relation>_3 of the person whose <attribute_name>_5 is <attribute_value>_5 ?
#5: Who is the <relation>_3 of <name>_4 ?
#6: Who is the person whose <attribute_name>_3 is <attribute_value>_3 ?
#7: What is the <attribute_name>_3 of the <relation>_4 of the <relation>_6 of the <relation>_8 of <name>_9 ?
#8: What is the <attribute_name>_3 of the <relation>_4 of the <relation>_6 of the person whose <attribute_name>_8 is <attribute_value>_8 ?
#9: What is the <attribute_name>_3 of the <relation>_4 of the <relation>_6 of <name>_7 ?
#10: What is the <attribute_name>_3 of the <relation>_4 of the person whose <attribute_name>_6 is <attribute_value>_6 ?
#11: What is the <attribute_name>_3 of the <relation>_4 of <name>_5 ?
#12: What is the <attribute_name>_3 of the person whose <attribute_name>_4 is <attribute_value>_4 ?
#13: How many <relation_plural>_2 does the <relation>_4 of the <relation>_6 of the <relation>_8 of <name>_9 have?
#14: How many <relation_plural>_2 does the <relation>_4 of the <relation>_6 of the person whose <attribute_name>_8 is <attribute_value>_8 have?
#15: How many <relation_plural>_2 does the <relation>_4 of the <relation>_6 of <name>_7 have?
#16: How many <relation_plural>_2 does the <relation>_4 of the person whose <attribute_name>_6 is <attribute_value>_6 have?
#17: How many <relation_plural>_2 does the <relation>_4 of <name>_5 have?
#18: How many <relation_plural>_2 does the person whose <attribute_name>_4 is <attribute_value>_4 have?
#19: How many <relation_plural>_2 does <name>_3 have?

Here are some questions we DON'T yet support?

1. How are X and Y related? (from CLUTRR paper)
2. How many people have a X hobby?

See also #40 , #38 , #37

[albertgong1]

Comparison to HotpotQA

Paper: https://arxiv.org/abs/1809.09600
Explorer: https://hotpotqa.github.io/explorer.html

Note: Multi-hop reasoning is defined as "the ability to reason with information taken from more than one paragraph" (Roy & Anand, p. 121).

Question types:

1. chain join over 2 variables:

when was the singer and songwriter of Radiohead born?

One would need to first reason that the “singer and songwriter of Radiohead” is “Thom Yorke”, and then figure out his birthday in the text. We call “Thom Yorke” a bridge entity in this example. Given an edge (a, b) in the hyperlink graph G¹, the entity of b can usually be viewed as a bridge entity that connects an and b.
Graphically, we can represent this question as

[Radiohead] -(singer and songwriter of)-> [Y] -(date of birth)-> [X]

Answer format: Free-form (note: X is an entity or literal.)
It's unclear whether the authors considered multiple-choice format.

2. comparison over 2 variables:

• Answer format: Free-form

Who has played for more NBA teams, Michael Jordan or Kobe Bryant?

• Answer format: Yes/no multiple-choice

Are Iron Maiden and AC/DC from the same country?

Footnotes

1. (edge (a, b) indicates there is a hyperlink from the first paragraph of article a to article b) ↩

[albertgong1]

@bogoliubon Does this look right to you?

[albertgong1]

Comparison to MultiHop-RAG

Paper: https://arxiv.org/abs/2401.15391
Dataset: https://huggingface.co/datasets/yixuantt/MultiHopRAG
Code: https://github.com/yixuantt/MultiHop-RAG/

1. star join over N variables ("Inference query")

Which report discusses the supply chain risk of Apple, the 2019 annual report or the 2020 annual report?

These queries are formulated by synthesizing the various characterizations of the bridge-entity across multiple claims, with the final answer being the identification of the entity itself. See also Table 8.
❓ What is the typical value of N?

2. comparison over two variables ("Comparison query")

Did Netflix or Google report higher revenue for the year 2023?

These queries are structured to compare the similarities and differences related to the bridge entity or topic. The resultant answer to such queries is typically a definitive “yes” or “no”, based on the comparison.

3. temporal intent ("Temporal query")

Did Apple introduce the AirTag tracking device before or after the launch of the 5th generation iPad Pro?

These queries explore the temporal ordering of events across different points in time. The answer to such queries is typically a “yes” or “no” or a single temporal indicator word like “before” or “after”.

4. "Null query"

What are the sales of company ABCD as reported in its 2022 and 2023 annual reports?

Null query is a query whose answer cannot be derived from the retrieved set. To create null queries, we generate multi-hop queries using entities that do not exist in the existing bridge-entities. To add complexity, we also include fictional news source metadata when formulating these questions, ensuring that the questions do not reference any contextually relevant content from the knowledge base. The answer to the null query should be “insufficient information” or similar.

[albertgong1]

Comparison to CLUTRR: https://arxiv.org/abs/1908.06177