Model-based Interactive Semantic Parsing (MISP)

This repository provides code implementations for the EMNLP'19 paper "Model-based Interactive Semantic Parsing: A Unified Framework and A Text-to-SQL Case Study".

Overview

1. Introduction

As a promising paradigm, interactive semantic parsing has shown to improve both semantic parsing accuracy and user confidence in the results. To facilitate its research, we propose Model-based Interactive Semantic Parsing (MISP), a unified framework that views the interactive semantic parsing problem as designing a model-based intelligent agent. The following figures show an overview of MISP and its instantiation (MISP-SQL) for text-to-SQL parsing.

A MISP agent maintains an agent state and has three major components:

World Model, which perceives the environment signals and predicts the future based on the agent's internal knowledge.
Error Detector, which introspects its states and decides whether and where human intervention is needed.
Actuator, which realizes the agent's action in a user-friendly way, e.g., by generating a natural language question.

This repository contains the implementation of MISP-SQL when the base semantic parser is SQLNet (Xu et al., 2017), SQLova (Hwang et al., 2019) or SyntaxSQLNet (Yu et al., 2018). However, the framework potentially can also be applied to other base semantic parsers. We provide two versions of implementations:

Branch emnlp: The original version supporting binary-choice Q&A interaction, which can be used to reproduce our EMNLP results. To use this version, please switch the branch by git checkout emnlp.
Branch multichoice_q (default, recommended): A refactored version supporting multi-choice Q&A interaction. This is the default branch of this repository.

Please cite our work if you use our implementation:

@inproceedings{yao-etal-2019-model,
    title = "Model-based Interactive Semantic Parsing: A Unified Framework and A Text-to-{SQL} Case Study",
    author = "Yao, Ziyu  and
      Su, Yu  and
      Sun, Huan  and
      Yih, Wen-tau",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)",
    month = nov,
    year = "2019",
    address = "Hong Kong, China",
    publisher = "Association for Computational Linguistics",
    url = "https://www.aclweb.org/anthology/D19-1547",
    pages = "5450--5461",
}

2. Agent Architecture

Our MISP-SQL framework is under the folder "MISP_SQL", with agent.py being the implemented MISP agent. The interactive semantic parsing session is implemented in the function interactive_parsing_session. As an overview:

The agent is initially given a SQL query (called hypothesis) generated by the base semantic parser;
The error detector examines (the meta data of) each SQL component (called semantic unit) and decides whether or not to ask the human user a question about this semantic unit.
In every interaction, if the user affirms the current unit, this positive feedback is taken and the parsing continues (Line 90). Otherwise, the agent incoporates the negative feedback, decodes alternative generations for this unit and lets the user decide. This step is realized by conducting "one-step" beam search over the current unit while keeping its preceding fixed (Line 95-103). The agent will generate a new hypothesis after receiving the user answers. The interaction loop then continues from this new hypothesis.

As elaborated above, there are two "cornerstones" supporting the implementation:

Hypthesis: Any (partial) SQL query is an instance of the class MISP_SQL.utils.Hypothesis. A hypothesis mainly stores:
- dec_seq: The sequence of decoding decisions generated while parsing. It allows us to pinpoint previous decisions and make changes (i.e., feedback incorporation). Note that the content in this sequence is specific to the base parser. One can have their own definition as long as it is easy to modify the decisions in feedback incorporation module.
- dec_prefix: The sequence of prefix decoding decisions that the agent enforces the base parser to take. This is used in "one-step" beam search to keep the preceding decisions fixed. Its format is the same as dec_seq.
- tag_seq: The sequence of semantic units generated while parsing. It records every generated SQL component as well as its context, which can be used by the Error Detector to identify potential mistakes and the Question Generator to formulate natural language questions. See MISP_SQL/tag_seq_logic.md for more details.
- logprob and length: The log probability and length of the current (partial) SQL query.
- sql: The generated SQL query in string.
Semantic Unit: A semantic unit defines the minimal unit that the system should interact on. A complete list of semantic units defined in the current framework can be found in MISP_SQL/tag_seq_logic.md.

2.1 Example

NL input: Who are all of the players on the Westchester High School club team?

True SQL: SELECT player WHERE school/club team EQL westchester high school

Generation by SQLNet (without interaction)

Before interaction:
initial SQL: SELECT (Player) WHERE School/Club Team = westchester high

logprob: -0.163793454746

# Note that the following units should follow the framework's definitions.
tag_seq: [('O', ('sc', None), 1.0, None), ('O', 'select', 1.0, None), # dummy tags
('SELECT_COL', (None, u'Player', 0), 0.9999038, 0), # picking a column "Player" (idx 0) in SELECT clause (probability=0.999)
('SELECT_AGG', (None, u'Player', 0), ('none_agg', 0), 0.99758995, 1), # do not pair any aggregator for "Player" (probability=0.998)
('O', 'where', 1.0, None), # dummy tags
('WHERE_COL', (None, u'School/Club Team', 5), 0.99908423, 2), # picking a column "School/Club Team" (idx 5) in WHERE clause
('WHERE_OP', ((None, u'School/Club Team', 5),), ('=', 0), 0.9999919, 3), # picking operator "=" for "School/Club Team"
('WHERE_VAL', ((None, u'School/Club Team', 5),), ('=', 0), ([0, 9, 10, 15], u'westchester high'), 0.8518489589551428, 4)] # WHERE School/Club Team = westchester high

# Note that the following dec_seq is specific to SQLNet.
dec_seq: [(('sc', None), 0), # for 'sc' (SELECT_COL), take column with idx=0 (i.e., "Player")
(('sa', (0, u'Player')), 0), # for 'sa' (SELECT_AGG), take aggregator with idx=0 (i.e., "none_agg", empty aggregator)
(('wc', None), 1, [5]), # for 'wc' (WHERE_COL), decide the number of conditions #cols=1 and take column with idx=5 (i.e., "School/Club Team")
(('wo', (5, u'School/Club Team')), 0), # for 'wo' (WHERE_OP) corresponding to column "School/Club Team", take operator with idx=0 (i.e., "=")
(('wv', (5, u'School/Club Team', 0, '=')), [0, 9, 10, 15])] # for 'wv' (WHERE_VAL) corresponding to "WHERE School/Club Team =", generate a sequence of words: <SOS>, 'westchester', 'high', <EOS>

# --------------------------------------------------
initial evaluation:
qm correct: 0.0, exe correct: 0.0

Interaction using MISP-SQL with SQLNet

# The agent decides to validate the WHERE_VAL unit "westchester high".
SU: [('WHERE_VAL', ((None, u'School/Club Team', 5),), ('=', 0), ([0, 9, 10, 15], u'westchester high'), 0.8518489589551428, 4)]
Question: The system considers the following condition: the table attribute "School/Club Team" equals to "westchester high". Is the condition correct?
User answer: no.

# The agent decodes alternative WHERE_VAL while keeping its preceding decisions the same as before. 
# This is done by setting dec_prefix=[(('sc', None), 0), (('sa', (0, u'Player')), 0), (('wc', None), 1, [5]), (('wo', (5, u'School/Club Team')), 0)] 
# and conducting one-step beam search on the next decoding step 'wv'.
Question: Please select any options from the following list that the system needs to consider as conditions:
(1) the table attribute "School/Club Team" equals to "westchester high school";
(2) the table attribute "School/Club Team" equals to "westchester high westchester";
(3) the table attribute "School/Club Team" equals to "westchester high high";
(4) None of the above options.
User answer: 1.

# --------------------------------------------------
After interaction:
final SQL: SELECT (Player) WHERE School/Club Team = westchester high school

logprob: -1.91301941393

tag_seq: [('O', ('sc', None), 1.0, None), ('O', 'select', 1.0, None), 
('SELECT_COL', (None, u'Player', 0), 0.9999038, 0), 
('SELECT_AGG', (None, u'Player', 0), ('none_agg', 0), 0.99758995, 1), 
('O', 'where', 1.0, None), ('WHERE_COL', (None, u'School/Club Team', 5), 0.99908423, 2), 
('WHERE_OP', ((None, u'School/Club Team', 5),), ('=', 0), 0.9999919, 3), 
('WHERE_VAL', ((None, u'School/Club Team', 5),), ('=', 0), ([0, 9, 10, 11, 15], u'westchester high school'), 0.14814377778482996, 4)]

dec_seq: [(('sc', None), 0), (('sa', (0, u'Player')), 0), (('wc', None), 1, [5]), (('wo', (5, u'School/Club Team')), 0), (('wv', (5, u'School/Club Team', 0, '=')), [0, 9, 10, 11, 15])]

final evaluation:
qm correct: 1.0, exe correct: 1.0

2.2 How to apply MISP-SQL to other base semantic parsers?

Basically, one can simply inherit the abstract classes defined in MISP_SQL. We use SQLNet as an example and summarize our coding work below:

world_model.py:
- def decode_per_pass: To support (one-step) beam search, we adapt the SQLNet's non-interactive forward implementation (Line 129) into interactive beam search (Line 184). A customized hypothesis inherited from the MISP_SQL.utils.Hypothesis class is used in beam search.
- def apply_pos_feedback, def apply_neg_feedback and def decode_revised_structure are implemented to support feedback incorporation.
environment.py:
- class ErrorEvaluator(BaseErrorEvaluator): This class is inherited from the MISP_SQL.environment.ErrorEvaluator class. One needs to implement the function compare to simulate the user's feedback (given she knows the ground truth, simply comparing the generated query with the ground truth).
- class RealUser(BaseRealUser): This class is inherited from the MISP_SQL.environment.RealUser class. We extend it with a function show_table to present table headers in user study.

Other classes, such as MISP.QuestionGenerator, MISP.ErrorDetectorProbability and MISP.Agent, can be directly used.

3. Experiments

Our experiments involve three different base semantic parsers: SQLNet, SQLova and SyntaxSQLNet. This section shows how to run MISP with each of the base semantic parsers.

3.1 MISP with SQLNet

Requirements

Python 2.7
Pytorch 0.2.0
Please follow SQLNet's instruction to install other dependencies.

Note: It is recommended to create a conda environment (download, usage) to install the above packages. If you do so, you may put your environment name here, i.e., source activate ENV_NAME.

Data preparation

Please download the .tar file from here and uncompress it under the folder SQLNet_model. This file has included necessary data as well as pretrained models.

To run the interactive system

Under the main directory (please revise the output paths accordingly before running the script):

bash interaction_SQLNet.sh

3.2 MISP with SQLova

Requirements

Python 3.6
Pytorch 1.0.1
Please follow SQLova's instruction to install other dependencies. Note: If you have created a new conda environment to install the above packages, please put its name here, i.e., source activate ENV_NAME.

Data preparation

Please download the .tar file from here and uncompress it under the folder SQLova_model. This file has included necessary data as well as pretrained models.

To run the interactive system

Under the main directory (please revise the output paths accordingly before running the script):

bash interaction_sqlova.sh

3.3 MISP with SyntaxSQLNet

Requirements

Python 2.7
Pytorch 0.2.0

(You can use the same environment as SQLNet for running SyntaxSQLNet).

Data preparation

Please download the .tar file from here and uncompress it under the folder syntaxSQL. This file has included necessary data as well as pretrained models. Please also download and unzip the pretrained Glove, and put it as syntaxSQL/glove/glove.42B.300d.txt.

To run the interactive system

Under the main directory (please revise the output paths accordingly before running the script):

bash interaction_syntaxSQL.sh

4. Acknowledgement

The implementations of MISP-SQL applied to SQLNet, SQLova and SyntaxSQLNet are adapted from their non-interactive implementations:

SQLNet
SQLova
SyntaxSQLNet

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

README.md

README.md

Model-based Interactive Semantic Parsing (MISP)

Overview

1. Introduction

2. Agent Architecture

2.1 Example

Generation by SQLNet (without interaction)

Interaction using MISP-SQL with SQLNet

2.2 How to apply MISP-SQL to other base semantic parsers?

3. Experiments

3.1 MISP with SQLNet

Requirements

Data preparation

To run the interactive system

3.2 MISP with SQLova

Requirements

Data preparation

To run the interactive system

3.3 MISP with SyntaxSQLNet

Requirements

Data preparation

To run the interactive system

4. Acknowledgement

Files

README.md

Latest commit

History

README.md

File metadata and controls

Model-based Interactive Semantic Parsing (MISP)

Overview

1. Introduction

2. Agent Architecture

2.1 Example

Generation by SQLNet (without interaction)

Interaction using MISP-SQL with SQLNet

2.2 How to apply MISP-SQL to other base semantic parsers?

3. Experiments

3.1 MISP with SQLNet

Requirements

Data preparation

To run the interactive system

3.2 MISP with SQLova

Requirements

Data preparation

To run the interactive system

3.3 MISP with SyntaxSQLNet

Requirements

Data preparation

To run the interactive system

4. Acknowledgement