listen-attend-and-walk

Code for converting natural language instructions into its corresponding action sequence. You can find my PyCon talk on this topic here.

We have followed two different approches to solve this problem

First approach : Using Multi-level aligner based model

• This uses both high-level and low level representation of the input sentence to calculate the context vector. Refer this paper for the complete architecture.

• PyCon talk explains this approach.

• PyTorch implementation of this approach can be found in MultiLevelAlignerBasedModel folder.

• Scratch code for this approach can be found in CodeFromScratch folder.

Second approch : Using Global attention based model

• This only uses the high level representation of the input sentence to calculate the context vector. Refer this paper for the complete architecture.

• PyTorch implementation of this approach can be found in GlobalAttentionBasedModel folder.

AnotherApproach folder contains keras implementation. This approach is still in progress, any sort of contribution in building the custom keras layer for multi-level aligner is most welcome.

Contributers

This repository is created and maintained by

• Padmaja Bhagwat
• Manisha Jhawar
• Nitya C K