This project aims to convert human language-defined tasks into a sequence of actions understandable by a robot. The inspiration comes from the Facebook AI Habitat Challenge.
Please follow the steps below to set up the required dependencies for the project:
Make sure to install the following dependencies:
-
Speech Recognition Module
pip install SpeechRecognition==3.10.1
-
PortAudio19-dev
sudo apt-get install portaudio19-dev
-
PyAudio
pip install pyaudio==0.2.14
-
NLTK (Natural Language Toolkit)
pip install nltk
-
TensorFlow
pip install tensorflow
-
Scikit-learn
pip install scikit-learn
Language is something we humans use to connect with each other to express emotions/opinions, to delegate tasks. It’s almost impossible for a human to give another a JSON file to execute something, so language is the primary aspect of everyday life. Our project inspiration started with how we can convert a task defined in Human language can be converted to a set of sequences of actions that can be understood by a Robot. The project is inspired by the FacebookAI Habitat Challenge Object Nav theme. The objective is to develop agents that can navigate unfamiliar environments and move away from closed object classes towards open-vocabulary natural language. So the challenge is for a robot (virtual/real) when placed in an unknown environment should be able to navigate the environment by using information from its State space (From IMU + GPS/Encoders) and Image (visual cues) information.
- Download the Project Voice2Nav and extract it into habitat-sim folder
- Run the python script
Scene no : 1-6
python main.py -s <SCENE_NO>
- Note that we have provided some sample scenes we used from the Matterport HM3D Data after getting access from them. For more scenes please fill out their application for Access and download and use. The example scenes we have given in this repo are purely for ACADEMIC USE.
- For more understanding please email us.