☁️ 👀 💬 Visual Chatbot
Branch: master
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Type Name Latest commit message Commit time
Failed to load latest commit information.
chat Minor fixes (#7) Apr 16, 2018
misc Initialized Repo Apr 19, 2017
models Update download_models.sh Sep 25, 2018
neuraltalk2 @ bd8c9d8
.gitignore fuller .gitignore that ignores data and model files Jun 23, 2017
README.md Add credit for Abot image Jan 11, 2019
captioning.lua Initialized Repo Apr 19, 2017
evaluate.lua Initialized Repo Apr 19, 2017
manage.py Initialized Repo Apr 19, 2017
prepro_ques.py Initialized Repo Apr 19, 2017
utils.lua Initialized Repo Apr 19, 2017
worker.py Fix the PostgreSQL db connection close problem Aug 18, 2017
worker_captioning.py Initialized Repo Apr 19, 2017


Visual Chatbot


Demo for the paper

Visual Dialog
Abhishek Das, Satwik Kottur, Khushi Gupta, Avi Singh, Deshraj Yadav, José M. F. Moura, Devi Parikh, Dhruv Batra
CVPR 2017 (Spotlight)

Live demo: http://visualchatbot.cloudcv.org

Visual Dialog requires an AI agent to hold a meaningful dialog with humans in natural, conversational language about visual content. Given an image, dialog history, and a follow-up question about the image, the AI agent has to answer the question. Putting it all together, we demonstrate the first ‘visual chatbot’!

Visual Chatbot

Installation Instructions

Installing the Essential requirements

sudo apt-get install -y git python-pip python-dev
sudo apt-get install -y python-dev
sudo apt-get install -y autoconf automake libtool curl make g++ unzip
sudo apt-get install -y libgflags-dev libgoogle-glog-dev liblmdb-dev
sudo apt-get install libprotobuf-dev libleveldb-dev libsnappy-dev libopencv-dev libhdf5-serial-dev protobuf-compiler

Install Torch

git clone https://github.com/torch/distro.git ~/torch --recursive
cd ~/torch; bash install-deps;
source ~/.bashrc

Install PyTorch(Python Lua Wrapper)

git clone https://github.com/hughperkins/pytorch.git
cd pytorch
source ~/torch/install/bin/torch-activate

Install RabbitMQ and Redis Server

sudo apt-get install -y redis-server rabbitmq-server
sudo rabbitmq-plugins enable rabbitmq_management
sudo service rabbitmq-server restart 
sudo service redis-server restart

Lua dependencies

luarocks install loadcaffe

The below two dependencies are only required if you are going to use GPU

luarocks install cudnn
luarocks install cunn

Cuda Installation

Note: CUDA and cuDNN is only required if you are going to use GPU

Download and install CUDA and cuDNN from nvidia website

Install dependencies

git clone https://github.com/Cloud-CV/visual-chatbot.git
cd visual-chatbot
git submodule init && git submodule update
sh models/download_models.sh
pip install -r requirements.txt

If you have not used nltk before, you will need to download a tokenization model.

python -m nltk.downloader punkt

Change lines 2-4 of neuraltalk2/misc/LanguageModel.lua to the following:

local utils = require 'neuraltalk2.misc.utils'
local net_utils = require 'neuraltalk2.misc.net_utils'
local LSTM = require 'neuraltalk2.misc.LSTM'

Create the database

python manage.py makemigrations chat
python manage.py migrate

Running the RabbitMQ workers and Development Server

Open 3 different terminal sessions and run the following commands:

python worker.py
python worker_captioning.py
python manage.py runserver

You are all set now. Visit and you will have your demo running successfully.

Cite this work

If you find this code useful, consider citing our work:

  title={{V}isual {D}ialog},
  author={Abhishek Das and Satwik Kottur and Khushi Gupta and Avi Singh
    and Deshraj Yadav and Jos\'e M.F. Moura and Devi Parikh and Dhruv Batra},
  booktitle={Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition},