Logic-Enhanced Foundation Model

What’s Left? Concept Grounding with Logic-Enhanced Foundation Models
Joy Hsu*, Jiayuan Mao*, Joshua B. Tenenbaum, Jiajun Wu
In Conference on Neural Information Processing Systems (NeurIPS) 2023

[paper] [project page] [colab]

Colab

Run this colab to train and evaluate LEFT on a new dataset in ~100 lines of code! Additionally, try out this colab to evaluate LEFT on one of our trained domains.

Setup

Run the following commands to setup LEFT.

Make a new conda environment.

  conda create -n left python=3.10
  conda activate left

Install Jacinle.

  git clone https://github.com/vacancy/Jacinle --recursive
  export PATH=<PATH_TO_JACINLE>/bin:$PATH

Install Concepts.

  git clone https://github.com/concepts-ai/Concepts.git
  cd Concepts
  pip install -e .

Install the following libraries.

  pip install PyYAML
  pip install peewee

Note: you may need to install a PyTorch version that has been compiled with your version of the CUDA driver. For example, conda install pytorch==1.11.0 torchvision==0.12.0 torchaudio==0.11.0 cudatoolkit=11.3 -c pytorch.

Demo

Check out this demo notebook to apply LEFT on a new dataset in ~100 lines of code! See the same notebook in colab form as well.

Install the following libraries to run the demo.

  conda install -c conda-forge notebook
  pip install opencv-python

You may need to add the below import in the demo notebook.

  import sys
  sys.path.append("<PATH_TO_JACINLE>")

Train & evaluation

Please see the individual READMEs to train and evaluate models.

Install the following library to train models.

  conda install tensorflow
  pip install chardet

Note: Before running train & eval commands, set export PATH=<PATH_TO_JACINLE>/bin:$PATH.

Warning

LEFT leverages a pre-trained large language model as its language interpreter, and hence, even though our prompts are general examples of first-order logic, we do not have direct control over the LLM's generation. The LLM may output harmful biases.