[pull] main from seonghyeonye:main

.gitignore

@@ -0,0 +1,142 @@
+# data
+data
+data/
+beaker_configs/ni_*.yaml
+backup/
+
+# Byte-compiled / optimized / DLL files
+__pycache__/
+*.py[cod]
+*$py.class
+
+# C extensions
+*.so
+
+# large file
+cc_news_text_sents.json
+
+# Distribution / packaging
+.Python
+build/
+develop-eggs/
+dist/
+downloads/
+eggs/
+.eggs/
+lib/
+lib64/
+parts/
+sdist/
+var/
+wheels/
+pip-wheel-metadata/
+share/python-wheels/
+*.egg-info/
+.installed.cfg
+*.egg
+MANIFEST
+
+# PyInstaller
+#  Usually these files are written by a python script from a template
+#  before PyInstaller builds the exe, so as to inject date/other infos into it.
+*.manifest
+*.spec
+
+# Installer logs
+pip-log.txt
+pip-delete-this-directory.txt
+
+# Unit test / coverage reports
+htmlcov/
+.tox/
+.nox/
+.coverage
+.coverage.*
+.cache
+nosetests.xml
+coverage.xml
+*.cover
+*.py,cover
+.hypothesis/
+.pytest_cache/
+
+# Translations
+*.mo
+*.pot
+
+# Django stuff:
+*.log
+local_settings.py
+db.sqlite3
+db.sqlite3-journal
+
+# Flask stuff:
+instance/
+.webassets-cache
+
+# Scrapy stuff:
+.scrapy
+
+# Sphinx documentation
+docs/_build/
+
+# PyBuilder
+target/
+
+# Jupyter Notebook
+.ipynb_checkpoints
+
+# IPython
+profile_default/
+ipython_config.py
+
+# pyenv
+.python-version
+
+# pipenv
+#   According to pypa/pipenv#598, it is recommended to include Pipfile.lock in version control.
+#   However, in case of collaboration, if having platform-specific dependencies or dependencies
+#   having no cross-platform support, pipenv may install dependencies that don't work, or not
+#   install all needed dependencies.
+#Pipfile.lock
+
+# PEP 582; used by e.g. github.com/David-OConnor/pyflow
+__pypackages__/
+
+# Celery stuff
+celerybeat-schedule
+celerybeat.pid
+
+# SageMath parsed files
+*.sage.py
+
+# Environments
+.env
+.venv
+env/
+venv/
+ENV/
+env.bak/
+venv.bak/
+
+# Spyder project settings
+.spyderproject
+.spyproject
+
+# Rope project settings
+.ropeproject
+
+# mkdocs documentation
+/site
+
+# mypy
+.mypy_cache/
+.dmypy.json
+dmypy.json
+
+# Pyre type checker
+.pyre/
+output/*
+cache/*
+wandb/*
+src/rouge/testdata/*

Dockerfile

@@ -0,0 +1,26 @@
+FROM pytorch/pytorch:1.10.0-cuda11.3-cudnn8-devel
+
+ENV LC_ALL=C.UTF-8
+ENV LANG=C.UTF-8
+ENV CUDA_HOME=/usr/local/cuda/
+
+RUN apt-get -y update
+RUN apt-get -y install git vim 
+
+RUN curl -s https://packagecloud.io/install/repositories/github/git-lfs/script.deb.sh | bash
+RUN apt-get -y install git-lfs
+
+WORKDIR /stage/
+
+COPY requirements.txt .
+RUN pip install -r requirements.txt
+RUN python -c "import nltk; nltk.download('punkt', quiet=True)"
+
+COPY ds_configs ds_configs
+
+COPY src src
+COPY scripts scripts
+RUN chmod +x scripts/*
+
+# for interactive session
+RUN chmod -R 777 /stage/

LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2022 Yizhong Wang
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

README.md

@@ -1,8 +1,84 @@
-# In-Context Instruction Learning
+# In-Context Instruction Learning (ICIL)
+This is the official github repository for [In-Context Instruction Learning](https://arxiv.org/abs/2302.14691).
 
-This repository includes the code for experiments done in "In-Context Instruction Learning" paper.
 
-We will upload the code soon. Stay tuned!
+Overview of In-Context Instruction Learning (ICIL)
 
+<p align="center">
+  <img src="./ICIL.png" width="80%" height="80%">
+</p>
 
-<img width="2136" alt="image" src="https://user-images.githubusercontent.com/50025266/221876653-9af07426-0bba-45c1-ad49-0b18508f784a.png">
+
+## Dataset
+For evaluation dataset, we used SuperNatural-Instructions, which can be assessed in the official [repo](https://github.com/allenai/natural-instructions). Simply clone the repo under this directory and change the directory name to `data`. We run our main experiments on the 119 tests tasks designated by this official repository. We also randomly selected  100 instances (at maximum) per each of these tasks.
+
+
+## Setting 
+The following command will clone the project: 
+```
+git clone https://github.com/seonghyeonye/ICIL.git
+```
+
+Before experimenting, you can make a virtual environment for the project.
+```
+conda create -n zeroshotlm python=3.8
+conda activate zeroshotlm
+pip install -r requirements.txt
+```
+
+## Run  
+You can run inference with various prompting schemes by running files under `scripts/gpt3` or `scripts/decoder`. For instance, if you want to test our ICIL prompt on GPT-3 (davinci) for 119 Test tasks from SuperNatural-Instructions, you can run
+```scripts/gpt3/run_ICIL.sh ```
+
+
+Result of In-Context Instruction Learning (ICIL)
+<p align="center">
+  <img src="./result.png" width="80%" height="80%">
+</p>
+
+
+### Irrelevant ICIL
+For Irrelevant ICIL, we randomly corrupt input sentences from the prompts with sentences from `cc_news`. To access this file, please visit our [google drive](https://drive.google.com/file/d/1Rl3jzQqNvP2Jzics9N4rhm1eF3pj5gix/view?usp=share_link). we have downloaded the huggingface `cc_news` dataset from [here](https://huggingface.co/datasets/cc_news) and parsed using `en_core_sm` from `spacy` and made it into a .txt file comprised of its sentences.
+
+But under `demo` directory, we have all the `irr_ICIL` demos already extracted, so instead, you can run `run_ICIL.sh`(instaed of `run_irr_ICIL.sh`) after you change the *--demo_path* option.
+
+
+## Experiments
+To replicate our experiments in the paper, please refer to the following resources:
+
+- **Figure 1,4**
+    - run files under `scripts/gpt3` for GPT3 API based models. 
+        - change the model type name by changing `for engine in "davinci"` code snippet to `for engine in {your model}`
+    - run files under `scripts/decoder` for models run with GPU. 
+        - change the model type name by changing `--model_name_or_path facebook/opt-30b` code snippet to `--model_name_or_path {your model}`)
+- **Table 1**
+    - run `scripts/gpt3/run_ICIL.sh` with --demo_path:
+        - `demos/analysis/distribution/irr_inst.json` for Random Instruction,
+        - `demos/irr_ICIL/irr_ICIL_seed1.json` for Random Input,
+        - `demos/analysis/distribution/irr_output.json` for Random Output.
+- **Figure 5**
+    - run `scripts/gpt3/run_ICIL.sh` and
+        - For (a): use demos in `demos/analysis/classification`
+        - For (b): use demos in `demos/analysis/shot`.
+        - For (c): use demos in `demos/random_ordering_ICIL`.
+- **Table 2**
+    - run `scripts/gpt3/run_ICIL.sh` with --demo_path `demos/analysis/answer_choice_overlap/eight_overlap_mixed.json.` for Overlap Setting.
+- **Figure 6**
+    - For (a), run `scripts/gpt3/run_ICIL.sh` and use demo in `demos/analysis/chatgpt/chatgpt.json` for Generated.
+    - For (b), run files under `scripts/ablation`. Note that this may take some time before starting the inference process.
+
+## OpenAI API
+For experiments with GPT-3 models (curie, davinci, text-curie, text-davinci), you should have a OpenAI API key, which can be applied [here](https://platform.openai.com/). After acquiring the key, insert your API key on ```export openai_key=""``` for each of the script file in ```scripts``` directory.
+
+## Acknowledgements
+Our code repository is mainly based on [Tk-Instruct](https://github.com/yizhongw/Tk-Instruct). Special thanks to the contributors of the repository!
+
+## Citations
+```
+@article{ye2023context,
+  title={In-Context Instruction Learning},
+  author={Ye, Seonghyeon and Hwang, Hyeonbin and Yang, Sohee and Yun, Hyeongu and Kim, Yireun and Seo, Minjoon},
+  journal={arXiv preprint arXiv:2302.14691},
+  year={2023}
+}
+```