book.4_test.ipynb

{
 "cells": [
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Testing our performance\n",
    "\n",
    "#### Installing HumanEval\n",
    "Clone and install the test suite:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%bash\n",
    "source ../venv/bin/activate\n",
    "pip install -r ../requirements.txt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import sys\n",
    "#!source venv/bin/activate\n",
    "!git clone https://github.com/openai/human-eval\n",
    "!{sys.executable} -m pip install -e human-eval\n",
    "sys.path.append('human-eval')\n",
    "sys.path.append('src')"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Code execution\n",
    "\n",
    "In order for the HumanEval test suite to test your code, you must uncomment the line that executes the code.\n",
    "\n",
    "#### WARNING: This will cause your machine to execute code generated by the LLM, which may be unsafe. But who cares, this is science!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "!sed -i '58s/# //' ../venv/src/human-eval/human_eval/execution.py"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Take the test\n",
    "Now let's have our agent take the test and generate responses, which we will grade in the next step.\n",
    "To run these in a jupyter notebook, we need to add the following sources to our PATH first. If you did not run this in the setup step above, run it now:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['/home/space/Documents/projects/ai-code-builder/git-molecul-ai/moleculai', '/home/space/miniconda3/lib/python38.zip', '/home/space/miniconda3/lib/python3.8', '/home/space/miniconda3/lib/python3.8/lib-dynload', '', '/home/space/Documents/projects/ai-code-builder/git-molecul-ai/venv/lib/python3.8/site-packages', '/home/space/Documents/projects/ai-code-builder/git-molecul-ai/venv/src/human-eval', 'human-eval', 'src']\n"
     ]
    }
   ],
   "source": [
    "import sys\n",
    "import os\n",
    "sys.path.append('human-eval')\n",
    "sys.path.append('src')\n",
    "print(sys.path)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now let's import the file with our test code, and run the test suite:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "from test_humaneval import generate_samples\n",
    "C_OUTPUT_FOLDER='test-output'\n",
    "C_TEST_NAME='human-eval-output-3.5'\n",
    "await generate_samples(output_dir = os.path.join(C_OUTPUT_FOLDER, C_TEST_NAME))\n"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Evaluating Performance"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Reading samples...\n",
      "164it [00:00, 7410.91it/s]\n",
      "Running test suites...\n",
      "100%|████████████████████████████████████████| 164/164 [00:01<00:00, 116.42it/s]\n",
      "Writing results to test-output/human-eval-output-3.5/samples.jsonl_results.jsonl...\n",
      "100%|██████████████████████████████████████| 164/164 [00:00<00:00, 30527.04it/s]\n",
      "{'pass@1': 0.6890243902439024}\n"
     ]
    }
   ],
   "source": [
    "!evaluate_functional_correctness {C_OUTPUT_FOLDER}/{C_TEST_NAME}/samples.jsonl"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# TEST WITH GPT-4\n",
    "\n",
    "Let us now, how they say, take it up a notch.\n",
    "\n",
    "Our LLM configs are stored in the `moleculai/config` folder. Let's do the test using the GPT-4 config."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from test_humaneval import generate_samples\n",
    "C_OUTPUT_FOLDER='test-output'\n",
    "C_TEST_NAME='human-eval-output-4.0'\n",
    "C_CONFIG_NAME='llm-gpt40.json'\n",
    "await generate_samples(output_dir = os.path.join(C_OUTPUT_FOLDER, C_TEST_NAME), config_llm=C_CONFIG_NAME)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Reading samples...\n",
      "164it [00:00, 15326.44it/s]\n",
      "Running test suites...\n",
      "100%|████████████████████████████████████████| 164/164 [00:01<00:00, 110.68it/s]\n",
      "Writing results to test-output/human-eval-output-4.0/samples.jsonl_results.jsonl...\n",
      "100%|██████████████████████████████████████| 164/164 [00:00<00:00, 22764.20it/s]\n",
      "{'pass@1': 0.8292682926829268}\n"
     ]
    }
   ],
   "source": [
    "!evaluate_functional_correctness {C_OUTPUT_FOLDER}/{C_TEST_NAME}/samples.jsonl"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.10"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}