    generative_ai/sm-jumpstart_foundation_finetuning_gpt_j_6b_domain_adaptation.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "e960490f",
   "metadata": {},
   "source": [
    "# SageMaker JumpStart Foundation Models - Fine-tuning text generation GPT-J 6B model on domain specific dataset"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "496fab09",
   "metadata": {},
   "source": [
    "---\n",
    "\n",
    "This notebook's CI test result for us-west-2 is as follows. CI test results in other regions can be found at the end of the notebook. \n",
    "\n",
    "![This us-west-2 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/us-west-2/generative_ai|sm-jumpstart_foundation_finetuning_gpt_j_6b_domain_adaptation.ipynb)\n",
    "\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "96f2327e",
   "metadata": {},
   "source": [
    "---\n",
    "Welcome to [Amazon SageMaker Built-in Algorithms](https://sagemaker.readthedocs.io/en/stable/algorithms/index.html)! You can use SageMaker Built-in algorithms to solve many Machine Learning tasks through [SageMaker Python SDK](https://sagemaker.readthedocs.io/en/stable/overview.html). You can also use these algorithms through one-click in SageMaker Studio via [JumpStart](https://docs.aws.amazon.com/sagemaker/latest/dg/studio-jumpstart.html).\n",
    "\n",
    "In this demo notebook, we demonstrate how to use the SageMaker Python SDK for finetuning Foundation Models and deploying the trained model for inference. The Foundation models perform Text Generation task. It takes a text string as input and predicts next words in the sequence.\n",
    "\n",
    "* **How to fine-tune [GPT-J 6B](https://huggingface.co/EleutherAI/gpt-j-6b) model on a domain specific dataset, and then run inference on the fine-tuned model. In particular, the example dataset we demonstrated is [publicly available SEC filing](https://www.sec.gov/edgar/searchedgar/companysearch) of Amazon from year 2021 to 2022. The expectation is that after fine-tuning, the model should be able to generate insightful text in financial domain.**\n",
    "\n",
    "Note: This notebook was tested on ml.t3.medium instance in Amazon SageMaker Studio with Python 3 (Data Science) kernel and in Amazon SageMaker Notebook instance with conda_python3 kernel.\n",
    "\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8091e1f6",
   "metadata": {},
   "source": [
    "1. Set Up\n",
    "2. Select Text Generation Model GTP-J 6B\n",
    "3. Finetune the pre-trained model on a custom dataset\n",
    "    * Set Training parameters\n",
    "    * Start Training\n",
    "    * Deploy & run Inference on the fine-tuned model"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2007b31a",
   "metadata": {},
   "source": [
    "## 1. Set Up\n",
    "Before executing the notebook, there are some initial steps required for setup."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "39b943ff",
   "metadata": {},
   "outputs": [],
   "source": [
    "!pip uninstall sagemaker\n",
    "!pip install sagemaker --quiet"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ee983c64",
   "metadata": {},
   "source": [
    "## 2. Select Text Generation Model GTP-J 6B\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "960ca9c8",
   "metadata": {},
   "outputs": [],
   "source": [
    "model_id = \"huggingface-textgeneration1-gpt-j-6b\""
   ]
  },
  {
   "cell_type": "markdown",
   "id": "70950bf9",
   "metadata": {},
   "source": [
    "## 3. Fine-tune the pre-trained model on a custom dataset"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1d1f6f8c",
   "metadata": {},
   "source": [
    "Fine-tuning refers to the process of taking a pre-trained language model and retraining it for a different but related task using specific data. This approach is also known as transfer learning, which involves transferring the knowledge learned from one task to another. Large language models (LLMs) like GPT-J 6B are trained on massive amounts of unlabeled data and can be fine-tuned on domain domain datasets, making the model perform better on that specific domain. \n",
    "\n",
    "We will use financial text from SEC filings to fine tune a LLM GPT-J 6B for financial applications. \n",
    "\n",
    "\n",
    "\n",
    "- **Input**: A train and an optional validation directory. Each directory contains a CSV/JSON/TXT file.\n",
    "    - For CSV/JSON files, the train or validation data is used from the column called 'text' or the first column if no column called 'text' is found.\n",
    "    - The number of files under train and validation (if provided) should equal to one.\n",
    "- **Output**: A trained model that can be deployed for inference.\n",
    "Below is an example of a TXT file for fine-tuning the Text Generation model. The TXT file is SEC filings of Amazon from year 2021 to 2022.\n",
    "\n",
    "---\n",
    "```\n",
    "This report includes estimates, projections, statements relating to our\n",
    "business plans, objectives, and expected operating results that are “forward-\n",
    "looking statements” within the meaning of the Private Securities Litigation\n",
    "Reform Act of 1995, Section 27A of the Securities Act of 1933, and Section 21E\n",
    "of the Securities Exchange Act of 1934. Forward-looking statements may appear\n",
    "throughout this report, including the following sections: “Business” (Part I,\n",
    "Item 1 of this Form 10-K), “Risk Factors” (Part I, Item 1A of this Form 10-K),\n",
    "and “Management’s Discussion and Analysis of Financial Condition and Results\n",
    "of Operations” (Part II, Item 7 of this Form 10-K). These forward-looking\n",
    "statements generally are identified by the words “believe,” “project,”\n",
    "“expect,” “anticipate,” “estimate,” “intend,” “strategy,” “future,”\n",
    "“opportunity,” “plan,” “may,” “should,” “will,” “would,” “will be,” “will\n",
    "continue,” “will likely result,” and similar expressions. Forward-looking\n",
    "statements are based on current expectations and assumptions that are subject\n",
    "to risks and uncertainties that may cause actual results to differ materially.\n",
    "We describe risks and uncertainties that could cause actual results and events\n",
    "to differ materially in “Risk Factors,” “Management’s Discussion and Analysis\n",
    "of Financial Condition and Results of Operations,” and “Quantitative and\n",
    "Qualitative Disclosures about Market Risk” (Part II, Item 7A of this Form\n",
    "10-K). Readers are cautioned not to place undue reliance on forward-looking\n",
    "statements, which speak only as of the date they are made. We undertake no\n",
    "obligation to update or revise publicly any forward-looking statements,\n",
    "whether because of new information, future events, or otherwise.\n",
    "\n",
    "GENERAL\n",
    "\n",
    "Embracing Our Future ...\n",
    "```\n",
    "---\n",
    "SEC filings data of Amazon is downloaded from publicly available [EDGAR](https://www.sec.gov/edgar/searchedgar/companysearch). Instruction of accessing the data is shown [here](https://www.sec.gov/os/accessing-edgar-data)."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c0bfe06b",
   "metadata": {},
   "source": [
    "### Set Training parameters\n",
    "Now that we are done with all the setup that is needed, we are ready to fine-tune our model. To begin, let us create a [``sageMaker.estimator.Estimator``](https://sagemaker.readthedocs.io/en/stable/api/training/estimators.html) object. This estimator will launch the training job. \n",
    "\n",
    "There are two kinds of parameters that need to be set for training. \n",
    "\n",
    "The first one are the parameters for the training job. These include: Training data path. This is S3 folder in which the input data is stored \n",
    "***\n",
    "The second set of parameters are algorithm specific training hyper-parameters.\n",
    "***"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "036bac37",
   "metadata": {},
   "outputs": [],
   "source": [
    "import json\n",
    "from sagemaker.jumpstart.estimator import JumpStartEstimator\n",
    "from sagemaker.jumpstart.utils import get_jumpstart_content_bucket\n",
    "\n",
    "# Sample training data is available in this bucket\n",
    "data_bucket = get_jumpstart_content_bucket()\n",
    "data_prefix = \"training-datasets/sec_data\"\n",
    "\n",
    "training_dataset_s3_path = f\"s3://{data_bucket}/{data_prefix}/train/\"\n",
    "validation_dataset_s3_path = f\"s3://{data_bucket}/{data_prefix}/validation/\""
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0d9d2622",
   "metadata": {},
   "source": [
    "### Start Training\n",
    "***\n",
    "We start by creating the estimator object with all the required assets and then launch the training job.  Since default hyperparameter values are model-specific, inspect estimator.hyperparameters() to view default values for your selected model.\n",
    "***"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "973d923c",
   "metadata": {},
   "outputs": [],
   "source": [
    "estimator = JumpStartEstimator(\n",
    "    model_id=model_id,\n",
    "    hyperparameters={\"epoch\": \"3\", \"per_device_train_batch_size\": \"4\"},\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5df9990b",
   "metadata": {},
   "outputs": [],
   "source": [
    "# You can now fit the estimator by providing training data to the train channel\n",
    "\n",
    "estimator.fit(\n",
    "    {\"train\": training_dataset_s3_path, \"validation\": validation_dataset_s3_path}, logs=True\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bd2d20f9",
   "metadata": {},
   "source": [
    "## Deploy & run Inference on the fine-tuned model\n",
    "***\n",
    "A trained model does nothing on its own. We now want to use the model to perform inference.\n",
    "***"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ce738168",
   "metadata": {},
   "outputs": [],
   "source": [
    "# You can deploy the fine-tuned model to an endpoint directly from the estimator.\n",
    "predictor = estimator.deploy()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ea22eef2",
   "metadata": {},
   "source": [
    "Next, we query the finetuned model and print the predictions."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a18b62f2",
   "metadata": {},
   "outputs": [],
   "source": [
    "payload = {\"inputs\": \"This Form 10-K report shows that\", \"parameters\": {\"max_new_tokens\": 400}}\n",
    "predictor.predict(payload)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "49f98c21",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Delete the SageMaker endpoint and the attached resources\n",
    "predictor.delete_predictor()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5eb59a9c",
   "metadata": {},
   "source": [
    "## Notebook CI Test Results\n",
    "\n",
    "This notebook was tested in multiple regions. The test results are as follows, except for us-west-2 which is shown at the top of the notebook.\n",
    "\n",
    "![This us-east-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/us-east-1/generative_ai|sm-jumpstart_foundation_finetuning_gpt_j_6b_domain_adaptation.ipynb)\n",
    "\n",
    "![This us-east-2 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/us-east-2/generative_ai|sm-jumpstart_foundation_finetuning_gpt_j_6b_domain_adaptation.ipynb)\n",
    "\n",
    "![This us-west-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/us-west-1/generative_ai|sm-jumpstart_foundation_finetuning_gpt_j_6b_domain_adaptation.ipynb)\n",
    "\n",
    "![This ca-central-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/ca-central-1/generative_ai|sm-jumpstart_foundation_finetuning_gpt_j_6b_domain_adaptation.ipynb)\n",
    "\n",
    "![This sa-east-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/sa-east-1/generative_ai|sm-jumpstart_foundation_finetuning_gpt_j_6b_domain_adaptation.ipynb)\n",
    "\n",
    "![This eu-west-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/eu-west-1/generative_ai|sm-jumpstart_foundation_finetuning_gpt_j_6b_domain_adaptation.ipynb)\n",
    "\n",
    "![This eu-west-2 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/eu-west-2/generative_ai|sm-jumpstart_foundation_finetuning_gpt_j_6b_domain_adaptation.ipynb)\n",
    "\n",
    "![This eu-west-3 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/eu-west-3/generative_ai|sm-jumpstart_foundation_finetuning_gpt_j_6b_domain_adaptation.ipynb)\n",
    "\n",
    "![This eu-central-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/eu-central-1/generative_ai|sm-jumpstart_foundation_finetuning_gpt_j_6b_domain_adaptation.ipynb)\n",
    "\n",
    "![This eu-north-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/eu-north-1/generative_ai|sm-jumpstart_foundation_finetuning_gpt_j_6b_domain_adaptation.ipynb)\n",
    "\n",
    "![This ap-southeast-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/ap-southeast-1/generative_ai|sm-jumpstart_foundation_finetuning_gpt_j_6b_domain_adaptation.ipynb)\n",
    "\n",
    "![This ap-southeast-2 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/ap-southeast-2/generative_ai|sm-jumpstart_foundation_finetuning_gpt_j_6b_domain_adaptation.ipynb)\n",
    "\n",
    "![This ap-northeast-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/ap-northeast-1/generative_ai|sm-jumpstart_foundation_finetuning_gpt_j_6b_domain_adaptation.ipynb)\n",
    "\n",
    "![This ap-northeast-2 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/ap-northeast-2/generative_ai|sm-jumpstart_foundation_finetuning_gpt_j_6b_domain_adaptation.ipynb)\n",
    "\n",
    "![This ap-south-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/ap-south-1/generative_ai|sm-jumpstart_foundation_finetuning_gpt_j_6b_domain_adaptation.ipynb)\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "conda_python3",
   "language": "python",
   "name": "conda_python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.13"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}