docs/notebooks/demo/gretel-demo-sequence-of-events.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a target=\"_blank\" href=\"https://colab.research.google.com/github/gretelai/gretel-blueprints/blob/main/docs/notebooks/demo/gretel-demo-sequence-of-events.ipynb\">\n",
    "  <img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/>\n",
    "</a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Create Complex Time Series sequences\n",
    "\n",
    "* This notebook demonstrates how to use Gretel DGAN with Gretel Tuner to generate synthetic time series adhering to a business logic or set of rules.\n",
    "* To run this notebook, you will need an API key from the [Gretel Console](https://console.gretel.ai/)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Getting Started"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "!pip install -Uqq gretel-client[tuner]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Download the file with helper functions needed to run this notebook\n",
    "!curl -o dgan_tuner_utils.py https://raw.githubusercontent.com/gretelai/gretel-blueprints/main/docs/notebooks/demo/demo-utilities-lib/dgan_tuner_utils.py"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Dataset description"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This dataset describes a project management process as shown in the figure below. The process is structured into five main phases, with each phase comprising mandatory events and optional events to provide flexibility and adaptability to the project's needs. \n",
    "\n",
    "<img src=\"https://gretel-datasets.s3.us-west-2.amazonaws.com/time-series-events/project_management_workflow.png\" width=\"25%\" height=\"25%\">\n",
    "<!--  -->\n",
    "<!-- ![Project Management Workflow](https://gretel-datasets.s3.us-west-2.amazonaws.com/project_management_sequences/project_management_workflow.png) -->\n",
    "\n",
    "Here's a breakdown of the dataset based on this workflow:\n",
    "\n",
    "### Phases:\n",
    "**A Initiation:** The starting point of the project, focused on establishing the project's foundation.  \n",
    "**B Planning:** Involves detailed preparation and strategizing for how the project will be executed.  \n",
    "**C Execution:** The phase where the planned activities are carried out to create the project's deliverables.  \n",
    "**D Monitoring and Controlling:** Concurrent with execution, this phase ensures the project stays on track and adheres to quality standards.  \n",
    "**E Closure:** Concludes the project by ensuring all aspects are completed satisfactorily and formally closing the project.  \n",
    "\n",
    "\n",
    "### Mandatory Events:\n",
    "**A-1 Project Kick-off**: Marks the official start of the project.  \n",
    "**B-1 Requirements Gathering**: Collection of all necessary project requirements.  \n",
    "**B-2 Resource Allocation**: Assignment and scheduling of resources needed for the project.  \n",
    "**C-1 Development Start**: Commencement of the project development activities.  \n",
    "**D-1 Quality Assurance Testing**: Testing to ensure the quality of the project's outputs.  \n",
    "**E-1 Final Review**: Comprehensive review of all project deliverables.  \n",
    "\n",
    "### Optional Events:\n",
    "**B-3 Risk Assessment**: Evaluation of potential project risks and their impacts.  \n",
    "**C-2 First Prototype Review**: Assessment of an early project prototype.  \n",
    "**C-3 Mid-Project Evaluation**: Evaluation of project progress before completion.  \n",
    "**D-2 Client Feedback Session**: Gathering feedback from clients or stakeholders.  \n",
    "**D-3 Adjustments Based on Feedback**: Making changes to the project based on received feedback.  \n",
    "**E-2 Deployment**: Release of the final product to the end-users or stakeholders.  \n",
    "**E-3 Project Retrospective**: Reflective meeting to discuss what went well and what could be improved.\n",
    "\n",
    "### Workflow Logic:\n",
    "The workflow is designed with a logical progression from initiation to closure, with mandatory events ensuring the project's essential milestones are met. Optional events provide opportunities to enhance project outcomes, address unforeseen challenges, or incorporate stakeholder feedback. Solid lines represent the flow between mandatory events, while dashed lines indicate where optional events can be integrated into the project lifecycle.\n",
    "\n",
    "This dataset and workflow visualization offer a comprehensive overview of the structured approach to managing projects, highlighting the flexibility to adapt to project-specific requirements and changes throughout the project"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Load and preview training data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "\n",
    "df = pd.read_csv(\n",
    "    \"https://gretel-datasets.s3.us-west-2.amazonaws.com/time-series-events/project_management_sequences.csv\"\n",
    ")\n",
    "\n",
    "EXAMPLE_COLUMN = \"PROJECT_ID\"\n",
    "EVENT_COLUMN = \"EVENT_TYPE\"\n",
    "\n",
    "selected_projects = np.random.choice(\n",
    "    df[EXAMPLE_COLUMN].unique(), 3, replace=False\n",
    ")\n",
    "\n",
    "pd.set_option(\"display.max_rows\", None)\n",
    "display(df[df[EXAMPLE_COLUMN].isin(selected_projects)])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Prepare data for DGAN"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from dgan_tuner_utils import pad_sequence\n",
    "\n",
    "# prepare for DGAN\n",
    "\n",
    "max_len = df.groupby(EXAMPLE_COLUMN).size().max()\n",
    "\n",
    "# Pad each group and concatenate back into a DataFrame\n",
    "data_source = pd.concat(\n",
    "    [pad_sequence(group, max_len, example_id_column=EXAMPLE_COLUMN, event_column=EVENT_COLUMN, pad_value=\"[END]\") for _, group in df.groupby(EXAMPLE_COLUMN)],\n",
    "    ignore_index=True,\n",
    ")\n",
    "\n",
    "# Number of sequences in the source dataset\n",
    "NUM_RECORDS = len(data_source.groupby(EXAMPLE_COLUMN))\n",
    "\n",
    "# MAX_SEQUENCE_LEN defines the total length of synthetic sequences generated and also the fixed length for all training examples.\n",
    "# This parameter ensures uniformity in sequence length across the dataset, set to 6 here indicating that each sequence (synthetic or training) will consist of 6 time points.\n",
    "MAX_SEQUENCE_LEN = int(data_source.groupby(EXAMPLE_COLUMN).size().max())\n",
    "\n",
    "# SAMPLE_LEN specifies the number of time points generated by a single RNN cell within the generator.\n",
    "# It must be a divisor of MAX_SEQUENCE_LEN. A value of 1 here means each RNN cell generates 1 time point in the sequence.\n",
    "# For optimal model learning and memory management, the ratio of MAX_SEQUENCE_LEN to SAMPLE_LEN should ideally be between 10 and 20.\n",
    "SAMPLE_LEN = 1\n",
    "\n",
    "print(f\"Number of Records: {NUM_RECORDS}\")\n",
    "print(f\"Maximum Sequence Length: {MAX_SEQUENCE_LEN}\")\n",
    "print(f\"Sample Length: {SAMPLE_LEN}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Visualize the sequences"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from dgan_tuner_utils import (\n",
    "    plot_event_sequences,\n",
    "    plot_transition_matrices,\n",
    "    plot_event_type_distribution,\n",
    ")\n",
    "\n",
    "# Define the valid sequence including optional events. Optional events are provided as a list.\n",
    "valid_sequence = [\n",
    "    \"A-1 Project Kick-off\",\n",
    "    \"B-1 Requirements Gathering\",\n",
    "    \"B-2 Resource Allocation\",\n",
    "    [\"B-3 Risk Assessment\"],  # Optional\n",
    "    \"C-1 Development Start\",\n",
    "    [\"C-2 First Prototype Review\", \"C-3 Mid-Project Evaluation\"],  # Optional\n",
    "    \"D-1 Quality Assurance Testing\",\n",
    "    [\"D-2 Client Feedback Session\", \"D-3 Adjustments Based on Feedback\",],  # Optional\n",
    "    \"E-1 Final Review\",\n",
    "    [\"E-2 Deployment\", \"E-3 Project Retrospective\", \"[END]\"],  # Optional\n",
    "]\n",
    "\n",
    "event_mapping = {\n",
    "    \"A-1 Project Kick-off\": \"A-1\",\n",
    "    \"B-1 Requirements Gathering\": \"B-1\",\n",
    "    \"B-2 Resource Allocation\": \"B-2\",\n",
    "    \"B-3 Risk Assessment\": \"B-3\",  # Optional\n",
    "    \"C-1 Development Start\": \"C-1\",\n",
    "    \"C-2 First Prototype Review\": \"C-2\",  # Optional\n",
    "    \"C-3 Mid-Project Evaluation\": \"C-3\",  # Optional\n",
    "    \"D-1 Quality Assurance Testing\": \"D-1\",\n",
    "    \"D-2 Client Feedback Session\": \"D-2\",  # Optional\n",
    "    \"D-3 Adjustments Based on Feedback\": \"D-3\",  # Optional\n",
    "    \"E-1 Final Review\": \"E-1\",\n",
    "    \"E-2 Deployment\": \"E-2\",  # Optional\n",
    "    \"E-3 Project Retrospective\": \"E-3\",  # Optional\n",
    "    \"[END]\": \"[END]\",  # Optional\n",
    "}\n",
    "\n",
    "plot_event_sequences(data_source, example_id_column=EXAMPLE_COLUMN, event_column=EVENT_COLUMN, num_sequences=5, event_mapping=event_mapping)\n",
    "plot_event_type_distribution(data_source, event_column=EVENT_COLUMN, event_mapping=event_mapping)\n",
    "plot_transition_matrices(data_source, example_id_column=EXAMPLE_COLUMN, event_column=EVENT_COLUMN, event_mapping=event_mapping)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Create Gretel Session and Project"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from gretel_client import Gretel\n",
    "\n",
    "gretel = Gretel(\n",
    "    project_name=\"gretel-demo-events\",\n",
    "    api_key=\"prompt\",\n",
    "    cache=\"yes\",\n",
    "    validate=True,\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Define base DGAN configuration"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# create a custom dgan config to modify some base setting\n",
    "import yaml\n",
    "from gretel_client.gretel.config_setup import create_model_config_from_base\n",
    "\n",
    "config = create_model_config_from_base(\n",
    "    \"time-series\",\n",
    "    params={\n",
    "        \"apply_example_scaling\": True,\n",
    "        \"max_sequence_len\": MAX_SEQUENCE_LEN,\n",
    "        \"sample_len\": SAMPLE_LEN,\n",
    "    },\n",
    "    example_id_column=EXAMPLE_COLUMN,\n",
    ")\n",
    "\n",
    "with open(\"custom_base_dgan_config.yaml\", \"w\") as file:\n",
    "    yaml.dump(config, file, default_flow_style=False, sort_keys=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Train DGAN models leveraging Gretel Tuner"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from dgan_tuner_utils import EventTypeHistogramAndTransitionDistance\n",
    "\n",
    "# This cell should take ~20 minutes to complete.\n",
    "tuner_config = \"\"\"\n",
    "base_config: custom_base_dgan_config.yaml\n",
    "\n",
    "params:\n",
    "\n",
    "    attribute_loss_coef:\n",
    "        choices: [1, 5, 10]\n",
    "\n",
    "    attribute_num_layers:\n",
    "        choices: [3, 4]\n",
    "\n",
    "    attribute_num_units:\n",
    "        choices: [50, 100, 200]\n",
    "\n",
    "    batch_size:\n",
    "        choices: [100, 200]\n",
    "\n",
    "    epochs:\n",
    "        choices: [1000, 2000, 4000, 8000]\n",
    "\n",
    "    generator_learning_rate:\n",
    "        log_range: [0.000001, 0.001]\n",
    "\n",
    "    discriminator_learning_rate:\n",
    "        log_range: [0.000001, 0.001]\n",
    "\n",
    "\"\"\"\n",
    "\n",
    "target_job = \"tune-dgan\"\n",
    "\n",
    "metric = EventTypeHistogramAndTransitionDistance(reference_df=data_source, example_id_column=EXAMPLE_COLUMN, event_column=EVENT_COLUMN, num_samples=5000)\n",
    "\n",
    "tuner_results = gretel.run_tuner(\n",
    "    tuner_config,\n",
    "    data_source=data_source,\n",
    "    n_jobs=4,\n",
    "    n_trials=24,\n",
    "    metric=metric,\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# show best config \n",
    "best_config = tuner_results.best_config\n",
    "print(yaml.dump(best_config))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Generate synthetic time-series"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# generate data\n",
    "trained = gretel.fetch_train_job_results(tuner_results.best_model_id)\n",
    "generated = gretel.submit_generate(trained.model_id, num_records=1000)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Assess synthetic time-series results"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from dgan_tuner_utils import (\n",
    "    calculate_percentage_of_valid_sequences,\n",
    "    remove_invalid_sequences,\n",
    "    check_series_order,\n",
    ")\n",
    "\n",
    "# Retrieve synthetic data and remove invalid sequences\n",
    "generated_data = generated.synthetic_data\n",
    "series_validity = generated_data.groupby(EXAMPLE_COLUMN)[EVENT_COLUMN].apply(\n",
    "    lambda x: check_series_order(x, valid_sequence)\n",
    ")\n",
    "generated_data_valid = remove_invalid_sequences(\n",
    "    generated_data, series_validity, EXAMPLE_COLUMN\n",
    ")\n",
    "\n",
    "# Calculate the percentage of valid sequences\n",
    "percentage_valid = calculate_percentage_of_valid_sequences(series_validity)\n",
    "\n",
    "print(f\"Percentage of Valid Sequences: {percentage_valid:.2f}%\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# show source vs synthetic data\n",
    "plot_event_sequences(\n",
    "    generated_data_valid,\n",
    "    example_id_column=EXAMPLE_COLUMN,\n",
    "    event_column=EVENT_COLUMN,\n",
    "    df_ref=data_source,\n",
    "    num_sequences=5,\n",
    "    event_mapping=event_mapping,\n",
    ")\n",
    "\n",
    "plot_event_type_distribution(\n",
    "    generated_data_valid, \n",
    "    event_column=EVENT_COLUMN,\n",
    "    df_ref=data_source, \n",
    "    event_mapping=event_mapping\n",
    ")\n",
    "\n",
    "plot_transition_matrices(\n",
    "    generated_data_valid,\n",
    "    example_id_column=EXAMPLE_COLUMN,\n",
    "    event_column=EVENT_COLUMN,\n",
    "    df_ref=data_source,\n",
    "    event_mapping=event_mapping\n",
    ")"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "venv",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}