WMS ID: # 11988

aidp-essentials/1-create-aidp-instance-and-populate-catalog/1-create-aidp-instance-and-populate-catalog.md

@@ -0,0 +1,217 @@
+# Provision AIDP Workbench Instance and Populate the Catalog
+
+## Introduction
+
+This lab guides you through the process of provisioning an Oracle AI Data Platform (AIDP) Workbench instance, creating a 26ai database, creating catalogs, and managing data within those catalogs. You'll learn how to set up access to your data and organize it for future use in notebooks and jobs.
+
+Estimated Time: 45 minutes
+
+### Objectives
+
+In this lab, you will:
+
+- Provision an AIDP Workbench instance.
+- Provision a 26ai Autonomous Data Warehouse.
+- Access the master catalog.
+- Create internal and external catalogs.
+- Manage data by creating schemas, tables, and volumes.
+
+### Prerequisites
+
+This lab assumes you have:
+
+- An Oracle Cloud account.
+- The necessary permissions to create and manage AIDP Workbench instances.
+
+## Task 1: Provision an AIDP Workbench Instance
+
+1. Log in to your Oracle Cloud Infrastructure (OCI) account and access the OCI Console.
+
+2. From the OCI Console homepage, select the Navigation Menu, navigate to **Analytics and AI**, and select **AI Data Platform Workbench**.
+
+> **Note:** If you do not see **AI Data Platform Workbench** available from the menu, change your region to **Sao Paulo**
+
+![Accessing AIDP Workbench area in OCI](images/oci-console-access-aidp.png)
+
+3. Select **Create AI Data Platform Workbench**. There are permissions required to be able to provision an AIDP Workbench instance, learn more about them [here](https://docs.oracle.com/en/cloud/paas/ai-data-platform/aidug/iam-policies-oracle-ai-data-platform.html#GUID-C534FDF6-B678-4025-B65A-7217D9D9B3DA).
+
+![Choose Create AIDP Workbench](images/begin-aidp-creation.png)
+
+4. Provide an AI Data Platform name and Workspace name. In the **Add Policies** section for **Choose access level**, select **Standard - Recommended access settings at tenancy level**.
+
+![Name instance and workspace](images/input-aidp-information.png)
+
+5. If a dialog appears indicating that further policies are required to create the AIDP Workbench instance, review the policies and choose **Add**.
+
+
+6. Under **Optional Policies** view other suggested policies related to your AIDP Workbench instance. Add any you would like and select **Create**. It may take a few minutes for the instance to provision.
+
+![Create AIDP Instance](images/create-aidp-instance.png)
+
+
+## Task 2: Provision an Autonomous Data Warehouse
+
+1. While waiting for the AIDP Workbench to provision, select **Oracle AI Database** and then **Autonomous AI Database** from the Navigation Menu.
+
+![Navigate to Automonous Database page](images/navigate-to-dbs.png)
+
+2. Select **Create Autonomous AI Database**.
+
+![Begin Database creation](images/begin-db-creation.png)
+
+3. Provide a Display name and Database name. Leave the **Workload type** as **Lakehouse**.
+
+![Name database and choose workload](images/name-database.png)
+
+4. Under **Database version** select **26ai**. Deselect **Compute auto scaling**.
+
+![Choose DB version](images/choose-db-version.png)
+
+5. Enter a strong password for the admin credentials, leave all other fields as is, and then select **Create**.
+
+![Choose DB password](images/create-db-password.png)
+
+6. When the database is finished provisioning, select **Database connection**.
+
+![Navigate to database connection](images/access-db-connection.png)
+
+7. Select **Download Wallet**. Provide a password for the wallet, then select **Download**.
+
+![Download wallet](images/choose-wallet-password.png)
+
+8. The final step of this task is to configure table access for the **ADMIN** user. Under **Database actions** select **SQL**.
+
+![Select SQL database action](images/database-actions.png)
+
+9. In the SQL worksheet, add in the following command and run the statement. This statement allows the **ADMIN** user to select from any table in the DB.
+
+```
+  <copy>
+  GRANT SELECT ANY TABLE TO ADMIN;
+  </copy>
+  ```
+
+![Run SQL statement](images/grant-select.png)
+
+10. Replace the previous statement with the following one and run it as well. This statement defines a database table that we will write gold tier data to as part of the medallion architecture.
+
+```
+  <copy>
+  CREATE TABLE gold_supplier_feedback (
+      supplier_name VARCHAR(100),
+      continent VARCHAR(100),
+      country VARCHAR(100),
+      city VARCHAR(100),
+      summary_review VARCHAR(1000),
+      rating VARCHAR(10)
+  );
+  </copy>
+  ```
+
+![Create database table](images/create-gold-table.png)
+
+
+## Task 3: Create an External Catalog in AIDP Workbench
+
+Now you'll navigate back to your AIDP Workbench instance to create an external catalog.
+
+1. Select the Navigation Menu then **Analytics and AI** and **AI Data Platform Workbench**
+
+![Navigate to AIDP Section](images/oci-console-access-aidp.png)
+
+2. Select the AIDP Workbench instance you created earlier. Sign in with the same credentials you use to login to OCI.
+
+![Select AIDP Instance](images/access-aidp-instance.png)
+
+3. Your first step in the AIDP Workbench will be to connect to your data. From the AIDP Workbench homepage, select the **Master catalog** tab from the left-hand menu. Notice that the default catalog is already present here. Expand the default catalog and the **oci\_ai\_models** folder to see LLMs that are available for use in AIDP Workbench.
+
+![Select Master Catalog](images/view-genai-models.png)
+
+
+4. Create your first catalog by choosing **Create Catalog**.
+
+![Select Master Catalog](images/create-catalog.png)
+
+5. Enter the Catalog Name **supplier\_external\_26ai** and select **External Catalog** for **Catalog type**.
+
+![Select Master Catalog](images/name-external-catalog.png)
+
+6. Select **Oracle Autonomous Data Warehouse** as the source type. Upload the wallet file you downloaded in the previous task. Select the low option for **Service Level**, enter **ADMIN** for **Username**, and then enter the password you created under **Password**. Leave all other sections as is. Select **Test connection** then **Create**.
+
+![Select Master Catalog](images/configure-catalog-db-access.png)
+
+7. When creation of the catalog is complete, expand the **admin** database to see the available tables. You will use this connection to save your gold tier data to this database.
+
+![View DB tables](images/view-external-catalog.png)
+
+## Task 4: Create and Populate a Standard Catalog in AIDP Workbench
+
+Next you will create a standard catalog
+
+1. Use the breadcrumb menu to return to the master catalog if you are not already there. Select **Create catalog**.
+
+![Create Catalog](images/create-catalog.png)
+
+2. Enter the catalog name **Supplier**. Leave the **Catalog type** as **Standard catalog**. Select the same compartment your other lab assets are in and select **Create**.
+
+![create catalog](images/create-supplier-catalog.png)
+
+3. When the creation of the catalog is complete, select its name to access it.
+
+![select catalog](images/view-standard-catalog.png)
+
+4. Select **Create schema**.
+
+![create schema](images/create-schema.png)
+
+5. Enter the Schema Name **supplier\_schema** and select **Create**.
+
+![create supplier schema](images/create-supplier-schema.png)
+
+6. Select the **supplier_schema**.
+
+![select schema](images/select-supplier-schema.png)
+
+7. Select **Add to schema** and then **Table**.
+
+![Create table](images/create-table-clicks.png)
+
+8. Keep the **Table type** as **Managed**. Upload the **basic\_supplier.csv** file. Select **Preview data** and then **Create**. You can download the **basic\_supplier.csv** file and all other lab files at [this link](https://objectstorage.us-ashburn-1.oraclecloud.com/n/idmqvvdwzckf/b/LiveLab-Files_Bucket/o/aidp-workbench-ll-files.zip).
+
+![create table](images/create-basic-supplier.png)
+
+9. Create another managed table, using the **supplier\_emotions.csv** file.
+
+![create emotions table](images/create-supplier-emotions-table.png)
+
+10. Now you'll create a volume. Select **Add to schema**, and then **Volume**.
+
+![create volume](images/create-volume-clicks.png)
+
+11. Enter the Volume Name **supplier\_volume** and select **Managed** as the Volume type.
+
+![set as managed volume](images/create-supplier-volume.png)
+
+12. Select the **Volumes** tab and then the **Supplier\_Volume** volume you just created.
+
+![select volume](images/access-supplier-volume.png)
+
+13. Select the plus icon and then **Upload file**. Select the **supplier\_info.txt** file from your computer then choose **Upload**.
+
+![upload file to volume](images/upload-to-volume.png)
+
+You have now created your structured and unstructured data assets in AIDP Workbench that are ready to be processed into the bronze, silver, and gold tiers of a medallion architecture.
+
+## Learn More
+
+- [Oracle AI Data Platform Community Site](https://community.oracle.com/products/oracleaidp/)
+- [Oracle AI Data Platform Documentation](https://docs.oracle.com/en/cloud/paas/ai-data-platform/)
+- [Oracle Analytics Training Form](https://community.oracle.com/products/oracleanalytics/discussion/27343/oracle-ai-data-platform-webinar-series)
+- [AIDP Workbench Creation Documentation](https://docs.oracle.com/en/cloud/paas/ai-data-platform/aidug/get-started-oracle-ai-data-platform.html#GUID-487671D1-7ACB-4A56-B3CB-272B723E573C)
+- [AIDP Workbench Master Catalog Documentation](https://docs.oracle.com/en/cloud/paas/ai-data-platform/aidug/manage-master-catalog.html)
+- [Permissions for AIDP Workbench Creation](https://docs.oracle.com/en/cloud/paas/ai-data-platform/aidug/iam-policies-oracle-ai-data-platform.html#GUID-C534FDF6-B678-4025-B65A-7217D9D9B3DA)
+
+## Acknowledgements
+* **Author** - Miles Novotny, Senior Product Manager, Oracle Analytics Service Excellence
+* **Contributors** -  Farzin Barazandeh, Senior Principal Product Manager, Oracle Analytics Service Excellence
+* **Last Updated By/Date** - Miles Novotny, December 2025

aidp-essentials/2-create-medallion-architecture/2-create-medallion-architecture.md

@@ -0,0 +1,121 @@
+# Create Medallion Architecture using AIDP Workbench
+
+## Introduction
+
+This lab guides you through creating a medallion architecture using the Oracle AI Data Platform (AIDP). You'll learn how to set up workspaces, configure compute resources, and implement a multi-tier data processing pipeline.
+
+A medallion architecture is a data engineering design that organizes data into 3 tiers: bronze, silver, and gold. The bronze tier holds unprocessed and mostly raw data, the silver holds cleaned and organized data, while the gold tier holds processed, transformed, clean data that is ready for its final use. This final use could be analytics, ingestion into an application, or an AI use case. This architecture provides clear data lineage and allows the data pipeline to be easily edited.
+
+Estimated Time: 45 minutes
+
+### Objectives
+
+In this lab, you will:
+
+- Create and configure a workspace in AIDP Workbench.
+- Create a compute cluster in the workspace.
+- Use the AIDP Workbench notebook interface to implement a medallion architecture.
+- Utilize AI functionalities to enhance data processing.
+
+### Prerequisites
+
+This lab assumes you have:
+
+- A basic understanding of data science.
+
+## Task 1: Create Workspace and Compute Cluster
+
+1. In AIDP Workbench, open the **Workspace** tab and select **Create**.
+
+![Create workspace](images/create-workspace-button.png)
+
+2. Enter the Workspace name **Medallion\_Arch** and then select **Create**. While your workspace is provisioning, you'll see the default workspace that was created when you set up the AIDP Workbench instance.
+
+![input workspace details](images/create-workspace.png)
+
+3. Select **Medallion\_Arch** to enter the workspace. To create folders for the notebooks associated with the bronze, silver, and gold medallion tiers, select the plus icon and choose **Folder**
+
+![create a folder](images/create-folder.png)
+
+4. Name the folder **Bronze** and select **Create**. Repeat these steps to make two additional folders named **Silver** and **Gold**.
+
+![input the folder details](images/create-bronze-folder.png)
+
+5. To review the permissions for the **Bronze** folder, select the actions menu then choose **Permissions**.
+
+![review workspace permissions](images/access-permissions.png)
+
+6. Select the plus icon and view how to grant permissions to roles or individual users. These role-based access controls allow fine grain control over security and collaboration in AIDP Workbench.
+
+![Look at adding a permission](images/add-permissions.png)
+
+7. Next, to create a compute cluster in the workspace, open the **Compute** tab from the menu then select the plus icon.
+
+![navigate to compute](images/create-compute.png)
+
+8. Enter the Compute Cluster name **Medallion\_Compute**. Cluster creation defaults to a **Quickstart** setting which creates a small cluster that spins up very quickly. Deselect **Autoscale** and leave all other selections as is. Feel free to browse and view other compute shape options under the **Custom** section. Select **Create**. 
+
+![create compute](images/compute-settings.png)
+
+9. Wait for the cluster to become active. Select **Medallion\_Compute** to view its details. Here you can learn more about a cluster, including which notebooks use the cluster. Additionally, in the **Library** section you can install libraries from a **requirements.txt** file.
+
+![review compute details](images/compute-details.png)
+
+## Task 2: Import and Run Notebooks
+
+Now you'll import the notebooks used to implement the three tiers of the medallion architecture. 
+
+1. Open the **Medallion_Arch** workspace and select the **Bronze** folder.
+
+![access bronze](images/access-bronze-folder.png)
+
+2. To import the notebook used to create the bronze tier, which can be downloaded at [this link](https://objectstorage.us-ashburn-1.oraclecloud.com/n/idmqvvdwzckf/b/LiveLab-Files_Bucket/o/aidp-workbench-ll-files.zip), select the upload file icon.
+
+![upload file](images/upload-file-workspace.png)
+
+3. Choose the **1\_create\_bronze\_tier** file and select **Upload**.
+
+![choose file](images/upload-bronze.png)
+
+4. Repeat the same process for the **Silver** and **Gold** folders. Upload **2\_silver\_transformation**, **3\_silver\_transformation\_continent**, and **4\_silver\_transformation\_summary** into the **Silver** folder. Upload **5\_gold\_join** and **6\_gold\_job\_into\_DB** into the **Gold** folder.
+
+5. Open the **Bronze** folder. Select the **create\_bronze\_tier** file to open it in the notebook interface.
+
+![open file](images/open-notebook.png)
+
+6. Once in the notebook interface you need to attach a compute cluster on which to run the code. Select **Cluster** then **Attach existing cluster**. Choose the **Medallion\_Compute** you created earlier.
+
+![attach cluster](images/attach-cluster.png)
+
+7. Scroll through the notebook and review the code you are about to run. The main actions in this notebook are the creation of a **bronze\_supplier** catalog and a schema. Then the **basic\_supplier** and **supplier\_emotions** datasets are written to the catalog as tables.
+
+8. Once you have reviewed the code, select **Run all**.
+
+![run code](images/run-notebook.png)
+
+10. When this process is complete, continue to the notebooks in the **Silver** folder. Review and run them in numerical order. These notebooks clean and organize the bronze tier data to create the silver tier. The code leverages LLMs from the OCI Gen AI Service to augment the data.
+
+![run silver notebooks](images/silver-notebooks.png)
+
+11. Finally run the two notebooks in the **Gold** folder. These noteboooks consolidate tables from the silver tier into a finalized gold table. This table is then loaded into the 26ai database using the external catalog you created earlier. Saving the gold tier data to this datastore allows it to be effectively leveraged for analytics workloads.
+
+![run gold notebooks.](images/gold-notebooks.png)
+
+9. Navigate to the master catalog to view the catalogs created by the notebook code. Notice that the bronze, silver, and gold tier catalogs are created and populated in the master catalog.
+
+![view catalogs](images/catalogs.png)
+
+You have now imported notebooks and run the code to create a basic medallion architecture based on the supplier data you stored in the AIDP Workbench Master Catalog.
+
+
+## Learn More
+
+- [Oracle AI Data Platform Community Site](https://community.oracle.com/products/oracleaidp/)
+- [Oracle AI Data Platform Documentation](https://docs.oracle.com/en/cloud/paas/ai-data-platform/)
+- [Oracle Analytics Training Form](https://community.oracle.com/products/oracleanalytics/discussion/27343/oracle-ai-data-platform-webinar-series)
+- [AIDP Workbench Notebooks Documentation](https://docs.oracle.com/en/cloud/paas/ai-data-platform/aidug/notebooks.html)
+
+## Acknowledgements
+* **Author** - Miles Novotny, Senior Product Manager, Oracle Analytics Service Excellence
+* **Contributors** -  Farzin Barazandeh, Senior Principal Product Manager, Oracle Analytics Service Excellence
+* **Last Updated By/Date** - Miles Novotny, December 2025