Generative AI Book Italian

BookLed Experiment 3 (AI Book!) jupyter notebook code

This wiki page provides step-by-step instructions for setting up and running the Generative AI jupyter notebook code from this GitHub repository.

Overview

This script is designed for an experimental setup of "BookLed," where pages of a physical card board book are read, and AI-generated audio content in Italian is supplied automatically as the pages turn. Here’s a detailed explanation of how the script works:

1) Copying the Jupyter Notebook Code from GitHub to Your Environment

Manual Download

1. Open the GitHub repository link: GitHub Link
2. Click on the Raw button to view the raw file.
3. Right-click on the page and select Save as... to download the file to your local machine.

Using Git

1. Open your terminal (or Git Bash if on Windows).
2. Navigate to the directory where you want to clone the repository:

   cd /path/to/your/directory

3. Clone the repository:

   git clone https://github.com/robotoons/BookLed.git

4. Navigate to the specific directory containing the Jupyter notebook:

   cd BookLed/codes/jupyter notebooks

2) Installing All Dependencies in the Environment with pip

1. Open your terminal.
2. Navigate to the directory containing the Jupyter notebook if you haven't already:

   cd /path/to/BookLed/codes/jupyter notebooks

3. Create a virtual environment (optional but recommended):

   python -m venv myenv

4. Activate the virtual environment:
   • On Windows:

     myenv\Scripts\activate

   • On macOS/Linux:

     source myenv/bin/activate

5. Install the required dependencies: Manually install the necessary packages:

   pip install pyserial jupyter

3) Activating the Environment and Running the Jupyter Notebook

1. Ensure your virtual environment is activated.
2. Launch Jupyter Notebook:

   jupyter notebook

3. In the Jupyter Notebook interface, navigate to the directory containing your downloaded notebook file (BookLed Experiment 3 (Generative AI Book Italian).ipynb) and open it.

4) Explanation of How the Code Works

Initial Setup and Configuration

1. Imports and Dependencies:

   • Various libraries are imported, including os, time, serial, threading, requests, pydub, wave, pyaudio, dotenv, json, and openai.
   • Deprecation warnings are suppressed.

2. Create a Temporary Directory:

   • A temporary directory named temp_mp3 is created to store downloaded MP3 files.

BookLed Serial Connection

3. Configure Serial Connection:
   • A serial connection to the BookLed device is established via a specified COM port (COM4).

Load OpenAI API Key

4. Load API Key:
   • The script loads the OpenAI API key from an environment variable using dotenv.

Prompts and Remote Access Gateway (RAG) Contents

5. Set URLs for Prompts and Images:
   • URLs for prompts and images stored on GitHub are specified. These are used later to retrieve content.

Functions Definition

6. Function: read_last_page_number:

   • Reads data from the serial port to extract the last page number detected by the BookLed device.

7. Function: read_file_to_string:

   • Reads the content of a local file and returns it as a string.

8. Function: download_file_from_github:

   • Downloads a file from a specified GitHub URL and returns its content as a string.

9. Function: download_mp3:

   • Downloads an MP3 file from a URL and saves it to the temporary directory.

10. Function: play_audio:

    • Converts an MP3 file to WAV format and plays it using pyaudio.

11. Function: play_mp3_in_thread:

    • Runs play_audio in a separate thread to allow concurrent playback.

12. Function: call_openai_api:

    • Sends a request to the OpenAI API to generate storytelling content based on the provided prompt and image URL.

13. Function: text_to_speech:

    • Converts generated text content to speech using the OpenAI API and saves it as an MP3 file.

Main Loop for BookLed Interaction

14. Prompt Source Selection:

    • Determines whether to use a local prompt file or download it from GitHub.

15. BookLed and AI Play Loop:

    • Initializes global variables (current_page, stop_audio, audio_thread).
    • Enters a loop where it continuously reads the current page number from the BookLed device.
    • When a new page is detected:
      • Stops any currently playing audio.
      • Calls the OpenAI API to convert the page image into storytelling content.
      • Converts the generated content into speech.
      • Starts playing the new audio in a separate thread.
    • Handles interruptions and ensures proper cleanup on exit.

Summary

The script integrates hardware (BookLed) with AI services (OpenAI API) to create an interactive experience where turning the pages of a physical book triggers the generation and playback of audio content. This is achieved through serial communication with the BookLed device, API calls to OpenAI for content generation and text-to-speech conversion, and concurrent audio playback using threading.