Skip to content

Generative AI Book Multilingual with soundtrack

Jump to bottom
Robotoons edited this page Jun 3, 2024 · 3 revisions

BookLed Experiment 6 (AI Book with Voiceover and Soundtrack) - MULTILINGUAL

Overview

This Python script is designed for an experiment where turning the pages of the BookLed physical book triggers the playback of an MP3 audio, generated in real-time by AI. The AI analyzes the graphical content of the book pages and produces a corresponding story. Each page turn prompts the AI to generate a unique narrative, which is then converted to speech using OpenAI's TTS API. Additionally, a contextual soundtrack is played alongside the narration.

Prerequisites

  1. BookLed Device: Connect the BookLed to the PC via USB and ensure the correct COM port is configured.
  2. OpenAI API: An active OpenAI API account is required.
  3. Internet Connection: The OpenAI API and GitHub content are accessed via the internet.
  4. FFmpeg: Ensure FFmpeg is installed and configured correctly for audio processing.

Setup Instructions

  1. Connect the BookLed to your PC via USB.
  2. Configure the correct COM port number.
  3. Ensure you have an active internet connection.
  4. Ensure you have an active OpenAI API account.
  5. Power on the BookLed device.

Configuration

TTS Voice

To select the TTS voice, you need to uncomment the desired TTS voice from the list provided in the script. By default, the variable tts_voice is set to "shimmer". To change the TTS voice, uncomment the line with the desired voice and comment out the current selection. Here is the relevant section of the code:

################################################################
# Uncomment the selected OpenAI TTS (text to speech) API voice #
################################################################
#tts_voice = "alloy"
#tts_voice = "echo"
#tts_voice = "fable"
#tts_voice = "onyx"
#tts_voice = "nova"
tts_voice = "shimmer"

For example, if you want to use the "nova" voice instead of "shimmer", modify the code as follows:

################################################################
# Uncomment the selected OpenAI TTS (text to speech) API voice #
################################################################
#tts_voice = "alloy"
#tts_voice = "echo"
#tts_voice = "fable"
#tts_voice = "onyx"
tts_voice = "nova"
#tts_voice = "shimmer"

Language Prompt

To set the language for the narration, you need to uncomment or add the desired language prompt. The variable prompt_addendum is used to define the language. Here is the relevant section of the code:

######################################################################################
# Uncomment the selected language for the narration or provide the language you want #
######################################################################################
#prompt_addendum = "Talk in English."
prompt_addendum = "Talk in Danish."
#prompt_addendum = "speaks in Japanese."
#prompt_addendum = "Produci un racconto in lingua italiana."

For example, if you want the narration in Italian, modify the code as follows:

######################################################################################
# Uncomment the selected language for the narration or provide the language you want #
######################################################################################
#prompt_addendum = "Talk in English."
#prompt_addendum = "Talk in Danish."
#prompt_addendum = "speaks in Japanese."
prompt_addendum = "Produci un racconto in lingua italiana."

Main Loop (Start the Play Loop)

Step-by-Step Explanation

  1. Initialization:

    • Serial Connection: The script sets up a serial connection to the BookLed device.
    • Temporary Directory: Creates a temporary directory to store MP3 files.

  2. Loading API Key:

    • The OpenAI API key is loaded from an environment file (.env).

  3. Set Remote and Local Contents:

    • URLs for the prompt file and images are defined.
    • The local filenames and URLs for soundtracks are set.

  4. Read BookLed Page Function:

    • A function (read_last_page_number) reads the page number from the BookLed device via the serial port.

  5. Main Play Loop:

    • The script enters a while loop, continuously checking for page changes.
    • When a new page is detected:
      • The current page number is updated.
      • The appropriate soundtrack file for the page is determined and played.
      • The previous voiceover audio playback is stopped if any.
      • The AI generates a story based on the page's image.
      • The generated text is converted to speech using the TTS API.
      • The voiceover audio is played.

  6. Exception Handling and Cleanup:

    • The script handles keyboard interrupts to gracefully stop audio playback and close the serial connection.

Code Explanation

# Initialize global variables
current_page = None
stop_audio = False
audio_thread = None
stop_audio_flag = threading.Event()
soundtrack_thread = None
stop_soundtrack_flag = threading.Event()

# Start the play loop
print("BookLed play start!")
try:
    while True:
        page_number = read_last_page_number()
        if page_number != current_page:
            current_page = page_number
            print("BookLed page number=", page_number)

            # Determine the appropriate soundtrack file
            soundtrack_file = None
            for page_range, soundtrack in soundtrack_map.items():
                if page_number in page_range:
                    soundtrack_file = soundtrack
                    break

            if 1 <= current_page <= 12:

                # Play or switch the soundtrack
                if soundtrack_file:
                    soundtrack_index = list(soundtrack_map.values()).index(soundtrack_file)
                    if soundtrack_thread is None or not soundtrack_thread.is_alive():
                        soundtrack_path = download_mp3(soundtrack_urls[soundtrack_index], soundtrack_file)
                        stop_soundtrack_flag.clear()
                        print("Start soundtrack MP3 audio play")
                        soundtrack_thread = play_mp3_in_thread(soundtrack_path, stop_soundtrack_flag, volume=SOUNDTRACK_VOLUME, loop=True)
                    elif soundtrack_file not in soundtrack_thread.name:
                        stop_soundtrack_flag.set()
                        soundtrack_thread.join()
                        soundtrack_path = download_mp3(soundtrack_urls[soundtrack_index], soundtrack_file)
                        stop_soundtrack_flag.clear()
                        print("Start soundtrack MP3 audio play")
                        soundtrack_thread = play_mp3_in_thread(soundtrack_path, stop_soundtrack_flag, volume=SOUNDTRACK_VOLUME, loop=True)

                # Stop the previous voiceover audio
                print("Stop voice-over MP3 audio play")
                stop_audio = True
                if audio_thread is not None:
                    stop_audio_flag.set()
                    audio_thread.join()

                image_url = image_urls[current_page - 1]
                print("OpenAI call #1: convert image in storytelling")
                content = call_openai_api(prompt_text, image_url)
                print("OpenAI call #2: convert text in speech")
                speech_file_path = text_to_speech(content, tts_voice)
                print("Start voice-over MP3 audio play")
                stop_audio_flag.clear()
                audio_thread = play_mp3_in_thread(speech_file_path, stop_audio_flag, volume=VOICEOVER_VOLUME, loop=False)

            else:
                stop_audio = True
                if audio_thread is not None:
                    audio_thread.join()
                stop_soundtrack_flag.set()
                if soundtrack_thread is not None:
                    soundtrack_thread.join()
                audio_thread = None
                soundtrack_thread = None
except KeyboardInterrupt:
    print("Program interrupted")
finally:
    ser.close()
    stop_audio = True
    if audio_thread is not None:
        audio_thread.join()
    stop_soundtrack_flag.set()
    if soundtrack_thread is not None:
        soundtrack_thread.join()
    print("Program terminated")

This loop continuously checks for new page numbers, updates the current page, manages audio playback (both voiceover and soundtrack), and makes necessary API calls to generate the story and convert it to speech.

RoboToons at work

Clone this wiki locally