Poetic Lens 📸→📝

An AI pipeline that transforms photos into poetry using computer vision and text generation.

How it Works

1. Take a Picture - Capture a photo using your webcam
2. Generate Caption - BLIP model describes what's in the image
3. Create Prompt - Truncated caption becomes the prompt
4. Generate Poem - Custom-trained GPT model creates a poem from the prompt

Setup

1. Create Virtual Environment

python3 -m venv venv
source venv/bin/activate

2. Install Dependencies

pip install -r requirements.txt

3. Train the Poetry Model (Optional)

python3 model.py

This will download poetry data and train the GPT model. But the latest checkpoint and model are saved so you don't need to retrain.

Usage

Quick Start

python3 main.py

Controls

• Spacebar - Take a picture
• ESC - Exit and generate poem

Pipeline Flow

📸 Camera → Image Saved → BLIP Caption → Prompt Extract → Poem Generation → 📄 poem.txt

Example

1. Take a picture of a sunset
2. BLIP generates: "a photography of beautiful sunset over mountains"
3. Prompt extracted: "beautiful sunset over"
4. GPT generates a poem starting with those words
5. Poem saved to poem.txt and printed to terminal

Project Structure

Files

• take_pic.py - Camera capture script
• blip.py - Generates caption for the picture using BLIP and truncate it into a prompt
• gen_poem.py - Poem generation from prompt
• model.py - GPT training script
• prompt.txt - Temporary file storing the image caption prompt
• poem.txt - Final generated poem output
• opencv_frame_0.png - Captured image

Folders

• checkpoints - stores the best model and latest checkpoint during training
• scene_images - test images for BLIP
• frame_msg - code to generate poem on AR Glasses (Not implemented fully. For later)

Customization

• Modify prompt length in the blip.py script
• Adjust poem length with max_new_tokens parameter

Outputs

Example 1:

a woman with a marche to each others grasp, and it will be commanding with those pumps stove is the space he crawled with pulses of dust corrents diaper. Or more wondering thee.

Home. I Touch the time through a hollow scale of oozild left, you met an unknown. All that you there is to have gone up to watch only this drawing; the living street, you do not, you would ragge from the taste but now you naught

Example 2:

a man sitting at my son. He was shoved the scrowling of a patfort power. What do I say here, if they think he crawled with pulse I sleep from my piece of worms and I feel at keeping the missing, be a better through into the wall of our bares.

Exception in the day I can shone their soutches, refused her women only this she moves near with a pilphone were durapped.

Here mocks, pull them, but now God. Alway

Acknowledgements

I fine-tuned Karpathy's GPT model with the poetry dataset. Link to reference: Github

---