CrewAI LinkedIn Post Generator

Overview

This repository contains a CrewAI-powered application that scrapes a LinkedIn profile to learn a writing style, researches a topic on the web, and then generates an engaging LinkedIn post that mirrors the learned style.

What it does

• Scrapes public posts from a target LinkedIn profile to infer tone, structure, and voice
• Performs focused web research on a chosen comparison/topic (default: GPT‑5 vs Claude)
• Synthesizes findings into a high‑quality LinkedIn post in the scraped style

How it works (at a glance)

• Agents: three CrewAI agents (LinkedIn Post Scraper, Web Researcher, LinkedIn Post Creator)
• Tasks: one task per agent, orchestrated in sequence; later tasks use outputs from earlier ones
• LLM: OpenAI Chat model via langchain-openai (model: gpt-5-mini)
• Tools: Serper (web search), website scraping, and a LinkedIn scraping tool

Prerequisites

• Python 3.10+ recommended (project uses pyproject.toml and uv for env management)
• LinkedIn account to authenticate scraping
• API keys:
  • OpenAI API key
  • Serper API key (for web search)

Installation

• Using uv (recommended)
  1. Install uv if you don’t have it: https://docs.astral.sh/uv/
  2. From the project root, install dependencies: sh uv sync
• Using pip (alternative)
  1. Create and activate a virtualenv
  2. Install dependencies: sh pip install -e .

Configuration

1. Copy the environment template and fill it in: sh cp .env.example .env
2. Edit .env and provide required values. Exact variable names expected by the app:
   • OPENAI_API_KEY
   • SERPER_API_KEY
   • LINKEDIN_EMAIL
   • LINKEDIN_PASSWORD
   • LINKEDIN_PROFILE_NAME (public handle or unique name of the target profile to scrape)

Run

• From the project root: sh uv run python main.py or, if using a plain venv: sh python main.py The program will:

1. Load environment variables from .env
2. Run the scraping agent to collect recent posts from LINKEDIN_PROFILE_NAME
3. Run the research agent to gather information about GPT‑5 vs Claude
4. Generate a LinkedIn post that emulates the scraped style and print it to the console

Customization

• Change the research topic:
  • File: src/tasks/tasks.py
  • Update the descriptions for web_research_task and create_linkedin_post_task to your desired topic (e.g., Llama 2 vs Llama 3). The default currently compares GPT‑5 and Claude.
• Adjust the writing style behavior:
  • File: src/agents/agents.py
  • Modify the doppelganger_agent backstory/goal to emphasize specific stylistic guidelines.
• Switch models:
  • File: src/agents/agents.py
  • Update the ChatOpenAI model name (currently gpt-5-mini). Ensure your API key has access.

Environment variables reference

• OPENAI_API_KEY: OpenAI API key used by langchain-openai
• SERPER_API_KEY: API key for Serper search tool
• LINKEDIN_EMAIL: Email used to sign in to LinkedIn
• LINKEDIN_PASSWORD: Password used to sign in to LinkedIn
• LINKEDIN_PROFILE_NAME: Target LinkedIn profile to scrape posts from

Project structure

• Root
  • main.py Entry point; creates a Crew and kicks it off
  • src/
    • agents/
      • agents.py Agent definitions (scraper, researcher, post creator)
    • tasks/
      • tasks.py Task definitions and orchestration
    • tools/
      • linkedin.py LinkedIn scraping tool
    • config/
      • config_settings.py pydantic settings mapping .env variables
  • .env.example Environment template
  • pyproject.toml Project configuration and dependencies
  • README.md This file

Troubleshooting

• Missing env vars: The app will fail early if required variables are absent. Double‑check .env matches .env.example.
• Authentication issues scraping LinkedIn: Confirm email/password are correct and that the profile is accessible. Consider 2FA/session challenges which may block scraping.
• Serper errors: Ensure SERPER_API_KEY is set and valid. Watch your rate limits.
• Model access errors: Verify your OpenAI API key and that the model name is available to your account.

Notes on responsible use

• Respect LinkedIn’s Terms of Service and applicable laws. Only scrape content you’re allowed to access.
• Rate limit requests and avoid disrupting target services.

Use cases

• Consistent content creation for personal branding
• Thought‑leadership posts using timely research
• Industry news commentary in your authentic voice
• Announcing professional milestones