An always-on AI agent that watches a Microsoft 365 inbox, understands every new email (and PDF attachment) using Google Gemini, and delivers a clean, founder-friendly briefing straight to WhatsApp within seconds.
- Business use case
- Features
- Architecture
- Tech stack
- Sample output
- Prerequisites
- Local setup
- Environment variables
- Running locally
- Cloud deployment
- Project structure
- How it works (deep dive)
- Spam filtering
- Self-monitoring & alerts
- Troubleshooting
- Roadmap
- Author
- License
Founders, CEOs, and operations leads receive hundreds of emails per day — invoices, partnership pitches, vendor updates, payment confirmations, HR escalations, customer support tickets — and most of the day is lost just triaging the inbox. Opening every mail to figure out which one needs action is slow, draining, and expensive in opportunity cost.
This agent solves that problem.
The user never has to open their inbox again.
Every new email is read, classified, and condensed into a 5–6 line WhatsApp briefing on their phone — formatted exactly the way a busy decision-maker wants to read it. PDF attachments (contracts, invoices, statements) are read end-to-end and the key numbers / terms / dates are pulled into the same briefing, with the original PDF attached.
Built originally as a freelance project for a startup founder who wanted to reclaim 2–3 hours a day spent on email triage.
Who this is for:
- Founders and CEOs who live on WhatsApp
- Executive assistants managing multiple inboxes
- Operations teams who need real-time visibility into vendor / partner emails
- Anyone who treats their inbox as a notification stream rather than a workspace
- Real-time email monitoring — polls Microsoft 365 inbox every 30 seconds via Microsoft Graph API
- AI summarization — uses Google Gemini Flash to produce structured, category-tagged briefings
- Auto-categorization — classifies every email into one of: Payments, Vendor, Updates, HR, Partnership, Legal, Customer, Operations, Marketing, Other
- PDF understanding — extracts text from PDF attachments and summarizes the actual contents (numbers, terms, deadlines), not just the file name
- Link extraction — pulls every URL out of the email body and surfaces them in a dedicated section
- WhatsApp delivery — sends the briefing to the user's WhatsApp via Twilio, with PDFs attached inline
- Smart spam filter — silently skips a configurable list of known noise senders (newsletters, billing receipts, automated notifications)
- Time-aware greeting — opens every message with "Good Morning / Afternoon / Evening" based on IST
- Self-monitoring — alerts the user on WhatsApp if Gemini quota is exhausted, Twilio balance runs out, Azure auth fails, or the internet drops
- Safe character handling — automatically truncates messages over the Twilio 1600-character WhatsApp limit
- Production-tested — runs 24/7 as a Railway worker with auto-recovery on transient failures
┌────────────────────────┐
│ Microsoft 365 Inbox │
│ (any mailbox in your │
│ Azure tenant) │
└───────────┬────────────┘
│ Graph API — poll every 30s
│ filter: unread, received after agent start
▼
┌────────────────────────┐
│ agent.py │
│ ──────────────────── │
│ 1. Fetch new emails │
│ 2. Skip spam senders │
│ 3. Read body + links │
│ 4. Download PDFs │
│ 5. Extract PDF text │
└─────┬────────────┬─────┘
│ │
│ └──────────────┐
▼ ▼
┌────────────────┐ ┌─────────────────────┐
│ Google Gemini │ │ File host │
│ Flash │ │ tmpfiles.org → │
│ │ │ gofile.io → │
│ Structured │ │ file.io │
│ summary │ │ (fallback chain) │
└────────┬───────┘ └──────────┬──────────┘
│ WhatsApp-formatted text │ public PDF URL
└─────────────┬─────────────┘
▼
┌────────────────┐
│ Twilio │
│ WhatsApp API │
└────────┬───────┘
▼
┌────────────────────┐
│ Founder's phone │
│ (WhatsApp) │
└────────────────────┘
Self-monitoring: any failure (Gemini quota, Twilio balance,
Azure auth, no internet) is sent as a system alert to the
same WhatsApp number.
| Layer | Technology | Why |
|---|---|---|
| Language | Python 3.10+ | Mature SDKs for every integration |
| Email source | Microsoft Graph API | Works with any M365 mailbox; uses app-only OAuth (no user password) |
| Auth | Azure AD client-credentials flow | Stateless, no token refresh needed |
| Summarization | Google Gemini gemini-flash-latest |
Cheap, fast, generous free tier, strong instruction following |
| Messaging | Twilio WhatsApp API | Reliable global delivery, supports media attachments |
| PDF parsing | PyPDF2 | Pure-Python, no system dependencies |
| File hosting | tmpfiles.org → gofile.io → file.io | Three-step fallback chain so PDFs always land |
| Scheduler | Native Python time.sleep loop |
Single-process worker, no extra infra |
| Deployment | Railway (Procfile) | One-click long-lived worker, easy env-var management |
| Secrets | python-dotenv + .env |
Standard, gitignored |
A real WhatsApp message produced by the agent:
Good Morning!
*Payments* | FYI: Stripe sent Invoice #INV-9201 paid
*Key Points:*
• Customer "Acme Inc." paid Rs. 84,500 on 14 May
• Net after Stripe fees: Rs. 82,180
• Funds settle to account on 16 May
• Linked subscription: Annual / Pro tier
*Links:*
https://dashboard.stripe.com/invoices/in_1Nq...
For PDF attachments (e.g. a vendor contract), the briefing summarizes the contents of the PDF — pricing, terms, deadlines — not just the filename, and the original PDF is delivered as a WhatsApp attachment in the same message.
You will need accounts on four services. All have free tiers sufficient for testing.
| Service | What you need | Where to get it |
|---|---|---|
| Python | 3.10 or newer | python.org |
| Microsoft Azure | An App Registration with Mail.Read (Application) permission, admin-consented |
portal.azure.com → Azure Active Directory → App registrations |
| Google Gemini | A free API key | aistudio.google.com/app/apikey |
| Twilio | Account SID, Auth Token, and a WhatsApp sender (the sandbox is fine to start) | console.twilio.com |
| Git | Any recent version | git-scm.com |
Estimated setup time: 30–45 minutes the first time (mostly waiting for Azure admin consent).
git clone https://github.com/PJDEEPESH/EmailSummarizeAgent.git
cd EmailSummarizeAgentWindows (PowerShell):
python -m venv venv
venv\Scripts\Activate.ps1macOS / Linux:
python -m venv venv
source venv/bin/activatepip install -r requirements.txtCopy the template:
# Windows
copy .env.example .env
# macOS / Linux
cp .env.example .envOpen .env and fill in your values (see Environment variables below).
All variables live in a .env file at the project root. Never commit this file — it is gitignored.
| Variable | Required | Description |
|---|---|---|
TENANT_ID |
yes | Azure AD tenant ID (Directory ID) of the M365 organization |
CLIENT_ID |
yes | Application (client) ID of the Azure App Registration |
CLIENT_SECRET |
yes | Client secret value from the App Registration |
EMAIL_TO_WATCH |
yes | The mailbox to monitor, e.g. founder@company.com |
GEMINI_API_KEY |
yes | Google Gemini API key from AI Studio |
TWILIO_SID |
yes | Twilio Account SID (starts with AC...) |
TWILIO_TOKEN |
yes | Twilio Auth Token |
FROM_WHATSAPP |
yes | Twilio WhatsApp sender, e.g. whatsapp:+14155238886 |
TO_WHATSAPP |
yes | Recipient WhatsApp number, e.g. whatsapp:+91XXXXXXXXXX |
Azure (TENANT_ID, CLIENT_ID, CLIENT_SECRET)
- Go to portal.azure.com → Azure Active Directory → App registrations → New registration
- Name it
EmailSummarizeAgent, leave defaults, click Register - Copy the Directory (tenant) ID →
TENANT_ID - Copy the Application (client) ID →
CLIENT_ID - Go to Certificates & secrets → Client secrets → New client secret → copy the Value (not the ID) →
CLIENT_SECRET - Go to API permissions → Add a permission → Microsoft Graph → Application permissions →
Mail.Read - Click Grant admin consent (you need to be an admin, or ask one)
Gemini (GEMINI_API_KEY)
- Go to aistudio.google.com/app/apikey
- Click Create API key → copy it
Twilio (TWILIO_SID, TWILIO_TOKEN, FROM_WHATSAPP, TO_WHATSAPP)
- Sign up at twilio.com (free trial gives you $15 credit)
- Copy the Account SID and Auth Token from the Console dashboard
- For
FROM_WHATSAPP: enable the WhatsApp sandbox at Messaging → Try it out → Send a WhatsApp message — the sender number iswhatsapp:+14155238886 - Join the sandbox from your phone (send the join code to that number on WhatsApp)
TO_WHATSAPPis your own WhatsApp number prefixed withwhatsapp:and the country code, e.g.whatsapp:+919581571616
With your .env filled in and venv activated:
python agent.pyExpected output:
[START] Email Agent Running 24/7...
[INFO] Watching: founder@company.com
[INFO] Only processing emails after: 2026-05-24T10:32:00Z
Send yourself a test email to the watched mailbox. Within ~30 seconds you should see:
[NEW] Test Sender <you@gmail.com>
[SUBJECT] Test email
[SUMMARY]
Good Morning!
...
[SUCCESS] WhatsApp sent!
And the WhatsApp briefing should land on your phone.
Tip: the agent only processes emails received after it started. Older unread emails are ignored on purpose, so you don't get a flood of summaries on first launch.
The agent is a long-lived worker (not a web server), so it needs a host that runs a persistent process. The repo includes a Procfile:
worker: python agent.py
- Push the repo to GitHub
- Go to railway.app → New Project → Deploy from GitHub repo → select this repo
- In the Variables tab, paste in every variable from
.env.examplewith your real values - Railway auto-detects the
Procfileand runsworker: python agent.py - Open the Logs tab to confirm
[START] Email Agent Running 24/7...
Cost: free tier is enough for low-volume mailboxes. ~$5/month on the Hobby plan for unlimited runtime.
- Push the repo to GitHub
- render.com → New → Background Worker → connect the repo
- Build command:
pip install -r requirements.txt - Start command:
python agent.py - Add every env var under Environment
# On the server
git clone https://github.com/PJDEEPESH/EmailSummarizeAgent.git
cd EmailSummarizeAgent
python -m venv venv && source venv/bin/activate
pip install -r requirements.txt
# Create .env with your values
# Run under systemd or pm2 so it auto-restarts
pm2 start "python agent.py" --name email-agent
pm2 save
pm2 startupThe project doesn't ship a Dockerfile out of the box, but a minimal one is just:
FROM python:3.11-slim
WORKDIR /app
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt
COPY . .
CMD ["python", "agent.py"]EmailSummarizeAgent/
├── agent.py # The entire agent — poll, filter, summarize, send
├── requirements.txt # Python dependencies (requests, twilio, dotenv, PyPDF2, pytz)
├── Procfile # Railway / Heroku worker definition
├── .env.example # Template for required environment variables
├── .env # Your real secrets (gitignored, NEVER commit)
├── .gitignore # Excludes .env, venv, __pycache__, IDE folders
└── README.md # This file
The whole agent is ~470 lines in a single file, agent.py. The main loop:
- Get an Azure access token via OAuth2 client-credentials flow (
get_access_token) - Fetch unread emails received after the agent's start time (
get_new_emails) - Skip ignored senders by checking
IGNORED_SENDERSset (agent.py:40) - For each new email:
- Extract URLs from the body (
extract_links) - Download attachments and extract PDF text (
get_attachments_content) - Build a structured Gemini prompt with strict formatting rules (
summarize_with_gemini) - Upload any PDFs to a public host with a 3-step fallback chain (
upload_pdf_for_whatsapp) - Send the summary + PDF to WhatsApp via Twilio (
send_whatsapp) - Mark the email as read so it isn't processed twice (
mark_as_read)
- Extract URLs from the body (
- Sleep 30 seconds, repeat.
Failures at any step are caught, counted, and (on the first or third occurrence depending on type) reported back to the same WhatsApp number as a system alert — so you find out about a broken integration before the founder does.
The IGNORED_SENDERS set at the top of agent.py holds email addresses to silently skip. Newsletters and automated senders go here. Add new senders as you encounter them — lowercase, exact match.
IGNORED_SENDERS = {
"noreply@mail.iaapa.org",
"dailynewsletter@iaapa.org",
"info@e.atlassian.com",
...
}Filtered emails are marked-as-read but never summarized or sent to WhatsApp.
The agent watches itself. If something breaks, a *[SYSTEM ALERT]* message goes to the same WhatsApp number:
| Trigger | Alert title |
|---|---|
Gemini returns 429 / RESOURCE_EXHAUSTED |
Gemini API Quota Exhausted |
| Gemini returns 401 / 403 | Gemini API Key Invalid |
| Twilio error code 21608 / "insufficient" / "balance" | Twilio Balance Finished |
| Twilio error code 21211 / "invalid" | Twilio Invalid Number |
Azure auth fails (bad CLIENT_SECRET etc.) |
Azure Auth Failed |
3 consecutive ConnectionErrors |
No Internet |
| Any other unhandled exception | Agent Error |
This means you never have to babysit logs — broken integrations announce themselves.
| Symptom | Likely cause | Fix |
|---|---|---|
Azure auth failed on startup |
Wrong TENANT_ID / CLIENT_ID / CLIENT_SECRET, or admin consent not granted |
Double-check the three values; re-grant admin consent for Mail.Read |
| Emails fetched but no WhatsApp arrives | Twilio sandbox not joined from your phone | Send the join code (e.g. join <two-words>) to +14155238886 on WhatsApp |
| WhatsApp says "Message truncated" | Email body or PDF was too long (>1500 chars after summarization) | This is expected — Twilio has a hard 1600-char limit for WhatsApp |
| PDF attachment not delivered | All three upload hosts failed | Check internet; verify tmpfiles.org / gofile.io / file.io aren't blocked |
Gemini API Quota Exhausted alert |
Free tier rate limit hit | Wait an hour, or upgrade your Gemini plan |
| Newsletter / spam is being summarized | Sender not in IGNORED_SENDERS |
Add the sender email (lowercase) to the set and redeploy |
Things that could be added but aren't (yet):
- Reply-from-WhatsApp (use Gemini to draft a reply and let the user approve via WhatsApp)
- Multi-mailbox support (one agent watching N mailboxes for N users)
- Calendar awareness (skip summarizing emails when the user is in a meeting)
- Slack / Telegram / Discord output adapters
- Per-sender priority routing (urgent senders get a separate "URGENT" prefix)
- Web dashboard for managing the ignore list
Built by Deepesh PJ as a freelance project for a startup founder, then open-sourced as a portfolio piece.
If this helped you or you have feedback, feel free to open an issue or reach out.
MIT — see LICENSE if present, or treat this as MIT-licensed (use freely, attribution appreciated).