How It Works

How It Works

JobRadar is one Python process. Every step is a task in a small queue stored in SQLite (data/jobradar.db), so a crash or Ctrl+C never loses work: unfinished tasks run again on the next start.

flowchart LR
    TG["Telegram channels"] -->|read-only| SRC["Telegram reader"]
    SRC -->|"save post first"| DB[("SQLite")]
    DB --> PM["process_message<br/>links → jobs"]
    PM -->|new job| UP1["notion_upsert<br/>create page"]
    PM -->|new job| F["fetch<br/>job page"]
    F --> X["extract<br/>Gemini"]
    X --> UP2["notion_upsert<br/>fill columns"]
    X --> B["notion_body<br/>details"]
    UP1 & UP2 & B --> N["Notion"]

Loading

1. Reading Telegram

• JobRadar logs in as you (bots can't read channels they don't run) and only reads the chats in your config.
• It starts the live listener first, then catches up on each chat from the last message it saw (last_msg_id). The very first run reads the last backfill_days days.
• Every post is written to SQLite before anything else happens. One row per chat and message id, so seeing a post twice is harmless.
• Telegram service notices (channel created, photo changed) are skipped.

2. Links become jobs

1. URLs are collected from the text, Telegram's link entities and "Apply" buttons.
2. Short links (bit.ly, lnkd.in, forms.gle, go.acciojob.com…) are expanded by following redirects, through public internet addresses only.
3. Each link is cleaned: tracking codes (utm_*, fbclid…), www., #fragments and trailing slashes are removed.
4. Non-job links (t.me, YouTube, social profiles, your links.deny rules) are dropped.
5. The job id is sha1(clean URL). A link seen before adds a sighting to the existing job instead of a new row.

Channel ads are hidden automatically: when one link is attached to posts about three or more different jobs, it's a WhatsApp group, app, course or YouTube promo, not a job. Posts are compared by their other links (titles as a fallback); links that travel with it in most posts and "Today's Job Updates" digests don't count.

3. Reading the job (needs an LLM key)

Fetch: public address check → robots.txt → at most one request per second per site → download (every redirect re-checked) → main text via trafilatura. Login walls, 404s and PDFs give no page text; temporary errors retry, and after the 4th failure the post text is used.

Extract: the post, its date and the page text go into a versioned prompt (prompts/extract.md) as untrusted data. Gemini returns JSON, which is validated (small slips are tidied, one fix attempt otherwise). Answers are cached, so a job is never paid for twice. Daily spend is capped; when the cap is reached, LLM work waits until midnight while listing carries on.

If the answer has low confidence and no company or role, it's not a job: the row is set to Hidden.

4. Notion

• A page is created as soon as a new job is found, titled with the post's first line.
• After extraction the title becomes Role @ Company, and the columns and the JobRadar toggle (summary, eligibility, skills, original post, where it was seen) are filled.
• Status is set only on creation, or when JobRadar hides a job. Everything else on the page is yours.
• Calls are paced at 2.5 per second; Notion's "slow down" replies are honoured.

Data stored locally

Table	Holds
source	each watched chat and the last message read
raw_message	every post, with its links
job / job_source	one row per job, and every post that mentioned it
fetched_page	the main text of each job page
llm_cache	Gemini answers and their estimated cost
task	the work queue: pending, running, done, failed