How to Populate DEMAND_SNAPSHOTS table(Daily Batch)

1. Populating DEMAND_SNAPSHOTS (Daily Batch)

DEMAND_SNAPSHOTS is the only table not filled by user actions — it's built by a scheduled job that runs once a day. The work splits across three separate layers, and keeping them separate is the key idea:

• When (scheduling) → cron, part of the operating system. Not Python, not the DB.
• What (the logic) → a Python Django management command. It calls Adzuna, reads the count per skill, and writes the rows.
• Where (the result) → the DEMAND_SNAPSHOTS table. The DB is a passive destination; it never triggers or runs anything.

flowchart TD
  cron["cron (OS)<br/>When · runs at 12:00 daily"]
  cmd["Django command<br/>What · capture_snapshots"]
  api["Adzuna API<br/>live counts"]
  db[("DEMAND_SNAPSHOTS<br/>Where · stores rows")]
  cron -->|triggers| cmd
  api -->|counts| cmd
  cmd -->|writes rows| db

Loading

The command

Write the logic as a Django management command so it can be tested on its own:

python manage.py capture_snapshots

The schedule

Then have cron call it at noon (crontab -e):

0 12 * * * cd /path/to/backend && /path/to/venv/bin/python manage.py capture_snapshots >> /var/log/snapshots.log 2>&1

Timezone: cron uses the server clock (usually UTC). For noon in Perth (UTC+8), either use 0 4 * * *, or pin CRON_TZ=Australia/Perth at the top of the crontab.

Idempotency: write with update_or_create keyed on (skill, captured_at), so an accidental double-run overwrites instead of inserting duplicate rows for the same day.

Why not schedule it inside the DB?

Some databases can schedule jobs (PostgreSQL pg_cron, MySQL events), but avoid that here: the job calls an external API and holds application logic, which belongs in Python — not inside the database.

---