Weather Station Airflow Setup

Apache Airflow setup for collecting weather station data using Docker Compose.

Overview

This project uses Apache Airflow to fetch weather data from the Weather.com PWS (Personal Weather Station) API and store it in a PostgreSQL database. The setup runs entirely in Docker containers using the CeleryExecutor.

Architecture

• Airflow Version: 2.7.0
• Executor: CeleryExecutor
• Database: PostgreSQL 13
• Message Broker: Redis
• Components:
  • Webserver (UI at http://localhost:8080)
  • Scheduler
  • Celery Worker
  • Triggerer
  • PostgreSQL database
  • Redis broker

Prerequisites

• Docker
• Docker Compose (or docker compose plugin)
• Weather.com API key (get one here)

---

Ambient Weather → Weather Underground Setup (Prerequisites)

This guide covers only the steps required to:

• Register your Ambient Weather station
• Connect it to Weather Underground (WU)
• Obtain the API key needed for downstream usage

Once completed, your station will be ready to use with any existing tooling (e.g., cloning and running a prebuilt Docker/Airflow setup).

---

Overview

Data flow established in this setup:

Ambient Weather Station
        ↓
AmbientWeather.net
        ↓ (WU Station ID + Station Key)
Weather Underground / Weather.com

---

Prerequisites

• Ambient Weather–brand weather station (console or gateway)
• Wi-Fi network with internet access
• Account on:
  • AmbientWeather.net
  • Weather Underground

---

Step 1: Register Your Station on AmbientWeather.net

1. Power on your Ambient Weather console or gateway.

2. Connect it to Wi-Fi using the Ambient Weather mobile app or the console's Wi-Fi setup menu.

3. Log into your AmbientWeather.net account.

4. Navigate to:

   https://ambientweather.net/devices
   

5. Confirm your device appears in the Devices list.

   • Verify:
     • Device name
     • MAC address
     • Live data updating

Result: Your station is now actively uploading data to AmbientWeather.net.

---

Step 2: Create a Weather Underground (WU) Station

1. Log into your Weather Underground account.
2. Create a Personal Weather Station (PWS).
3. Record the following credentials:
   • Station ID (example: KMOKANSA763)
   • Station Key (keep this secret)

These credentials uniquely identify your station within the Weather Underground ecosystem.

---

Step 3: Connect AmbientWeather.net to Weather Underground

1. In AmbientWeather.net, open:

   Devices → [Your Device]
   

2. Locate the Weather Underground (WU) integration option.

3. Enter:

   • WU Station ID
   • WU Station Key

4. Save the configuration.

Result: AmbientWeather.net now forwards your station's observations to Weather Underground automatically.

---

Step 4: Verify Weather Underground Data Flow

1. Open your station's public WU page:

   https://www.wunderground.com/pws/<YOUR_STATION_ID>
   

2. Confirm:

   • Current conditions are visible
   • Observation timestamps are updating

This confirms the Ambient → Weather Underground connection is working.

---

Step 5: Obtain a Weather.com / WU API Key

To access your station data programmatically:

1. Log into your Weather.com / Weather Underground account.

2. Navigate to Account / Member Settings.

3. Create or retrieve an API Key for PWS data access.

4. Store the key securely (for example, as an environment variable):

   export WEATHER_API_KEY=your_api_key_here

Note: The API key is associated with your Weather.com / WU account, not directly with AmbientWeather.net.

---

Completion Checklist

• Station visible and updating on AmbientWeather.net
• Weather Underground PWS created
• AmbientWeather.net connected to WU via Station ID + Key
• Live data visible on wunderground.com
• Weather.com / WU API key obtained

Once all items above are complete, your station is ready for use with downstream tooling.

---

Setup Instructions

1. Clone the Repository

git clone git@github.com:Andrew-InTheBox/weather-station.git
cd weather-station

2. Configure Environment Variables

Copy the example environment file and add your API key:

cp .env.example .env

Edit .env and replace your_api_key_here with your actual Weather.com API key:

WEATHER_API_KEY=your_actual_api_key_here

3. Start Airflow

docker compose up -d

This will:

• Pull the required Docker images
• Initialize the Airflow database
• Create the default admin user (username: airflow, password: airflow)
• Start all Airflow services

4. Access the Web UI

Open your browser to http://localhost:8080

• Username: airflow
• Password: airflow

5. Configure Database Connection

Before running the weather DAG, you need to configure the postgres_warehouse connection in Airflow:

1. Go to Admin > Connections in the Airflow UI
2. Add a new connection with ID postgres_warehouse
3. Configure it to point to your target PostgreSQL warehouse

Project Structure

.
├── dags/                   # Airflow DAG definitions
│   └── weather_api.py      # Weather data collection DAG
├── logs/                   # Airflow execution logs (not in git)
├── plugins/                # Custom Airflow plugins
├── config/                 # Additional configuration files
├── docker-compose.yaml     # Docker Compose configuration
├── airflow.sh              # Helper script for running Airflow CLI commands
├── .env                    # Environment variables (configure before use)
└── .env.example            # Example environment file

Weather DAG

The weather_data_dag collects data from two weather stations every 5 minutes:

• KMOKANSA763
• KMOALDRI5

To add more stations, edit dags/weather_api.py and add station IDs to the station_ids list.

Useful Commands

Run Airflow CLI commands

Use the provided helper script:

./airflow.sh <command>

Example:

./airflow.sh dags list

Check service status

docker compose ps

View logs

docker compose logs -f airflow-scheduler
docker compose logs -f airflow-worker

Stop all services

docker compose down

Stop and remove all data

docker compose down -v

Security Notes

• This is a development setup - do not use in production without hardening
• Default credentials are used (airflow/airflow)
• The Fernet key is empty - connections are not encrypted
• API key is loaded from environment variables (never commit the actual key)

Troubleshooting

Permission Issues

If you encounter permission errors with logs or DAG files, ensure AIRFLOW_UID in .env matches your user ID:

echo "AIRFLOW_UID=$(id -u)" > .env

DAG not appearing

• Check that the DAG file is in the dags/ directory
• Wait up to 30 seconds for the scheduler to pick up changes
• Check scheduler logs: docker compose logs airflow-scheduler

License

Apache License 2.0 (following Apache Airflow's license)