Skip to content
No description, website, or topics provided.
Python PLpgSQL Shell
Branch: master
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
dags
etl_sql_code
README.md
dashboard_queries.sql
etl_handler.py
extractor.py
kill_airflow.sh
manual_init.sql
oceanrecordshandler.py
os_create_statements.sql
prod_raw
requirements.txt
spin_osdb_init.py
start_airflow.sh
update_osdb.py

README.md

Spin OceanRecords example database

  1. Set up postgres database on your local
  2. Execute sql statements located in manual_init.sql (those which refers to OceaReocrds operational sysytem)
  3. execute: python spin_osdb_init.py OceanRecordsInit --local-scheduler

notes:

  • make sure you've installed all necessery dependencies (requirements.txt)
  • make sure that in dir when script is executed also prod_raw file exists
  • script will output some temporary luigi related files in tmp_init_files folder

Update OceanRecords database

usage: python update_osdb.py Requires that initial state of databse already exists.

Airflow initialization

  1. Install airflow and any other prerequisite libraries
  2. Set up arflow home dir, e.g.: export AIRFLOW_HOME=~/airflow
  3. Create/edit airflow.cfg file. The first time you run Airflow (e.g. by running airflow command), it will create a config file in $AIRFLOW_HOME. Set load_examples = False, so example dags are not loaded.
  4. In airflow.cfg change executor type to: LocalExecutor
  5. In airflow.cfg modify sqlalchemy connection string to postgres airflow db
  6. Init ariflow database: airflow initdb
  7. Copy dags and etl_sql_code folders into AIRFLOW_HOME
  8. Run airflow by: bash start_airflow.sh &> output.log

notes:

  • all related to DWH scripts in manual_init.sql have to be executed before
  • make sure that all paths are correct. for example I've hardcoded mine local paths in start_airflow.sh
You can’t perform that action at this time.