reporting-unified

Flask applications for accessing eRSA reporting databases.

Deployment

The package only works with PostgreSQL because it uses:

• UUID
• INET
• MACADDR

Set up a database for a package in development

0. Modify prepare_db.sql.demo and save as prepare_db.sql then run sudo -u postgres psql -f prepare_db.sql
1. Create config.py file
2. Activate env
3. Create tables from either command:

bin/ersa-reporting-prep ersa_reporting.PACKAGE
# or
export APP_SETTINGS=config.py
bin/ersa-reporting-prep-tables PACKAGE

unified package

The package can be served by, for example, nginx (proxy) + gunicorn.

If deploy to a CentOS 7 cloud instance, script can be used to prepare the instance with nginx and gunicorn if one application is being deployed. For multiple application deployment, run service_generator.sh and then gconf_generator.sh. To finish off, in /usr/lib/ersa_reporting, create a config-PACKAGE.py file described in this document and example.

Access and error logs of gunicorn of an application are saved under /var/log/gunicorn/.

An application's log is named as application.log, e.g. 'hnas.log' in /usr/lib/ersa_reporting. Where this log is saved can be configured in config-PACKAGE.py with key LOG_DIR.

Example of gunicorn configuration file generated by gconf_generator.sh:

timeout = 7200
proc_name = PACKAGE
workers = 2
PDIRfile = "/run/gunicorn/PACKAGE.PDIR"
raw_env = ["APP_SETTINGS=config-PACKAGE.py"]
accesslog = "/var/log/gunicorn/PACKAGE_access.log"
errorlog = "/var/log/gunicorn/PACKAGE_error.log"
loglevel = "info"

ersa-reporting package - to be deprecated

The package can be served by, for example, nginx (proxy) + gunicorn.

If deploy to a CentOS 7 cloud instance, script can be used to set up nginx and gunicorn.

The log of an application is currently hard-coded to be saved in /var/log/gunicorn/ which assumes gunicorn is configured to save logs to there.

An application's log is named as ersa_reporting.application.log, e.g. 'ersa_reporting.hnas.log'.

This package contains multiple applications. Main package ersa_reporting expects config.py to have relevant settings:

• SQLALCHEMY_TRACK_MODIFICATIONS
• SQLALCHEMY_DATABASE_URI
• ERSA_REPORTING_PACKAGE
• ERSA_AUTH_TOKEN

configure gunicorn

The packge needs to talk to a database so there is a configuration file for gunicorn to use which is listed below in the example of hnas.conf. It also can be generated by a script like gconf_example. Each web api application served by gunicorn can be configured by a conf file like this:

raw_env = ["ERSA_REPORTING_PACKAGE=hnas", "ERSA_DEBUG=True",
           "ERSA_DATABASE_URI=postgresql://user:pass@host/db",
           "ERSA_AUTH_TOKEN=DEBUG_TOKEN"]
timeout = 7200
proc_name = "hnas"
workers = 2
pidfile = "/run/gunicorn/hnas.pid"
accesslog = "/var/log/gunicorn/hnas_access.log"
errorlog = "/var/log/gunicorn/hnas_error.log"
loglevel = "info"

Development - unified package - the replacement of ersa-reporting

Applications in unified get their configurations from the file set in environment variable APP_SETTINGS. See example for common settings of an API application. For other applications, they can have different keys.

A basic config.py should have these keys:

• SQLALCHEMY_TRACK_MODIFICATIONS
• SQLALCHEMY_DATABASE_URI
• ERSA_REPORTING_PACKAGE
• ERSA_AUTH_TOKEN (optional for development)
• DEBUG = True (optional, make Flask give more error messages)

run an unified application

To interact with the package:

import time
import os
os.environ['APP_SETTINGS'] = 'config-xfs.py'
from unified.models.xfs import Snapshot, Filesystem, Usage, Owner, Host
start = time.time()
x = Snapshot.summarise(1452676846, 1452930321)
end = time.time()
print(end - start)

To serve an application for testing:

# run dev instance by Flask
export APP_SETTINGS=config-xfs.py
export FLASK_APP=unified/apis/xfs.py
flask run -h 0.0.0.0 --reload

# run dev instance by Gunicorn
gunicorn -e APP_SETTINGS=config-xfs.py --access-logfile - -b 0.0.0.0:5000 unified.apis.xfs:app

Scripts

Ingest

The ingest scripts need boto package installed. Typical usage looks like this:

./ingest_hcp.py config_target.json

TODO: put description of ingest here

Biller script biller.py - FIXME with unified models

This script is for generating bills directly. It has a few custom Flask commands. It needs environment variables: APP_SETTINGS and FLASK_APP. For generating Nova bills, it needs credential for accessing Nectar Keystone (not from reporting database). As with other API applications, the config.py should have all keys in those applications, and at least these three keys:

NECTAR_USER = "username"
NECTAR_USER_PASS = "password"
SQLALCHEMY_BINDS = {
    "keystone": "postgresql://apiuser:password@host/keystone-db"
}

flask export_tenants flinders.edu.au

Usage script

Default config is config.json. You can use other by calling with --conf. The structure of config.json looks like this:

{
  "Nova": {
    "APP": "comm-config.py",
    "workers": 3,
    "crm": { "url": "crm" }
  },
  "Hpc": {
    "APP": "hpc-config.py",
    "crm": { "url": "crm" }
  }
}

There are two ways to define a time period for usage calculation:

1. by year and month

python calculator.py -y 2016 -m 8 nova

2. or by start and end:

python calculator.py -s 20160801 -e 20160831 nova

Tests

Tests are run through unittest package. They need a running API application to test against and auth_token to connect to it. auth_token can be the development server token set in config file set by APP_SETTINGS by ERSA_AUTH_TOKEN or a valid token in other set ups.

python -m unittest usage/tests/test_calculator.py
python -m unittest usage.tests.test_calculator.TestUsages.test_filename_of_usage_save
python -m unittest unified.models.tests.test_xfs
python -m unittest unified.tests.test_xfs