Use .env
files specifying environment (.env.development
, .env.production
) and local (.env.local
) via dotenv-flow
. Local .env
files are not tracked in the repo.
Required variables
# api server port (local development only)
PORT
# aws region
REGION
# bucket name for storing data and images
BUCKET
# database connection secret
DB_SECRETS_NAME
The database schema is managed using knex
migrations and seeds .
Database migrations (db/migrations
) keep the schema synchronized across environments.
export NODE_ENV=development # set NODE_ENV prior to running migrations (important!)
knex migrate:latest
knex migrate:rollback
knex migrate:up
knex migrate:down
knex migrate:make name_in_snake_case
Database seeds (db/seeds/development
) populate the database with initial data (currently only used for development).
# set NODE_ENV prior to running migrations (important!)
export NODE_ENV=development
knex seed:run
Example datasets can be generated for existing (production) stations using the r/scripts/export-dev-station.R
script. The output from that script can be place in the db/seeds/development/data/users/<userId>/stations
folder to add a station and its processed datasets and imagesets to an existing user. Be sure to sync the <station code>/storage
folder to the S3 storage bucket for development.
cd db/seeds/development/data/users/<userId>/stations/<station code>
aws s3 sync storage s3://<bucket name>/ --exclude "*" --include "imagesets/*" --include "datasets/*"
Run development server via nodemon
.
cd api
npm start
The cli.js
file defines a CLI for managing the database.
To make it globally available:
npm link
Be sure to set the NODE_ENV
environmental variable.
export NODE_ENV=development
Now access the CLI through the fpe
program.
fpe help
fpe stations list
Or for brief usage, set the NODE_ENV
inline:
NODE_ENV=development fpe stations list
Create a station.
$ fpe stations create -u 10b8ece7... -n "My Station" -l 42.123 -g -72.394
ID | USER_ID | NAME | DESCRIPTION | LATITUDE | LONGITUDE
2 | 1 | My Station | | 42.123 | -72.394
Create an image set for new station, and upload image files from specified folder.
$ fpe imagesets create cli/test/images -s 2
folder listed (n files=10)
configuration validated
processing image PICT0001.JPG (1/10)
processing image PICT0002.JPG (2/10)
processing image PICT0003.JPG (3/10)
processing image PICT0004.JPG (4/10)
processing image PICT0005.JPG (5/10)
processing image PICT0006.JPG (6/10)
processing image PICT0007.JPG (7/10)
processing image PICT0008.JPG (8/10)
processing image PICT0009.JPG (9/10)
processing image PICT0010.JPG (10/10)
images processed (n images=10)
imageset saved to db (id=11)
Create dataset and upload specified file.
$ fpe datasets create -s 1 -t datetime -v "flow_cfs"="FLOW_CFS" -v "stage_ft"="STAGE_FT" cli/test/dataset.csv
dataset parsed (n rows=6)
configuration validated
series generated (n series=2)
dataset saved to db (id=53)
To manage files in S3 using the aws cli.
export BUCKET=<bucket name>
# list all files
aws s3 ls s3://${BUCKET}/ --recursive
# list dataset files
aws s3 ls s3://${BUCKET}/datasets/ --recursive
# list imageset uuids (no files)
aws s3 ls s3://${BUCKET}/imagesets/
# delete all image files for imageset uuid
aws s3 rm s3://${BUCKET}/ --recursive --exclude "*" --include "imagesets/<uuid>/*"
# delete all csv files for dataset uuid
aws s3 rm s3://${BUCKET}/ --recursive --exclude "*" --include "datasets/<uuid>/*"
- AWS Security Best Practices Part 2 (Uploading)
- robmclarty/knex-express-project-sample: Express + Knex template
- Fodark/express-knex-objection: Express + Knex + Objection template
- Lambda payload size workaround: how to deal with lambda payload limits (6 MB)