Open Pension parsing framework
Switch branches/tags
Clone or download
Pull request Compare This branch is 32 commits ahead, 1 commit behind SoapSeller:master.
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
common @ 5894cb0
excel
sql
tests
xlsx
.gitignore
.gitmodules
.travis.yml
CSVWriter.js
Cakefile
LICENSE
LevDistance.js
README.md
config.json
db.js
dbLoader.js
detectors.js
dirs.js
elasticLoader.js
fetcher.common.js
fetcher.harel.js
fetcher.js
files_loader.js
genericImporter.js
logger.js
op.js
package.json
quarter.js
test.js
utils.js
validator.js
xlsxparser.js

README.md

Open Pension

Build Status

Import pension "single asset" files into unified DB.

Installation

Prerequisites

ssconvert

sudo apt-get install gnumeric

Clone and initialize submodules

git clone ...
git submodule init
git submodule update
npm install

Commands

Usage: $ node op.js [command] [options]

  Commands:

    convert-files [options]     convert excel files to csv
    db-create-table [options]   create table in database
    db-empty-table [options]    truncate table in database
    db-load-files [options]     load csv files to database
    fetch-google [options]      download files in Google Doc.
    recreate-elastic-indices    copy database data to elastic search.

Convert Excel files to CSV

Usage: $ node op.js convert-files [options]

  Options:

    -y, --year <year>         year
    -q, --quarter <quarter>   quarter
    -b, --body <body>         body
    -f, --fund <fund number>  fund
    -s, --srcdir <name>       path of Excel files, default:./excel/
    -t, --trgdir <name>       path of CSV files, default:./csv/
    -o, --overwrite           overwrite existing CSV files, default: false

Create Empty DB Table

Usage: $ node op.js db-create-table [options]

  Options:

    -t, --table <name>  table name

Truncate DB Table

Usage: $ node op.js db-empty-table [options]

  Options:

    -t, --table <name>  table name

Load CSV Files to Database

Usage: $ node op.js db-load-files [options]

  Options:

    -y, --year <year>           year
    -q, --quarter <quarter>     quarter
    -b, --body <body>           body
    -f, --fund <fund number>    fund number
    -t, --table <name>          table name
    -s, --srcdir <name>         path of CSV files, default:./csv/
    -c, --concurrency <number>  number of concurrent DB connections, defaults to 4

Fetch Excel files from Google Sheet

Usage: $ node op.js fetch-google [options]

  Options:

    -y, --year <year>         year
    -q, --quarter <quarter>   quarter
    -b, --body <body>         body
    -f, --fund <fund number>  fund number
    -t, --trgdir <name>       path of Excel files, default:./excel/
    -o, --overwrite           overwrite existing Excel files, default: false

License

See LICENSE file.