Permalink
Switch branches/tags
Find file
Fetching contributors…
Cannot retrieve contributors at this time
29 lines (25 sloc) 1.07 KB
TODO
* Add build-in support for audit_dimension
* Do not rerun the processing if it isn't needed, i.e. the source and control files have not been modified (allow forced override)
* Provide greater control in error handling
** Allow a error threshold
** Don't die completely if a parse error, just stop processing that specific file if error threshold is reached
** Allow mismatch row length error in delimited parser to be ignored
* Improve error messages throughout, but especially in problems with the control files
* Add support for paritioned views during the insert process. Use specifiable columns as the trigger columns for determining the data output destination.
* Check if a temp table exists and the last job run was successful, in which case skip during the current run
* Create models for each of the tables in each of the databases defined in ETL::Engine.connections
Audit Record
Process-Level
* Start Time
* End Time
* (Duration)
* Rows Read
* Rows Written
* Rows Rejected
* Errors
* Destination
Record-Level
* Source
* Timestamp
* Transformation Log