…in ruby 1.9 CSV *is* fasterCSV and codebase has always used fasterCSV -- which is now "CSV".
…s it thru to the block it calls.
bugs in the implementation
instead of the full record. This should save on memory. - Added new processor: FillRowProcessor This is a row processor that allows for filling an existing source row with data merged from another data source. It has options for overwriting the existing data on like columns. It can also add new row columns from the alternate data source.
…scd fields when specifying :unique in a destination, which was also done here
- Allow for multiple environments for database config: database.<env>.yaml - Allow setting of log level by ENV['LOG_LEVEL'] = x. 0 = Debug 1 = Info etc - Added millisecond precision to timestamp method - Use UTC for created_at/completed_at for Batch/Job - Better handling of paths for control files - Beginning (not functional - Syntax broken - Code does not run!!) work on multiprocessing transforms
allow already existing records to be removed from the db and readded (data will continue into output file)
- Fixed minor issues with pathname expecting a string - Added require 'forkify' (working on multiprocessing the transforms) - Added require 'pidify' (for etl script) - Fixed (I think) an issue with "new_records_only" in database_source. Was possible race condition (records with timestamp of same second as last run, but not yet committed when last run began, so missed)
location (timestamped). This allows for concurrent runs against differernt databases from the same codebase (with the same named output files)
correct previous .trig file in source dir