Version 0.5

@sb2nov sb2nov released this Jan 26, 2016 · 53 commits to develop since this release

  • Cleanup commands being passed in QA steps
  • Add support for postgres
  • Status bar for uploading large files
  • Minor bug fixes

Downloads

Dataduct v4

@sb2nov sb2nov released this Nov 19, 2015 · 104 commits to develop since this release

  • Support for starting database shell from dataduct cli
  • Fix bug in logger configuration
  • More performance tuning for analyze and vacumm
  • Improved subject line for SNS messages
  • More informed logging for load errors
  • Improvements to decorators
  • PK enforcement changes
  • New load-reload-pk step
  • Support for worker groups
  • Steps to move away from scripts to all code being contained in the library

Downloads

Dataduct v3

@sb2nov sb2nov released this Sep 22, 2015 · 185 commits to develop since this release

  • More documentation
  • Bug fixes in SQL parser
  • Hooks framework
  • Default bootstrap
  • Teardown
  • Frequency fixes

Downloads

Dataduct v2

@sb2nov sb2nov released this Mar 20, 2015 · 202 commits to master since this release

Changes

  • Added Travis integration for continuous builds
  • Added QA steps and logging to S3
  • Added pipeline visualization command
  • Updated Dataduct CLI to a single entry point
  • Added RDS connections for scripts
  • Added bootstrap step for pipelines
  • Added backfill or delay activation
  • Added utput path and input path options
  • Added ability to specify a script directory for transform steps
  • Added SQL sanitization for DBA actions
  • Added SQL parser for select and create table statements
  • Added logging across the library
  • Introduced support for custom steps
  • Added pipeline dependency step
  • Reduced verbosity of imports
  • Step parsing is now isolated in steps
  • Added more examples for steps
  • Added configuration file sync with S3
  • Added config overrides with modes
  • Renamed keywords and safe config failure handling
  • Added EMR streaming support with hadoop 2
  • Cleaned up exceptions
  • Added read the docs support
  • Tables are now created automatically for various steps
  • Added history table support
  • EC2 and EMR config control from YAML
  • Added slack integration
  • Added support for regions in AWS

Downloads

Version 0.1.0

@sb2nov sb2nov released this Dec 23, 2014 · 447 commits to master since this release

Just to mark the really old release we had in November.

https://pypi.python.org/pypi/dataduct/0.1.0

Downloads