New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Bl 1195 move sftp to s3 #191

Merged

sensei100 merged 2 commits into qa from BL-1195-move-sftp-to-s3

Jun 5, 2020

Contributor

bibliotechy commented Jun 4, 2020

Adds a new dag to automate

copying the most recent almasftp dump files from sftp server to s3 in a namespace based on the timestamp of the files
archiving the moved almasftp files to a namedspaced location in the sftp server so they can be retrieved if needed

It does this by adding a custom operator for batches of sftp to s3 that subclasses the airflow SFTPtoS3Operator.

Chad Nelson added 2 commits

June 3, 2020 09:46


          WIP - Almost done sftp to s3 in batch

2c58c15


          Add tests and archiving moved files

eb5ef87

sensei100 approved these changes

View reviewed changes

sensei100 merged commit fa78fb4 into qa

sensei100 deleted the BL-1195-move-sftp-to-s3 branch

June 5, 2020 13:58

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet