AP-1011 archive_load_files feature

[llehtinen]

Problem

https://transferwise.atlassian.net/browse/AP-1011

Proposed changes

When enabled, this feature will result in PipelineWise load files being copied to a separate folder for archival.

When used in conjunction with incremental replication, the replication key and its min/max value in each archived file are included in S3 metadata.

Original grooming document (slightly outdated now):
https://docs.google.com/document/d/11UTlmWVJS9aGickmyxpXOUhrv4RTPnm8zip_IqeR2J0/edit?ts=609e4106

(Old WIP PR: #174)

Types of changes

What types of changes does your code introduce to PipelineWise?
Put an x in the boxes that apply

• Bugfix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause existing functionality to not work as expected)
• Documentation Update (if none of the other choices apply)

Checklist

• Description above provides context of the change
• I have added tests that prove my fix is effective or that my feature works
• Unit tests for changes (not needed for documentation changes)
• CI checks pass with my changes
• Bumping version in setup.py is an individual PR and not mixed with feature or bugfix PRs
• Commit message/PR title starts with [AP-NNNN] (if applicable. AP-NNNN = JIRA ID)
• Branch name starts with AP-NNN (if applicable. AP-NNN = JIRA ID)
• Commits follow "How to write a good git commit message"
• Relevant documentation is updated including usage instructions

[koszti]

Can you please change the multi level archive_load_files.enabled option to be simply archive_load_files? Every option in the config.json is single level and keeping it consistent would make life easier. I added suggestions to the PR above.

Also, can you please document the new archive_load_files optional parameter in README.md in the Config settings section?

[llehtinen]

@koszti Requested changes have been made. In addition, as discussed on Slack, the archive-load-files-primary-column term was replaced with incremental-key in metadata, variable names, and stream_utils method name.