[ENG-338] [OATHPIT] Throw S3 into the Oath Pit

[cslzchen]

Ticket

https://openscience.atlassian.net/browse/ENG-338

Purpose

Enable and update the S3 provider for aiohttp 3.

Changes

It is unknown why aiohttp-0.18.x stores HTTP response header names in all-uppercase (e.g. 'CONTENT-LENGTH', 'ETAG', etc.). aiohttp-3.5.x now uses the more popular title case (e.g. 'Content-Length', 'ETAG', etc.). This breaks the old implementation for class S3FileMetadataHeaders whose calling self.raw['ETAG'] throws Key Error. The class and related tests has been updated.

Side effects

No

QA Notes

Here is a list of tests that I have done locally, all of which have passed. Extra notes and quirks are added under each list if there is any. In short, S3 behaves as expected.

• Getting metadata for a file and folder

  • This is automatic when you list files and expand folders

• Downloading (20B, 1 MB, 11 MB, 88 MB, 264MB)

• Uploading (20B, 1 MB, 11 MB, 88 MB, 264MB one-by-one and together)

  • Both contiguous and chunked

• DAZ (a folder containing five files of 20B, 1 MB, 11 MB, 88 MB, 264MB)

• Delete a file and a folder; delete multiple files and folders

• Folder creation and deletion

• Renaming files and folders

  • Similar to ownCloud, after renaming a folder or file (successfully), clicking on the renamed folder triggers the following error.

• Intra move and copy

  • Between two components connected to the same account and the same bucket.

• Inter move and copy

  • Between S3 and OSFStorage
  • Between S3 and another 3rd-party (ownCloud in this case)

• Comments persist with moves

• Revisions

• Updating a file

• Project root is storage root vs. a subfolder

  • N / A: s3 only allow bucket root as OSF project root

Deployment Notes

TBD

[coveralls]

Coverage increased (+4.6%) to 61.524% when pulling 38a2e92 on cslzchen:feature/oathpit-s3 into 61107f3 on CenterForOpenScience:feature/oathpit.

[cslzchen]

Local tests have been done and QA notes updated. Please rebase and fix conflicts.