Merge branch 'develop' into feature/remove-unsizable-response-reader-…

…flag

.mailmap

@@ -14,6 +14,7 @@ Joshua Carp <jm.carp@gmail.com> jmcarp <jm.carp@gmail.com>
 Kevin Wierman <kwierman@gmail.com>
 Kushagra Gupta <imkushagra@gmail.com> kushG <imkushagra@gmail.com>
 Lauren Revere <lauren.revere@gmail.com>
+Longze Chen <cs.longze.chen@gmail.com>
 Lyndsy Simon <lyndsy@lyndsysimon.com>
 Matt Frazier <maf7sm@virginia.edu>
 Michael Haselton <icereval@gmail.com> <michael.haselton@gmail.com>
@@ -24,5 +25,3 @@ Robert Liebowitz <rliebz@gmail.com> rliebz <rliebz@gmail.com>
 Sam Chrisinger <s.chrisinger@gmail.com>
 Steven Loria <sloria1@gmail.com>
 Tom Baxter <tkb608@gmail.com> TomBaxter <tkb608@gmail.com>
-
-

CHANGELOG

@@ -2,6 +2,51 @@
 ChangeLog
 *********
 
+0.36.0 (2017-12-05)
+===================
+- Feature: WaterButler now supports two new read-only providers, GitLab and Microsoft OneDrive!
+Read-only providers support browsing, downloading, downloading-as-zip, getting file revision
+history, and copying from connected repositories.  Thanks to the following devs for their hard
+work!
+ - GitLab: @danielneis, @luismulinari, @rafaeldelucena
+ - OneDrive: @caseyrygt, @alexandr-melnikov-dev-pro, @johnetordoff
+
+0.35.0 (2017-11-13)
+===================
+- Feature: Allow copying from public resources with the OSF provider.  WaterButler had been
+requiring write permissions on the source resource for both moves and copies, but copy only needs
+read.  Update the v1 API to distinguish between the two types of requests.
+- Docs: Document supported query parameters in the v1 API.
+- Code: Improve test coverage for osfstorage and figshare.
+- Code: Cleanups for Box, Google Drive, and GitHub providers.
+- Code: Don't include test directories in module search paths.
+- Code: Don't let query parameters override the HTTP verb in v1.
+
+0.34.1 (2017-10-18)
+===================
+- Fix: Don't crash when a file on Google Drive is missing an md5 in its metadata.  This occurs
+for non-exportable files like Google Maps, Google Forms, etc.
+
+0.34.0 (2017-09-29)
+===================
+- ANNOUNCEMENT!  Sadly, the WaterButler v0 API is now *undeprecated*.  We've discovered that the
+OSF is still using it in a few places, so it's been given a temporary reprieve.  Once those are
+converted to v1, v0 will be re-deprecated then removed after an appropriate warning period.
+- Feature: For providers that return hashes on upload, WaterButler will calculate the same hash
+as the file streams and throw an informative error if its hash and the provider's hash differ.
+- Fix: Stop throwing exceptions when building exceptions to throw.  Pickled exceptions are
+resurrected in a peculiar fashion that some of WaterButler's exception classes could not survive.
+- Fix: Validate that the move/copy destination path is really a folder.
+- Fix: Update the Box and Google Drive intra-{move,copy} actions to include children in the
+returned metadata for folders (and document it).
+- Fix: Release Box responses on error.
+- Code: Update the Postman test suite to include CRUD and move tests.
+- Code: Start testing with python-3.6 on Travis.
+- Code: Improve test coverage for all providers except osfstorage and figshare (coming soon!).
+- Code: Teach WaterButler to listen for a SIGTERM signal and exit immediately upon receiving it.
+This bypasses the 10 second wait for shutdown when running it in Docker.
+- Code: Fix sphinx syntax errors in the WaterButler docs.
+
 0.33.1 (2017-09-05)
 ===================
 - Reject requests for Box IDs if the ID is valid, but the file or folder is outside of the

PULL_REQUEST_TEMPLATE.md

@@ -0,0 +1,40 @@
+<!-- Use the following format for the title of the Pull Request:
+
+    [Status] [Ticket] Title
+
+    - For PR ready for review, no need for status
+    - For PR in progress, use [WIP]
+    - For PR on hold, use [HOLD]
+-->
+
+<!-- Before submit your Pull Request, make sure you picked the right branch:
+
+    - For hotfixes, select "master" as the target branch
+    - For new features and improvements, select "develop" as the target branch
+-->
+
+<!-- For security related tickets, talk with the team lead before submit your PR -->
+
+## Ticket
+
+<!-- Link to JIRA ticket, if applicable e.g. https://openscience.atlassian.net/browse/SVCS-1234 -->
+
+## Purpose
+
+<!-- Describe the purpose of your changes -->
+
+## Changes
+
+<!-- Briefly describe or list your changes  -->
+
+## Side effects
+
+<!-- Any possible side effects? -->
+
+## QA Notes
+
+<!-- If applicable, briefly describe how QA should test this ticket/PR -->
+
+## Deployment Notes
+
+<!-- Any special configurations for deployment? -->

docs/api.rst

@@ -140,6 +140,8 @@ Move and copy actions both use the same request structure, a POST to the move ur
 
 Files and folders can also be moved between nodes and providers. The resource parameter is the id of the node under which the file/folder should be moved. It must agree with the path parameter, that is the path must identify a valid folder under the node identified by resource. Likewise, the provider parameter may be used to move the file/folder to another storage provider, but both the resource and path parameters must belong to a node and folder already extant on that provider. Both resource and provider default to the current node and providers.
 
+The return value for a successful move or copy will be the metadata associated with the file or in the case of foldersm the metadata associated with that folder and its immediate children.
+
 If a moved/copied file is overwriting an existing file, a 200 OK response will be returned. Otherwise, a 201 Created will be returned.
 
 **Delete (file, folders)**
@@ -156,4 +158,179 @@ To delete a file or folder send a DELETE request to the delete link. Nothing wil
 Magic Query Parameters
 ----------------------
 
-* ``direct``: issuing a download request with a query parameter named ``direct`` indicates that WB should handle the download, even if a redirect would be possible (e.g. osfstorage and s3).  In this case, WB will act as a middleman, downloading the data from the provider and passing through to the requestor.
+Provider Handler Params
++++++++++++++++++++++++
+
+These query parameters apply to all providers. These are used, along with the request method, to specify what operation to perform, whether to upload, download, move, rename .etc.
+
+
+meta
+****
+
+Indicates that WaterButler should return metadata about the file instead of downloading the contents.  Not necessary for folders, which return metadata by default.
+
+* **Type**: flag
+* **Expected on**: ``GET`` requests for files
+* **Interactions**:
+
+  * ``revisions`` / ``versions``: ``meta`` takes precedence.  File metadata is returned, the revision list is not.
+  * ``revision`` / ``version``: These are honored and passed to the the metadata method.  Metadata for the file at the specified revision is returned.
+
+* **Notes**:
+
+  * The ``meta`` query parameter is not required to fetch folder metadata; a bare ``GET`` folder request suffices. To download a folder, the ``zip`` query parameter should be provided.
+
+
+zip
+***
+
+Tells WaterButler to download a folder's contents as a .zip file.
+
+* **Type**: flag
+* **Expected on**: ``GET`` requests against folder paths
+* **Interactions**:
+
+  * Take precendence over all other query parameters, which will be ignored.
+
+* **Notes**:
+
+  * A ``GET`` request against a folder with no query parameters will return metadata, but the same request on a file will download it.
+
+
+kind
+****
+
+Indicates whether a ``PUT`` request should create a file or a folder.
+
+* **Type**: string, either "``file``" or "``folder``", defaulting to "``file``"
+* **Expected on**: ``PUT`` requests
+* **Interactions**: None
+* **Notes**:
+
+  * Issuing a ``PUT`` request against a file with ``?kind=folder`` will always fail, throwing a ``400 Bad Request``.
+
+
+name
+****
+
+Indicates the name of the file or folder to be created.
+
+* **Type**: string
+* **Expected on**: ``PUT`` requests for folders
+* **Interactions**: None
+* **Notes**:
+
+  * The ``name`` parameter is only valid when creating a new file or folder.  Including it in a ``PUT`` request against a file will result in a ``400 Bad Request``.  Renaming files is done with ``POST`` requests.
+
+
+revisions / versions
+********************
+
+Indicates the user wants a list of metadata for all available file revisions.
+
+* **Type**: flag
+* **Expected on**: ``GET`` for file paths
+* **Interactions**:
+
+  * Both parameters are overridden by the ``meta`` parameter.  Neither should be used with other parameters.
+  * ``revisions`` and ``versions`` are currently used interchangeably, with ``versions`` taking precedence if both are provided.
+
+* **Notes**:
+
+  * The pluralization is vital, ``version`` and ``revision`` are used for identifying particular versions.
+
+
+revision / version
+******************
+
+This is the id of the version or revision of the file or folder which Waterbuter is to return.
+
+* **Type**: int
+* **Expected on**: ``GET`` or ``HEAD`` requests for files or folders
+* **Interactions**:
+
+  * is used as a parameter of the metadata provider function.
+
+* **Notes**:
+
+  * If both are provided, ``version`` takes precendence over ``revision``.
+  * ``revision`` and ``version`` can be used interchangeably. Comments within the code indicate ``version`` is preferred, but no reason is supplied.
+  * Note the lack of pluralization.
+
+
+direct
+******
+
+Issuing a download request with a query parameter named ``direct`` indicates that WB should handle the download, even if a direct download via redirect would be possible (e.g. osfstorage and s3). In this case, WB will act as a middleman, downloading the data from the provider and passing it through to the requestor.
+
+* **Type**: flag
+* **Expected on**: ``GET`` file paths
+* **Interactions**: None
+* **Notes**:
+
+  * Only supported by/relevant to OwnCloud, osfstorage (Cloudfiles), and S3.`
+
+
+displayName
+***********
+
+When downloading a file, sets the name to download it as.  Replaces the original file name in the Content-Disposition header.
+
+* **Type**: string
+* **Expected on**: ``GET`` download requests for files
+* **Interactions**: None
+* **Notes**:
+
+  * Currently only supported by S3.
+
+
+mode
+****
+
+Indicates if a file is being downloaded to be rendered. Outside OSF's MFR this isn't useful.
+
+* **Type**: string
+* **Expected on**: ``GET`` requests for files
+* **Interactions**: None
+* **Notes**:
+
+  * ``mode`` is only used by the osfstorage provider for MFR.
+
+
+confirm_delete
+**************
+
+WaterButler does not permit users to delete the root folder of a provider, as this would break the connection between the resource and the storage provider.  This request has been repurposed to recursively delete the entire contents of the root, leaving the root behind.  For safety, this request requires an additional query parameter ``confirm_delete`` to be present and set to ``1``.
+
+* **Type**: bool
+* **Expected on**: ``DELETE`` requests against a root folder
+* **Interactions**: None
+* **Notes**:
+
+  * Currently supported by: Figshare, Dropbox, Box, Github, S3, Google Drive, and osfstorage
+
+
+Auth Handler Params
++++++++++++++++++++
+
+These query parameters are relayed to the auth handler to support authentication and authorization of the request.
+
+
+cookie
+******
+
+Allows WaterButler to authenticate as a user using a cookie issued by the auth handler.
+
+* **Type**: string
+* **Expected on**: All calls
+* **Notes**: This is a legacy method of authentication and will be discontinued in the future.
+
+
+view_only
+*********
+
+OSF-specific parameter used to identify special "view-only" links that are used to give temporary read access toa protected resource.
+
+* **Type**: string
+* **Expected on**: ``GET`` requests for files or folders
+* **Notes**: Only used internally for the Open Science Framework.

docs/provider.gitlab.rst

@@ -0,0 +1,8 @@
+GitLab Provider
+===============
+
+.. autoclass:: waterbutler.providers.gitlab.provider.GitLabProvider
+    :members:
+    :undoc-members:
+    :show-inheritance:
+    :inherited-members:

docs/provider.onedrive.rst

@@ -0,0 +1,8 @@
+OneDrive Provider
+=================
+
+.. autoclass:: waterbutler.providers.onedrive.provider.OneDriveProvider
+    :members:
+    :undoc-members:
+    :show-inheritance:
+    :inherited-members:

docs/providers.rst

@@ -11,7 +11,9 @@ Providers
    provider.figshare
    provider.filesystem
    provider.github
+   provider.gitlab
    provider.googledrive
+   provider.onedrive
    provider.osfstorage
    provider.owncloud
    provider.cloudfiles

docs/waterbutler.providers.rst

@@ -12,6 +12,7 @@ Subpackages
     provider.figshare
     provider.filesystem
     provider.github
+    provider.gitlab
     provider.googledrive
     provider.osfstorage
     provider.s3

setup.py

@@ -45,13 +45,15 @@ def parse_requirements(requirements):
             'figshare = waterbutler.providers.figshare:FigshareProvider',
             'filesystem = waterbutler.providers.filesystem:FileSystemProvider',
             'github = waterbutler.providers.github:GitHubProvider',
+            'gitlab = waterbutler.providers.gitlab:GitLabProvider',
             'bitbucket = waterbutler.providers.bitbucket:BitbucketProvider',
             'osfstorage = waterbutler.providers.osfstorage:OSFStorageProvider',
             'owncloud = waterbutler.providers.owncloud:OwnCloudProvider',
             's3 = waterbutler.providers.s3:S3Provider',
             'dataverse = waterbutler.providers.dataverse:DataverseProvider',
             'box = waterbutler.providers.box:BoxProvider',
             'googledrive = waterbutler.providers.googledrive:GoogleDriveProvider',
+            'onedrive = waterbutler.providers.onedrive:OneDriveProvider',
         ],
         'waterbutler.providers.tasks': [
             'osfstorage_parity = waterbutler.providers.osfstorage.tasks.parity',