File Upload and Download

Overview

Initiates a file upload that will allow a single file to be uploaded to the "loading dock" bucket. From a registration standpoint, this endpoint will result in the creation of a new source Business Object Data entry to track the file being uploaded into the loading dock, and also create a separate Business Object Data entry for the second "external" file to track it from an auditing perspective. The result of invoking this endpoint contains credentials that allow the caller to manually upload the file into a specific S3 path in the loading dock. Clients should then upload their file into S3. Once the file is uploaded successfully, the loading dock S3 bucket posts a message to a DM monitored SQS queue which initiates the re-encryption process to the "external" bucket. Once the re-encryption process is done, the loading dock file is deleted and the external file is available.

During each phase of the upload, the source Business Object Data status will change from "UPLOADING" to "RE-ENCRYPTING" to "DELETED" while the target Business Object Data status will change from "UPLOADING" to "RE-ENCRYPTING" to "VALID" or "INVALID". Each of these status changes results in an ESB message being sent. Clients should register an interest in these messages to know when the upload process is complete and to determine the final status of the upload. If there were any problems encountered during the re-encryption process, the final status will be set to "INVALID". In this situation, the client should consider the upload as failed and either notify the user that the upload failed and have them re-try their upload again or potentially re-try the upload themselves if they still have access to the original source file.

A request to initiate a single file upload results in creation of a business object data instance for the specified business object format with the partition value set to a UUID value generated by the Data Management system. That partition value (UUID) would need to be used as part of the business object data key when accessing the following upload/download specific endpoints. For example: /download/single/initiation/namespaces/Application_A/businessObjectDefinitionNames/Definition_A/businessObjectFormatUsages/UPLD/ businessObjectFormatFileTypes/ATCHM/businessObjectFormatVersions/0/ partitionValues/54947df8-0e9e-4471-a2f9-9af509fb5889/businessObjectDataVersions/0

Uploading to custom target storage

By default, a successfully uploaded object will be re-encrypted into a managed S3 bucket. If a different target bucket is desired, the target storage specified in the request must already be created and configured properly. When setting up a new storage (one time setup), the required storage attributes are:

• bucket.name - The name of the S3 bucket. This is not the full ARN, but only the bucket name.
• kms.key.id - The KMS key ID to use to encrypt the object. This must be a full ARN.
• key.prefix.velocity.template - A velocity template to use to generate the prefix of the object.
• download.role.arn - If download feature is desired, you must specify here a full ARN to an IAM role which has the access to GetObject, use the KMS key, and must have a trust relationship to herd's instance role.