Investigate the use of S3 versioning to manage updates to assets

[chrisroos]

The Asset Manager API allows you to update an asset. When updating an asset and replacing it with a file that has a different name the Asset Manager will put a redirect in place from the old asset to the new asset. We want to understand whether it's possible to get this behaviour by using the versioning functionality built into S3.

[chrisroos]

@floehopper: I've assigned you to this issue as we discussed just now.

[floehopper]

If you switch on versioning for an S3 bucket, versions are automatically created when you upload a new version of a file to a given key or delete an object. You can access older versions of an object by specifying a versionId parameter in the query string. However, in order for this to work for anonymous public access you need to add the s3:GetObjectVersion action to your bucket policy as well as the s3:GetObject action.

You can see this in action below.

• The latest version of object content: "Wed 2 Aug 2017 13:40:36 BST"
• Pevious version of object content: "Wed 2 Aug 2017 13:33:38 BST"
• The x-amz-version-id response header echoes the version of the object being returned.

Latest version (no versionId specified)

$ curl -s -v "https://gds-asset-manager-spike.s3.amazonaws.com/5980b65c30239d05c1441893"
*   Trying 52.95.150.17...
* Connected to gds-asset-manager-spike.s3.amazonaws.com (52.95.150.17) port 443 (#0)
* TLS 1.2 connection using TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA
* Server certificate: *.s3.amazonaws.com
* Server certificate: DigiCert Baltimore CA-2 G2
* Server certificate: Baltimore CyberTrust Root
> GET /5980b65c30239d05c1441893 HTTP/1.1
> Host: gds-asset-manager-spike.s3.amazonaws.com
> User-Agent: curl/7.43.0
> Accept: */*
> 
< HTTP/1.1 200 OK
< x-amz-id-2: 67osV4gJMDcIKFUXRJZubwyf1KfO80dEMfKmRY+K5ybptdVJcX93MNJdTu6Smckbx7Fmavx+nHI=
< x-amz-request-id: 4A9E1A40BFEE1952
< Date: Wed, 02 Aug 2017 13:22:56 GMT
< Last-Modified: Wed, 02 Aug 2017 12:41:06 GMT
< ETag: "ff8fc1c88853210b4974fc29a4c0d1d2"
< x-amz-version-id: YpbV_QtwZpYikNb97jixGPGNlqbgue1u
< Accept-Ranges: bytes
< Content-Type: binary/octet-stream
< Content-Length: 28
< Server: AmazonS3
< 
Wed 2 Aug 2017 13:40:36 BST
* Connection #0 to host gds-asset-manager-spike.s3.amazonaws.com left intact

Latest version (versionId specified)

$ curl -s -v "https://gds-asset-manager-spike.s3.amazonaws.com/5980b65c30239d05c1441893?versionId=YpbV_QtwZpYikNb97jixGPGNlqbgue1u" 
*   Trying 52.95.149.1...
* Connected to gds-asset-manager-spike.s3.amazonaws.com (52.95.149.1) port 443 (#0)
* TLS 1.2 connection using TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA
* Server certificate: *.s3.amazonaws.com
* Server certificate: DigiCert Baltimore CA-2 G2
* Server certificate: Baltimore CyberTrust Root
> GET /5980b65c30239d05c1441893?versionId=YpbV_QtwZpYikNb97jixGPGNlqbgue1u HTTP/1.1
> Host: gds-asset-manager-spike.s3.amazonaws.com
> User-Agent: curl/7.43.0
> Accept: */*
> 
< HTTP/1.1 200 OK
< x-amz-id-2: /FMbPA+Ucg9kvzrSq5ybI4tLE1uP9hMc9cKxLq9pz2fio18PtoYFgQe81V/2hu9i4DGv0Xoyjb4=
< x-amz-request-id: 7BD0C6D4C8B47CE0
< Date: Wed, 02 Aug 2017 13:23:48 GMT
< Last-Modified: Wed, 02 Aug 2017 12:41:06 GMT
< ETag: "ff8fc1c88853210b4974fc29a4c0d1d2"
< x-amz-version-id: YpbV_QtwZpYikNb97jixGPGNlqbgue1u
< Accept-Ranges: bytes
< Content-Type: binary/octet-stream
< Content-Length: 28
< Server: AmazonS3
< 
Wed 2 Aug 2017 13:40:36 BST
* Connection #0 to host gds-asset-manager-spike.s3.amazonaws.com left intact

Previous version

$ curl -s -v "https://gds-asset-manager-spike.s3.amazonaws.com/5980b65c30239d05c1441893?versionId=XWXpp0hssoe1f5t535qZAz7jI_FqUwBE"
*   Trying 52.95.149.21...
* Connected to gds-asset-manager-spike.s3.amazonaws.com (52.95.149.21) port 443 (#0)
* TLS 1.2 connection using TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA
* Server certificate: *.s3.amazonaws.com
* Server certificate: DigiCert Baltimore CA-2 G2
* Server certificate: Baltimore CyberTrust Root
> GET /5980b65c30239d05c1441893?versionId=XWXpp0hssoe1f5t535qZAz7jI_FqUwBE HTTP/1.1
> Host: gds-asset-manager-spike.s3.amazonaws.com
> User-Agent: curl/7.43.0
> Accept: */*
> 
< HTTP/1.1 200 OK
< x-amz-id-2: 3ZLEnAXrdauvTsQ7PGmf5+FHqJN2ROHX3m+tPe60Mm0R/66/omN36yf0CJ+ttfK09+/e7dnLeBc=
< x-amz-request-id: 643A64A530960968
< Date: Wed, 02 Aug 2017 13:24:11 GMT
< Last-Modified: Wed, 02 Aug 2017 12:34:17 GMT
< ETag: "522eb7569487e3dcfbc035b37633e767"
< x-amz-version-id: XWXpp0hssoe1f5t535qZAz7jI_FqUwBE
< Accept-Ranges: bytes
< Content-Type: binary/octet-stream
< Content-Length: 28
< Server: AmazonS3
< 
Wed 2 Aug 2017 13:33:38 BST
* Connection #0 to host gds-asset-manager-spike.s3.amazonaws.com left intact

[floehopper]

I think these investigation tasks are "done", although we will need to decide how/whether we're going to handle updating assets. I plan to create a new issue or two to capture that.