Skip to content

Storage backend compatibility

Andrew Gaul edited this page Feb 13, 2019 · 29 revisions

Some storage backends do not support all the features of S3:

Atmos

  • does not return meaningful ETag
  • does not support multi-part upload
  • only supports object listing when delimiter is "/" -- does not support recursive listing
  • only supports object listing with prefixes when the prefix matches a directory name

Azure

  • limits container names to 63 characters and restricts character set to lowercase alphanumeric and dash
  • limits metadata keys to "C# identifiers". As far as I can tell, that's alphanumeric characters and dash; must not start with a number
  • does not support blob names with \ or with sequences of ./ and /.
  • maximum multipart upload size of 4.77 TB
  • does not support Expires header but does support Cache-Control header
  • does not support object-level access control
  • does not allow arbitrary list markers, partially emulated by mapping the last key to the opaque marker
  • does not return ETag as an MD5 hash, instead returning opaque data

B2

  • bucket names must be between 6 and 50 characters and consistent of only letters, number, and hyphens
  • does not return ETag as an MD5 hash, instead returning SHA1 hash
  • does not support blob-level access control
  • does not support conditional GET
  • does not support copying objects
  • does not support Content-Disposition, Content-Encoding, Content-Language, Content-MD5, or Expiry headers
  • multipart upload requires at least 2 parts

filesystem

Google Cloud Storage

  • does not allow arbitrary list markers, partially emulated by mapping the last key to the opaque marker
  • does not return ETag as an MD5 hash, instead returning opaque data
  • does not support listing in-progress multipart uploads
  • limits multi-part uploads to 32 parts

OpenStack Swift

  • does not support Content-Language headers
  • does not support copying objects with the If-None-Match header
  • does not support deleting user metadata keys, only replacing them
  • does not support listing in-progress multipart uploads
  • does not support object-level access control
  • exposes multipart upload parts as regular keys
  • multi-delete does not delete SLO segments