Fixing Formatters read/write functionality.

All Formatters now have both read/write and to/from/Bytes methods.
Those specific formatter implementations that also implement the
_from/_to/Bytes method will default to directly downloading the
bytes to memory. The rest will be downloaded/uploaded to/from
a temporary file.

Checks if file exists or does not, in S3Datastore, were changed
since now we definitely incurr a GET charge for a Key's header
everytime so there is no need to duplicate the checks with
s3CheckFileExists calls.

python/lsst/daf/butler/core/butlerConfig.py

@@ -43,11 +43,12 @@
 
 class ButlerConfig(Config):
     """Contains the configuration for a `Butler`
+    
     The configuration is read and merged with default configurations for
     the particular classes. The defaults are read according to the rules
     outlined in `ConfigSubset`. Each component of the configuration associated
     with a configuration class reads its own defaults.
-    
+
     Parameters
     ----------
     other : `str`, `Config`, optional

python/lsst/daf/butler/core/formatter.py

@@ -87,6 +87,51 @@ def write(self, inMemoryDataset, fileDescriptor):
         """
         raise NotImplementedError("Type does not support writing")
 
+    def fromBytes(self, serializedDataset, fileDescriptor, component=None):
+        """Reads serialized data into a Dataset or its component.
+
+        Parameters
+        ----------
+        dataset : `bytes`
+            Bytes object to unserialize.
+        fileDescriptor : `FileDescriptor`
+            Identifies type to read it into and parameters to be used for reading.
+        component : `str`, optional
+            Component to read from the Dataset. Only used if the `StorageClass`
+            for reading differed from the `StorageClass` used to write the
+            file.
+
+        Returns
+        -------
+        inMemoryDataset : `object`
+            The requested data as a Python object. The type of object
+            is controlled by the specific formatter.
+
+        Raises
+        ------
+        ValueError
+            Component requested but this Dataset does not seem to be a concrete
+            composite.
+        """
+        raise NotImplementedError("Type does not support reading from bytes.")
+
+    def toBytes(self, inMemoryDataset, fileDescriptor):
+        """Serialize the Dataset to bytes based on formatter.
+
+        Parameters
+        ----------
+        inMemoryDataset : `object`
+            The Python object to serialize.
+        fileDescriptor : `FileDescriptor`
+            Identifies type to read it into and parameters to be used for reading.
+
+        Returns
+        -------
+        serializedDataset : `str`
+            bytes representing the serialized dataset.
+        """
+        raise NotImplementedError("Type does not support writing to bytes.")
+
     @abstractmethod
     def predictPath(self, location):
         """Return the path that would be returned by write, without actually

python/lsst/daf/butler/core/location.py

@@ -204,6 +204,8 @@ def __init__(self, bucket, datastoreRoot):
 
         Parameters
         ----------
+        bucket : `str`
+            Name of the Bucket that is used.
         datastoreRoot : `str`
             Root location of the `S3Datastore` in the Bucket.
         """