[Schema Registry + Avro Serializer] 1.0.0b1

sdk/schemaregistry/azure-schemaregistry-avroserializer/CHANGELOG.md

@@ -0,0 +1,5 @@
+# Release History
+
+## 1.0.0b1 (Unreleased)
+
+Version 1.0.0b1 is the first preview of our efforts to create a user-friendly and Pythonic client library for Azure Schema Registry Avro Serializer. 

sdk/schemaregistry/azure-schemaregistry-avroserializer/MANIFEST.in

@@ -0,0 +1,4 @@
+include *.md
+include azure/__init__.py
+recursive-include tests *.py
+recursive-include samples *.py

sdk/schemaregistry/azure-schemaregistry-avroserializer/README.md

@@ -0,0 +1,122 @@
+# Azure Schema Registry Avro Serializer client library for Python
+
+Azure Schema Registry Avro Serializer provides the ability to serialize and deserialize data according
+to the given avro schema. It is integrated with Azure Schema Registry SDK and will automatically register and get schema.
+
+## Getting started
+
+### Install the package
+
+Install the Azure Schema Registry Avro Serializer client library for Python with [pip][pip]:
+
+```Bash
+pip install azure-schemaregistry-avroserializer
+```
+
+### Prerequisites: 
+To use this package, you must have:
+* Azure subscription - [Create a free account][azure_sub]
+* Azure Schema Registry
+* Python 2.7, 3.5 or later - [Install Python][python]
+
+### Authenticate the client
+Interaction with Schema Registry Avro Serializer starts with an instance of SchemaRegistryAvroSerializer class. You need the endpoint, AAD credential and schema group name to instantiate the client object. 
+
+**Create client using the azure-identity library:**
+
+```python
+from azure.schemaregistry.serializer.avro_serializer import SchemaRegistryAvroSerializer
+from azure.identity import DefaultAzureCredential
+
+credential = DefaultAzureCredential()
+endpoint = '<< ENDPOINT OF THE SCHEMA REGISTRY >>'
+schema_group = '<< GROUP NAME OF THE SCHEMA >>'
+schema_registry_client = SchemaRegistryAvroSerializer(endpoint, credential, schema_group)
+```
+
+## Key concepts
+
+- Avro: Apache Avro™ is a data serialization system.
+
+## Examples
+
+The following sections provide several code snippets covering some of the most common Schema Registry tasks, including:
+
+- [Serialization](serialization)
+- [Deserialization](deserialization)
+
+### Serialization
+
+```python
+import os
+from azure.schemaregistry.serializer.avro_serializer import SchemaRegistryAvroSerializer
+from azure.identity import DefaultAzureCredential
+
+token_credential = DefaultAzureCredential()
+endpoint = os.environ['SCHEMA_REGISTRY_ENDPOINT']
+schema_group = "<your-group-name>"
+
+serializer = SchemaRegistryAvroSerializer(endpoint, token_credential, schema_group)
+
+schema_string = """
+{"namespace": "example.avro",
+ "type": "record",
+ "name": "User",
+ "fields": [
+     {"name": "name", "type": "string"},
+     {"name": "favorite_number",  "type": ["int", "null"]},
+     {"name": "favorite_color", "type": ["string", "null"]}
+ ]
+}"""
+
+with serializer:
+    dict_data = {"name": "Ben", "favorite_number": 7, "favorite_color": "red"}
+    encoded_bytes = serializer.serialize(dict_data, schema_string)
+```
+
+### Deserialization
+
+```python
+import os
+from azure.schemaregistry.serializer.avro_serializer import SchemaRegistryAvroSerializer
+from azure.identity import DefaultAzureCredential
+
+token_credential = DefaultAzureCredential()
+endpoint = os.environ['SCHEMA_REGISTRY_ENDPOINT']
+schema_group = "<your-group-name>"
+
+serializer = SchemaRegistryAvroSerializer(endpoint, token_credential, schema_group)
+
+with serializer:
+    encoded_bytes = b'<data_encoded_by_azure_schema_registry_avro_serializer>'
+    decoded_data = serializer.deserialize(encoded_bytes)
+```
+
+## Troubleshooting
+
+Azure Schema Registry Avro Serializer raise exceptions defined in [Azure Core][azure_core].
+
+## Next steps
+
+### More sample code
+
+Please find further examples in the [samples](./samples) directory demonstrating common Azure Schema Registry Avro Serializer scenarios.
+
+## Contributing
+
+This project welcomes contributions and suggestions.  Most contributions require you to agree to a
+Contributor License Agreement (CLA) declaring that you have the right to, and actually do, grant us
+the rights to use your contribution. For details, visit https://cla.microsoft.com.
+
+When you submit a pull request, a CLA-bot will automatically determine whether you need to provide
+a CLA and decorate the PR appropriately (e.g., label, comment). Simply follow the instructions
+provided by the bot. You will only need to do this once across all repos using our CLA.
+
+This project has adopted the [Microsoft Open Source Code of Conduct](https://opensource.microsoft.com/codeofconduct/).
+For more information see the [Code of Conduct FAQ](https://opensource.microsoft.com/codeofconduct/faq/) or
+contact [opencode@microsoft.com](mailto:opencode@microsoft.com) with any additional questions or comments.
+
+[pip]: https://pypi.org/project/pip/
+[python]: https://www.python.org/downloads/
+[azure_core]: https://github.com/Azure/azure-sdk-for-python/blob/master/sdk/core/azure-core/README.md
+[azure_sub]: https://azure.microsoft.com/free/

sdk/schemaregistry/azure-schemaregistry-avroserializer/azure/__init__.py

@@ -0,0 +1,26 @@
+# --------------------------------------------------------------------------
+#
+# Copyright (c) Microsoft Corporation. All rights reserved.
+#
+# The MIT License (MIT)
+#
+# Permission is hereby granted, free of charge, to any person obtaining a copy
+# of this software and associated documentation files (the ""Software""), to
+# deal in the Software without restriction, including without limitation the
+# rights to use, copy, modify, merge, publish, distribute, sublicense, and/or
+# sell copies of the Software, and to permit persons to whom the Software is
+# furnished to do so, subject to the following conditions:
+#
+# The above copyright notice and this permission notice shall be included in
+# all copies or substantial portions of the Software.
+#
+# THE SOFTWARE IS PROVIDED *AS IS*, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
+# FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS
+# IN THE SOFTWARE.
+#
+# --------------------------------------------------------------------------
+__path__ = __import__("pkgutil").extend_path(__path__, __name__)  # type: ignore

sdk/schemaregistry/azure-schemaregistry-avroserializer/azure/schemaregistry/__init__.py

@@ -0,0 +1,25 @@
+# --------------------------------------------------------------------------
+#
+# Copyright (c) Microsoft Corporation. All rights reserved.
+#
+# The MIT License (MIT)
+#
+# Permission is hereby granted, free of charge, to any person obtaining a copy
+# of this software and associated documentation files (the ""Software""), to
+# deal in the Software without restriction, including without limitation the
+# rights to use, copy, modify, merge, publish, distribute, sublicense, and/or
+# sell copies of the Software, and to permit persons to whom the Software is
+# furnished to do so, subject to the following conditions:
+#
+# The above copyright notice and this permission notice shall be included in
+# all copies or substantial portions of the Software.
+#
+# THE SOFTWARE IS PROVIDED *AS IS*, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
+# FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS
+# IN THE SOFTWARE.
+#
+# --------------------------------------------------------------------------

sdk/schemaregistry/azure-schemaregistry-avroserializer/azure/schemaregistry/serializer/__init__.py

@@ -0,0 +1,25 @@
+# --------------------------------------------------------------------------
+#
+# Copyright (c) Microsoft Corporation. All rights reserved.
+#
+# The MIT License (MIT)
+#
+# Permission is hereby granted, free of charge, to any person obtaining a copy
+# of this software and associated documentation files (the ""Software""), to
+# deal in the Software without restriction, including without limitation the
+# rights to use, copy, modify, merge, publish, distribute, sublicense, and/or
+# sell copies of the Software, and to permit persons to whom the Software is
+# furnished to do so, subject to the following conditions:
+#
+# The above copyright notice and this permission notice shall be included in
+# all copies or substantial portions of the Software.
+#
+# THE SOFTWARE IS PROVIDED *AS IS*, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
+# FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS
+# IN THE SOFTWARE.
+#
+# --------------------------------------------------------------------------

sdk/schemaregistry/azure-schemaregistry-avroserializer/azure/schemaregistry/serializer/avro_serializer/__init__.py

@@ -0,0 +1,34 @@
+# --------------------------------------------------------------------------
+#
+# Copyright (c) Microsoft Corporation. All rights reserved.
+#
+# The MIT License (MIT)
+#
+# Permission is hereby granted, free of charge, to any person obtaining a copy
+# of this software and associated documentation files (the ""Software""), to
+# deal in the Software without restriction, including without limitation the
+# rights to use, copy, modify, merge, publish, distribute, sublicense, and/or
+# sell copies of the Software, and to permit persons to whom the Software is
+# furnished to do so, subject to the following conditions:
+#
+# The above copyright notice and this permission notice shall be included in
+# all copies or substantial portions of the Software.
+#
+# THE SOFTWARE IS PROVIDED *AS IS*, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
+# FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS
+# IN THE SOFTWARE.
+#
+# --------------------------------------------------------------------------
+from ._version import VERSION
+
+__version__ = VERSION
+
+from ._schema_registry_avro_serializer import SchemaRegistryAvroSerializer
+
+__all__ = [
+    "SchemaRegistryAvroSerializer"
+]

sdk/schemaregistry/azure-schemaregistry-avroserializer/azure/schemaregistry/serializer/avro_serializer/_avro_serializer.py

@@ -0,0 +1,114 @@
+# --------------------------------------------------------------------------
+#
+# Copyright (c) Microsoft Corporation. All rights reserved.
+#
+# The MIT License (MIT)
+#
+# Permission is hereby granted, free of charge, to any person obtaining a copy
+# of this software and associated documentation files (the ""Software""), to
+# deal in the Software without restriction, including without limitation the
+# rights to use, copy, modify, merge, publish, distribute, sublicense, and/or
+# sell copies of the Software, and to permit persons to whom the Software is
+# furnished to do so, subject to the following conditions:
+#
+# The above copyright notice and this permission notice shall be included in
+# all copies or substantial portions of the Software.
+#
+# THE SOFTWARE IS PROVIDED *AS IS*, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
+# FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS
+# IN THE SOFTWARE.
+#
+# --------------------------------------------------------------------------
+import abc
+from typing import BinaryIO, Union, Type, TypeVar, Optional, Any, Dict
+from io import BytesIO
+import avro
+from avro.io import DatumWriter, DatumReader, BinaryDecoder, BinaryEncoder
+
+try:
+    ABC = abc.ABC
+except AttributeError:  # Python 2.7, abc exists, but not ABC
+    ABC = abc.ABCMeta("ABC", (object,), {"__slots__": ()})  # type: ignore
+
+ObjectType = TypeVar("ObjectType")
+
+
+class AvroObjectSerializer(ABC):
+
+    def __init__(self, codec=None):
+        """A Avro serializer using avro lib from Apache.
+        :param str codec: The writer codec. If None, let the avro library decides.
+        """
+        self._writer_codec = codec
+        self._schema_writer_cache = {}  # type: Dict[str, DatumWriter]
+        self._schema_reader_cache = {}  # type: Dict[str, DatumReader]
+
+    def serialize(
+        self,
+        data,  # type: ObjectType
+        schema,  # type: Optional[Any]
+    ):
+        # type: (...) -> bytes
+        """Convert the provided value to it's binary representation and write it to the stream.
+        Schema must be a Avro RecordSchema:
+        https://avro.apache.org/docs/1.10.0/gettingstartedpython.html#Defining+a+schema
+        :param data: An object to serialize
+        :param schema: A Avro RecordSchema
+        """
+        if not schema:
+            raise ValueError("Schema is required in Avro serializer.")
+
+        if not isinstance(schema, avro.schema.Schema):
+            schema = avro.schema.parse(schema)
+
+        try:
+            writer = self._schema_writer_cache[str(schema)]
+        except KeyError:
+            writer = DatumWriter(schema)
+            self._schema_writer_cache[str(schema)] = writer
+
+        stream = BytesIO()
+
+        writer.write(data, BinaryEncoder(stream))
+        encoded_data = stream.getvalue()
+
+        stream.close()
+        return encoded_data
+
+    def deserialize(
+        self,
+        data,  # type: Union[bytes, BinaryIO]
+        schema,  # type:
+        return_type=None,  # type: Optional[Type[ObjectType]]  # pylint: disable=unused-argument
+    ):
+        # type: (...) -> ObjectType
+        """Read the binary representation into a specific type.
+        Return type will be ignored, since the schema is deduced from the provided bytes.
+        :param data: A stream of bytes or bytes directly
+        :type data: BinaryIO or bytes
+        :param schema: A Avro RecordSchema
+        :param return_type: Return type is not supported in the Avro serializer.
+        :returns: An instantiated object
+        :rtype: ObjectType
+        """
+        if not hasattr(data, 'read'):
+            data = BytesIO(data)
+
+        if not isinstance(schema, avro.schema.Schema):
+            schema = avro.schema.parse(schema)
+
+        try:
+            reader = self._schema_reader_cache[str(schema)]
+        except KeyError:
+            reader = DatumReader(writers_schema=schema)
+            self._schema_reader_cache[str(schema)] = reader
+
+        bin_decoder = BinaryDecoder(data)
+        decoded_data = reader.read(bin_decoder)
+        data.close()
+
+        return decoded_data