Skip to content

Latest commit

 

History

History
271 lines (168 loc) · 9.92 KB

serializers.rst

File metadata and controls

271 lines (168 loc) · 9.92 KB
Raw

How-to — Serializers

Table of Contents

The Basics

Where Are They Used?

A :term:`serializer` is used by a :term:`protocol object`:

Which Ones Are Already Available?

Several serializers are provided in the :mod:`easynetwork.serializers` module. Do not hesitate to use them.

If nothing fits your needs, you can implement your own serializer.

Writing A One-Shot Serializer

:term:`One-shot serializers <one-shot serializer>` are the easiest piece of code to write. They can be used directly on :class:`.DatagramProtocol` instances, and you can use a :term:`serializer wrapper` to use the :class:`.StreamProtocol` class.

.. seealso::

   :mod:`easynetwork.serializers.wrapper` module
      The full list of available wrappers for serializers.

To write a :term:`one-shot serializer`, you must create a subclass of :class:`~.AbstractPacketSerializer` and override its :meth:`~.AbstractPacketSerializer.serialize` and :meth:`~.AbstractPacketSerializer.deserialize` methods.

A naive implementation of :class:`.JSONSerializer` should look something like this:

.. literalinclude:: ../_include/examples/howto/serializers/one_shot_serializer/example1.py
   :linenos:

Parsing Error

The :meth:`~.AbstractPacketSerializer.deserialize` function must raise a :exc:`.DeserializeError` to indicate that a parsing error was "expected" so that the received data is considered invalid.

.. literalinclude:: ../_include/examples/howto/serializers/one_shot_serializer/example2.py
   :linenos:
   :emphasize-lines: 6,19,22-23

Warning

Otherwise, any other error is considered a serializer crash.

The Use Of self

A :term:`serializer` is intended to be shared by multiple :term:`protocols <protocol object>` (and :term:`protocols <protocol object>` are intended to be shared by multiple endpoints).

Therefore, the object should only store additional configuration used for serialization/deserialization.

For example:

.. literalinclude:: ../_include/examples/howto/serializers/one_shot_serializer/example3.py
   :linenos:

Warning

Do not store per-serialization data. You might see some magic.

!DANGER!

Seriously, don't do that.

Using A One-Shot Serializer

You must pass the :term:`serializer` to the appropriate :term:`protocol object` that is expected by the endpoint class:

.. tabs::

   .. group-tab:: DatagramProtocol

      .. literalinclude:: ../_include/examples/howto/serializers/one_shot_serializer/example4_datagram.py
         :pyobject: main
         :linenos:

   .. group-tab:: StreamProtocol

      .. literalinclude:: ../_include/examples/howto/serializers/one_shot_serializer/example4_stream.py
         :pyobject: main
         :linenos:

Note

Using a :term:`serializer wrapper` means that the transferred data can be completely different from the original output.

If this is important to you, don't choose one of them lightly.

Writing An Incremental Serializer

:term:`Incremental serializers <incremental serializer>` are a bit trickier to implement. They can be used directly on both :class:`.StreamProtocol` and :class:`.DatagramProtocol` instances.

To write an :term:`incremental serializer`, you must create a subclass of :class:`~.AbstractIncrementalPacketSerializer` and override its :meth:`~.AbstractIncrementalPacketSerializer.incremental_serialize` and :meth:`~.AbstractIncrementalPacketSerializer.incremental_deserialize` methods. The :meth:`~.AbstractIncrementalPacketSerializer.serialize` and :meth:`~.AbstractIncrementalPacketSerializer.deserialize` methods have a default implementation that uses the incremental serialization methods.

Option 1: Using Common Patterns

Chances are that the communication protocol uses a simple principle to determine the end of a packet. The most common cases are:

  • All your packet frames use a precise byte sequence (most likely a newline).
  • Each packet has a fixed size.

In these cases you can use the base classes in :mod:`easynetwork.serializers.base_stream`.

Let's say that for the incremental part, we consider each line received to be a JSON object, separated by \r\n:

.. literalinclude:: ../_include/examples/howto/serializers/incremental_serializer/example1.py
   :linenos:
   :emphasize-lines: 7,13,15

:class:`.AutoSeparatedPacketSerializer` adds the following behaviors:

  • incremental_serialize() will append \r\n to the end of the serialize() output.
  • incremental_deserialize() waits until \r\n is found in the input, removes the separator, and calls deserialize() on it.

Tip

Take a look at other available base classes in :mod:`easynetwork.serializers` before rewriting something that already exists.

Option 2: From Scratch

Let's see how we can get by without using the :class:`.AutoSeparatedPacketSerializer`:

.. literalinclude:: ../_include/examples/howto/serializers/incremental_serializer/example2.py
   :linenos:

This adds a lot of code! Let's take a closer look at the implementation.

Code Mutualization

To avoid duplication of code between the one-shot part and the incremental part, the serialization/deserialization part of the code goes to a private method.

.. literalinclude:: ../_include/examples/howto/serializers/incremental_serializer/example2.py
   :start-at: def _dump
   :end-at: return json.loads
   :dedent:
   :lineno-match:

And now serialize() and deserialize() use them instead:

.. literalinclude:: ../_include/examples/howto/serializers/incremental_serializer/example2.py
   :start-at: def serialize
   :end-at: raise DeserializeError
   :dedent:
   :lineno-match:
   :emphasize-lines: 2,6

The Purpose Of incremental_serialize()

:meth:`~.AbstractIncrementalPacketSerializer.incremental_serialize` must be a :term:`generator` function (or at least return a :term:`generator iterator`) that yields all the parts of the serialized packet. It must also add any useful metadata to help :meth:`~.AbstractIncrementalPacketSerializer.incremental_deserialize` find the end of the packet.

.. literalinclude:: ../_include/examples/howto/serializers/incremental_serializer/example2.py
   :pyobject: MyJSONSerializer.incremental_serialize
   :dedent:
   :lineno-match:

Most of the time, you will have a single :keyword:`yield`. The goal is: each :keyword:`yield` must send as many :class:`bytes` as possible without copying or concatenating.

Tip

There may be exceptions, like this example. (Your RAM will not cry because you added 2 bytes to a byte sequence of almost 100 bytes. The question may be asked if the byte sequence is ending up to 4 GB.)

It is up to you to find the balance between RAM explosion and performance degradation.

Note

The endpoint implementation can decide to concatenate all the pieces and do one big send. However, it may be more attractive to do something else with the returned bytes. :meth:`~.AbstractIncrementalPacketSerializer.incremental_serialize` is here to give endpoints this freedom.

The Purpose Of incremental_deserialize()

:meth:`~.AbstractIncrementalPacketSerializer.incremental_deserialize` must be a :term:`generator` function (or at least return a :term:`generator iterator`) that yields :data:`None` until all the data parts of the packet have been retrieved and parsed.

This generator must return a pair of (packet, remainder) where packet is the deserialized packet and remainder is any superfluous trailing bytes that was useless.

At each :keyword:`yield` checkpoint, the endpoint implementation sends to the generator the data received from the remote endpoint.

.. literalinclude:: ../_include/examples/howto/serializers/incremental_serializer/example2.py
   :pyobject: MyJSONSerializer.incremental_deserialize
   :dedent:
   :lineno-match:
   :emphasize-lines: 2,5

Note

Even if we could create 5 more JSON packets from remainder, incremental_deserialize() must always deserialize the first one available and return the rest as is.

This allows the endpoint implementation to deserialize only the needed packet. The rest is reused when the application wants an other packet.

.. seealso::

   :doc:`/api/serializers/tools`
      Regroups helpers for (incremental) serializer implementations.

   :pep:`255` — Simple Generators
      The proposal for adding generators and the :keyword:`yield` statement to Python.

   :pep:`342` — Coroutines via Enhanced Generators
      The proposal to enhance the API and syntax of generators, making them usable as simple coroutines.

   :pep:`380` — Syntax for Delegating to a Subgenerator
      The proposal to introduce the ``yield_from`` syntax, making delegation to subgenerators easy.

Parsing Error

The :meth:`~.AbstractIncrementalPacketSerializer.incremental_deserialize` function must raise an :exc:`.IncrementalDeserializeError` to indicate that a parsing error was "expected" so that the received data is considered invalid.

.. literalinclude:: ../_include/examples/howto/serializers/incremental_serializer/example2.py
   :pyobject: MyJSONSerializer.incremental_deserialize
   :dedent:
   :lineno-match:
   :emphasize-lines: 12-13

Warning

Otherwise, any other error is considered a serializer crash.

If :exc:`.DeserializeError` is raised instead, this is converted to a :exc:`RuntimeError`.

Note

:exc:`.IncrementalDeserializeError` needs the possible valid remainder, that is not the root cause of the error. In the example, even if data is an invalid JSON object, all bytes after the \r\n token (in remainder) are not lost.