diff --git a/docs/dev/internals.rst b/docs/dev/internals.rst index b12eb49e..c46e4b1f 100644 --- a/docs/dev/internals.rst +++ b/docs/dev/internals.rst @@ -10,7 +10,9 @@ keys were used to fetch them. A double index allows O(1) lookups. The second hash key takes O(K) to build, where K is the number of keys for the object (either 1 for a hash-only table, or 2 for a hash + range key). -Objects are sent in a dict that looks like:: +Objects are sent in a dict that looks like: + +.. code-block:: python { "table_name": { @@ -27,7 +29,9 @@ Objects are sent in a dict that looks like:: ... } -And returned in a similar dict:: +And returned in a similar dict: + +.. code-block:: python { "table_name": { @@ -55,11 +59,15 @@ and for each table (table name -> key names). Then the lookup is as follows: A table "objects" with a hash_key named "first" and range key named "last" -(assume that the dynamo and model names are the same):: +(assume that the dynamo and model names are the same): + +.. code-block:: python instance = Foo(first="foo", last="bar") -This will be loaded as:: +This will be loaded as: + +.. code-block:: python table_keys = {"objects": ("first", "last")} indexed_objects = {"objects": {("foo", "bar"): instance}} @@ -71,7 +79,9 @@ This will be loaded as:: } } -And the response will contain:: +And the response will contain: + +.. code-block:: python response = { "objects": { @@ -85,12 +95,16 @@ And the response will contain:: } } -Processing this object will first find the table_key for "objects":: +Processing this object will first find the table_key for "objects": + +.. code-block:: python ("first", "last") And then pull the corresponding values from the item in that order, to -construct the object index:: +construct the object index: + +.. code-block:: python indexed_objects["objects"][("foo", "bar")] diff --git a/docs/index.rst b/docs/index.rst index f2f90e32..c9edc68e 100644 --- a/docs/index.rst +++ b/docs/index.rst @@ -22,7 +22,12 @@ Installation Quickstart ========== -First, define and bind our model:: +First, define and bind our model: + +.. code-block:: python + + import uuid + from bloop import ( Engine, Column, UUID, String, @@ -42,7 +47,9 @@ First, define and bind our model:: engine = Engine() engine.bind(base=Base) -Save an instance, load by key, and get the first query result:: +Save an instance, load by key, and get the first query result: + +.. code-block:: python account = Account( id=uuid.uuid4(), @@ -59,7 +66,9 @@ Save an instance, load by key, and get the first query result:: .key(Account.email == "foo@bar.com") also_same = q.first() -Kick it up a notch with conditional operations:: +Kick it up a notch with conditional operations: + +.. code-block:: python # Exactly the same save as above, but now we # fail if the id isn't unique. diff --git a/docs/user/custom_types.rst b/docs/user/custom_types.rst index 7095159c..e485b376 100644 --- a/docs/user/custom_types.rst +++ b/docs/user/custom_types.rst @@ -16,7 +16,9 @@ In rare cases, you may want to implement ``bind`` to provide engine-specific pai Quick example ============= -Here's a trivial type that prepends a string with its length. So ``"hello world"`` is stored as ``"11:hello world"``:: +Here's a trivial type that prepends a string with its length. So ``"hello world"`` is stored as ``"11:hello world"``: + +.. code-block:: python class LengthString(bloop.String): def __init__(self, sep=":"): @@ -33,7 +35,9 @@ Here's a trivial type that prepends a string with its length. So ``"hello world return super().dynamo_dump( prefix + value, context=context, **kwargs) -We can now use our custom type anywhere a String would be allowed:: +We can now use our custom type anywhere a String would be allowed: + +.. code-block:: python class SomeModel(bloop.new_base()): id = Column(LengthString("|"), hash_key=True) @@ -51,7 +55,9 @@ This is stored in Dynamo like so: | 11\|hello world | 13:hello, world! | +-----------------+------------------+ -Because the ``dynamo_load`` function strips the length prefix off, it's not present when loading from DynamoDB:: +Because the ``dynamo_load`` function strips the length prefix off, it's not present when loading from DynamoDB: + +.. code-block:: python obj = SomeModel(id="hello world") engine.load(obj) @@ -84,7 +90,9 @@ arbitrarily typed values, is useless for an Object Mapper which enforces types p Base Type ========= -Before diving into the available types, here's the structure of the base ``Type``:: +Before diving into the available types, here's the structure of the base ``Type``: + +.. code-block:: python class Type(declare.TypeDefinition): python_type = None @@ -137,7 +145,9 @@ Unlike ``python_type``, this field is **required** and must be one of the types used to dump a value eg. ``"some string"`` into the proper DynamoDB wire format ``{"S": "some string"}``. Usually, you'll want to define this on your custom type. In some cases, however, you won't know this value until the type is instantiated. For example, the built-in :ref:`user-set-type` type constructs the backing type based on its inner -type's backing type with roughly the following:: +type's backing type with roughly the following: + +.. code-block:: python def __init__(self, typedef=None): if typedef is None: @@ -157,7 +167,9 @@ None. If you want to handle ``None``, you will need to implement ``_load`` your The bloop engine that is loading the value can always be accessed through ``context["engine"]``; this is useful to return different values depending on how the engine is configured, or performing chained operations. For example, you -could implement a reference type that loads a value from a different model like so:: +could implement a reference type that loads a value from a different model like so: + +.. code-block:: python class ReferenceType(bloop.Type): def __init__(self, model=None, blob_name=None): @@ -179,7 +191,9 @@ could implement a reference type that loads a value from a different model like context["engine"].load(obj) return obj -And its usage:: +And its usage: + +.. code-block:: python class Data(bloop.new_base()): id = Column(String, hash_key=True) @@ -198,7 +212,9 @@ And its usage:: The exact reverse of ``dynamo_load``, this method takes the modeled value and turns it a string that contains a DynamoDB-compatible format for the given backing value. For binary objects, this means base64 encoding the value. -For the ``ReferenceType`` defined above, here is the corresponding ``dynamo_dump``:: +For the ``ReferenceType`` defined above, here is the corresponding ``dynamo_dump``: + +.. code-block:: python def dynamo_dump(self, value, *, context, **kwargs): # value is an instance of the loaded object, @@ -240,7 +256,9 @@ need to implement ``_register`` if your custom type has a reference to another t For example, the built-in :ref:`user-set-type` uses a type passed as an argument during ``__init__`` to load and dump values from a String Set, Number Set, or Binary Set. To ensure the type engine can handle the nested load/dump calls -for that type, it implements ``_register`` like so:: +for that type, it implements ``_register`` like so: + +.. code-block:: python class Set(Type): """Adapter for sets of objects""" @@ -277,7 +295,9 @@ doesn't know about a particular type. You may store a custom config value on yo a full or partial load. You may want to associate different engines with particular views of data (say, one for users and one for admins) and return appropriate functions for both. -By implementing a custom ``bind`` you may remove the need to implement the ``_load`` and ``_dump`` functions:: +By implementing a custom ``bind`` you may remove the need to implement the ``_load`` and ``_dump`` functions: + +.. code-block:: python import declare @@ -307,7 +327,9 @@ By implementing a custom ``bind`` you may remove the need to implement the ``_lo # Users can modify this field but only admins can view it return value -Its usage is exactly the same as any other type:: +Its usage is exactly the same as any other type: + +.. code-block:: python class PlayerReport(bloop.new_base()): id = Column(Integer, hash_key=True) @@ -341,7 +363,9 @@ Here are two simple enum types that can be built off existing types with minimal the :ref:`user-integer-type` type and consumes little space, while the second is based on :ref:`user-string-type` and stores the Enum values. -For both examples, let's say we have the following :py:class:`enum.Enum`:: +For both examples, let's say we have the following :py:class:`enum.Enum`: + +.. code-block:: python import enum class Color(enum.Enum): @@ -355,7 +379,7 @@ Integer Enum In this type, dump will transform ``Color -> int`` using ``color.value`` and hand the int to ``super``. Meanwhile, load will transform ``int -> Color`` using ``Color(value)`` where value comes from ``super``. -:: +.. code-block:: python class EnumType(bloop.Integer): def __init__(self, enum_cls=None): @@ -372,7 +396,9 @@ load will transform ``int -> Color`` using ``Color(value)`` where value comes fr value = super().dynamo_load(value, context=context, **kwargs) return self.enum_cls(value) -Usage:: +Usage: + +.. code-block:: python class Shirt(new_base()): id = Column(String, hash_key=True) @@ -397,7 +423,7 @@ String Enum This will look remarkably similar, with the only difference that ``Enum.name`` gives us a string, and ``Enum[value]`` gives us an enum value by string. -:: +.. code-block:: python class EnumType(bloop.String): def __init__(self, enum_cls=None): @@ -414,7 +440,9 @@ gives us an enum value by string. value = super().dynamo_load(value, context=context, **kwargs) return self.enum_cls[value] -And usage is exactly the same:: +And usage is exactly the same: + +.. code-block:: python class Shirt(new_base()): id = Column(String, hash_key=True) @@ -437,7 +465,9 @@ Stored in DynamoDB as: RSA Example =========== -This is a quick type for storing a public RSA key in binary:: +This is a quick type for storing a public RSA key in binary: + +.. code-block:: python from Crypto.PublicKey import RSA @@ -454,7 +484,9 @@ This is a quick type for storing a public RSA key in binary:: value = value.exportKey(format="DER") return super().dynamo_dump(value, context=context, **kwargs) -Usage:: +Usage: + +.. code-block:: python class PublicKey(bloop.new_base()): id = Column(String, hash_key=True) diff --git a/docs/user/patterns.rst b/docs/user/patterns.rst index b3cd7109..8552d5a5 100644 --- a/docs/user/patterns.rst +++ b/docs/user/patterns.rst @@ -6,7 +6,7 @@ DynamoDB Local Connect to a local DynamoDB instance. -:: +.. code-block:: python import boto3 import bloop @@ -29,7 +29,7 @@ Generic "if not exist" Condition to ensure an object's hash (or hash + range) key are not set (item doesn't exist). -:: +.. code-block:: python def if_not_exist(obj): hash_key = obj.Meta.hash_key diff --git a/docs/user/types.rst b/docs/user/types.rst index d6891fe6..1c4cb04a 100644 --- a/docs/user/types.rst +++ b/docs/user/types.rst @@ -19,7 +19,9 @@ String Since everything has to become a string to build the request body, this is the simplest type. String is one of three base types that can be used as a hash or range key. -The constructor takes no args, and values are stored in DynamoDB 1:1 with their python type:: +The constructor takes no args, and values are stored in DynamoDB 1:1 with their python type: + +.. code-block:: python # equivalent Column(String) @@ -34,7 +36,9 @@ Binary Binary data corresponds to the ``bytes`` type in python, sent over the wire as a base64 encoded string, and stored in DynamoDB as bytes. Binary is one of three base types that can be used as a hash or range key. -The constructor takes no args:: +The constructor takes no args: + +.. code-block:: python # equivalent Column(Binary) @@ -54,7 +58,9 @@ but makes the translation easier for some uses). You should absolutely review the documentation_ before using python floats, as errors can be subtle. -The constructor takes no args:: +The constructor takes no args: + +.. code-block:: python # equivalent Column(Float) @@ -70,7 +76,9 @@ Boolean ------- Unlike String, Binary, and Float, the Boolean type cannot be used as a hash or range key. Like the other basic types, -it takes no args. It will coerce any value using ``bool``:: +it takes no args. It will coerce any value using ``bool``: + +.. code-block:: python bool_type = Boolean() bool_type.dynamo_dump(["foo", "bar"]) # true @@ -85,7 +93,9 @@ UUID ---- Backed by the ``String`` type, this stores a UUID as its string representation. It can handle any -:py:class:`uuid.UUID`, and its constructor takes no args:: +:py:class:`uuid.UUID`, and its constructor takes no args: + +.. code-block:: python import uuid @@ -101,7 +111,7 @@ DateTime is backed by the ``String`` type and maps to an :py:class:`arrow.arrow. can be instructed to use a particular timezone, values are always stored in UTC ISO8601_ to enable the full range of comparison operators. -:: +.. code-block:: python import arrow u = DateTime() @@ -126,7 +136,9 @@ Integer ------- Integer is a very thin wrapper around the ``Float`` type, and simply calls ``int()`` on the values passed to and from -its parent type:: +its parent type: + +.. code-block:: python int_type = Integer() int_type.dynamo_dump(3.5) # "3" @@ -154,7 +166,9 @@ whose ``backing_type`` is one of ``S``, ``N``, or ``B``). This is because the D ``SN``, or ``SB``. When loading or dumping a set, the inner type's load and dump functions will be used for each item in the set. If the -set type does not need any arguments, you may provide the class instead of an instance:: +set type does not need any arguments, you may provide the class instead of an instance: + +.. code-block:: python # type class uses no-arg __init__ float_set = Set(Float) @@ -168,10 +182,12 @@ set type does not need any arguments, you may provide the class instead of an in floats = set([3.5, 2, -1.0]) float_set.dynamo_dump(floats) # ["3.5", "2", "-1.0"] +.. _user-list-type: + List ---- -While DynamoDB's ``LIST`` type allows any combination of types, bloop's built-in ``List`` type requires you to +While DynamoDB's ``List`` type allows any combination of types, bloop's built-in ``List`` type requires you to constrain the list to a single type. This type is constructed the same way as ``Set`` above. This limitation exists because there isn't enough type information when loading a list to tell subclasses apart. @@ -187,7 +203,7 @@ functions). If you need to support multiple types in lists, the type system is general enough that you can define your own List type, that stores the type information of each object when your type is dumped to Dynamo. -:: +.. code-block:: python # type class uses no-arg __init__ float_list = List(Float) @@ -203,9 +219,90 @@ List type, that stores the type information of each object when your type is dum Map --- -General document that expects a type for each key. +Like the List type above, ``Map`` is a restricted subset of the general DynamoDB ``Map`` and only loads/dumps the +modeled structure you specify. For more information on why bloop does not support arbitrary types in Maps, see the +:ref:`user-list-type` type above. + +You construct a map type through ``**kwargs``, where each key is the document key, and each value is a type definition +or type instance (``DateTime`` or ``DateTime(timezone="...")``). There is no restriction on what types can be used for +keys, including nested maps and other document-based types. + +.. code-block:: python + + ProductData = Map(**{ + 'Rating': Float(), + 'Stock': Integer(), + 'Description': Map(**{ + 'Heading': String, + 'Body': String, + 'Specifications': String + }), + 'Id': UUID, + 'Updated': DateTime + }) + + + class Product(new_base()): + id = Column(Integer, hash_key=True) + old_data = Column(ProductData) + new_data = Column(ProductData) TypedMap -------- -Map with a single type for values. Any number of string keys. +Like ``Map`` above, ``TypedMap`` is not a general map for any typed data. Unlike Map however, TypedMap allows an +arbitrary number of keys, so long as all of the values have the same type. This is useful when you are storing data +under user-provided keys, or mapping for an unknown key size. + +As with List and Map, you can nest TypedMaps. For example, storing the event data for an unknown number of +instances might look something like: + +.. code-block:: python + + # Modeling some events + # ----------------------------------- + # The unpacking dict above can also just be + # direct kwargs + EventCounter = Map( + last=DateTime, + count=Integer, + source_ips=Set(String)) + + + class Metric(new_base()): + name = Column(String, hash_key=True) + host_events = Column(TypedMap(EventCounter)) + + + # Initial save, during service setup + # ----------------------------------- + metric = Metric(name="email-campaign-2016-06-29") + metric.host_events = {} + engine.save(metric) + + + # Recording an event during request handler + # ----------------------------------- + host_name = "api.control-plane.host-1" + metric = Metric(name="...") + engine.load(metric) + + # If there were no events, create an empty dict + events = metric.host_events.get(host_name) + if events is None: + events = { + "count": 0, + "source_ips": set() + } + metric.host_events[host_name] = events + + # Record this requester event + events["count"] += 1 + events["last"] = arrow.now() + events["source_ips"].add(request.get_ip()) + + # Atomic save helps us here because DynamoDB doesn't + # support multiple updates with overlapping paths yet: + # https://github.com/numberoverzero/bloop/issues/28 + # https://forums.aws.amazon.com/message.jspa?messageID=711992 + engine.save(metric, atomic=True)