Skip to content

Commit

Permalink
docs finish Query/Scan. Use ..attribute #13
Browse files Browse the repository at this point in the history 
Didn't know ..attribute existed, and now it's beautiful.
My hacked-together attempt was failing as the notes expanded past
speciying a default value.

:party-parrot:
  • Loading branch information
@numberoverzero
numberoverzero committed Aug 8, 2016
1 parent a1e63aa commit c107ac3
Show file tree
Hide file tree
Showing 4 changed files with 164 additions and 103 deletions.
8 changes: 4 additions & 4 deletions bloop/filter.py
View file
Original file line number Diff line number Diff line change
Expand Up @@ -146,7 +146,7 @@ def validate_key_for(model, index, key):
class Filter:
def __init__(
self, *, engine, mode, model, index, strict, select, select_attributes=None, prefetch=0,
consistent=False, forward=True, limit=0, key=None, filter=None):
consistent=False, forward=True, limit=None, key=None, filter=None):
self.engine = engine
self.mode = mode
self.model = model
Expand All @@ -158,7 +158,7 @@ def __init__(
self._prefetch = prefetch
self._consistent = consistent
self._forward = forward
self._limit = limit
self._limit = limit or 0

self._key = key
self._filter = filter
Expand Down Expand Up @@ -390,12 +390,12 @@ def __iter__(self):
return self

def __next__(self):
# Still have buffered elements available
# still have buffered elements available
if len(self._buffer) > 0:
return self._buffer.popleft()

# Refill the buffer until we hit the limit, or Dynamo stops giving us continue tokens.
while (not self.exhausted) and (len(self._buffer) < self._prefetch):
while (not self.exhausted) and len(self._buffer) < self._prefetch:
response = self._call(self._prepared_request)
continuation_token = response.get("LastEvaluatedKey", None)
self._prepared_request["ExclusiveStartKey"] = continuation_token
Expand Down
103 changes: 55 additions & 48 deletions docs/user/01_define.rst
View file
Original file line number Diff line number Diff line change
Expand Up @@ -91,18 +91,21 @@ You can customize how the table is created with an inner ``Meta`` class:
Available properties:

**table_name**
| *(default is class name)*
| The table name in DynamoDB.
**read_units**
| *(default is 1)*
| The provisioned read units for the table.
**write_units**
| *(default is 1)*
| The provisioned write units for the table.
**abstract**
| *(default is False)*
| True if this model is not backed by a DynamoDB table.
.. attribute:: table_name

The table name for this model in DynamoDB. Defaults to the class name.

.. attribute:: read_units

The provisioned read units for the table. Defaults to 1.

.. attribute:: write_units

The provisioned write units for the table. Defaults to 1.

.. attribute:: abstract

True if this model is not backed by a DynamoDB table. Defaults to False.

Instances of abstract models can't be used with an Engine since there is no table to modify or query. Their
columns and indexes are not inherited.
Expand All @@ -121,19 +124,21 @@ Columns
name: Optional[str]=None,
**kwargs)
**typedef**
| *(required)*
| A type or instance of a type to use when loading and saving this column.
**hash_key**
| *(default is False)*
| True if this column is the model's hash key.
**range_key**
| *(default is False)*
| True if this column is the model's range key.
**name**
| *(default is None)*
| If None, the column's model name is used.
| The name this column is stored as in DynamoDB.
.. attribute:: typedef

A type or instance of a type to use when loading and saving this column.

.. attribute:: hash_key

True if this column is the model's hash key. Defaults to False.

.. attribute:: range_key

True if this column is the model's range key. Defaults to False.

.. attribute:: name

The name this column is stored as in DynamoDB. Defaults to the column's name in the model.

Each ``Column`` must have a type. Many types can be passed directly without instantiating. Sometimes, an
instance of a type can provide customization. These are equivalent:
Expand Down Expand Up @@ -176,29 +181,31 @@ Indexes
range_key: str,
name: Optional[str]=None)
**projection**
| *(required)*
| Columns in the index projection. ``"all"``, ``"keys"``, or a list of column names.
**hash_key**
| *(required for GSIs)*
| The model name of the column that will be this index's hash key.
| ``LocalSecondaryIndex`` always shares the model hash key.
**range_key**
| *(required for LSIs)*
| The model name of the column that will be this index's range key.
| ``GlobalSecondaryIndex`` does not require a range key.
**name**
| *(defaults is None)*
| If None, the index's model name is used.
| The name this index is stored as in DynamoDB.
**read_units**
| *(default is 1 for GSIs)*
| The provisioned read capacity for reads against this index.
| ``LocalSecondaryIndex`` shares the model's read units.
**write_units**
| *(default is 1 for GSIs)*
| The provisioned write capacity for writes through this index.
| ``LocalSecondaryIndex`` shares the model's write units.
.. attribute:: projection

The columns to project into this Index. Must be one of ``"all"``, ``"keys"``, or a list of column names.
The index and model hash and range keys are always included in the projection.

.. attribute:: hash_key

Required for GSIs. The model name of the column that will be this index's hash key.
You cannot specify the hash key for an LSI since it always shares the model's hash key.

.. attribute:: range_key

Required for LSIs. Optional for GSIs. The model name of the column that will be this index's range key.

.. attribute:: name

The name this index is stored as in DynamoDB. Defaults to the index's name in the model.

.. attribute:: read_units

The provisioned read units for the index. LSIs share the model's read units. Defaults to 1.

.. attribute:: write_units

The provisioned write units for the index. LSIs share the model's write units. Defaults to 1.

Specific column projections always include key columns. A query against the following ``User`` index would
return objects that include all columns except ``created_on`` (since ``id`` and ``email`` are the model
Expand Down
23 changes: 12 additions & 11 deletions docs/user/02_mutate.rst
View file
Original file line number Diff line number Diff line change
Expand Up @@ -66,17 +66,18 @@ Save and Delete share the same interface; they both conditionally modify the sta
condition: Optional[bloop.Condition]=None,
atomic: Optional[bool]=None) -> None
**objs**
| *(required)*
| Any number of objects to save (may be from different models).
**condition**
| *(default is None)*
| Each object will only be saved if the condition holds for that object
**atomic**
| *(default is None)*
| If None, ``engine.config["atomic"]`` is used.
| The default engine config does not enable atomic operations.
| If True, DynamoDB and the local state must match to perform the save.
.. attribute:: objs

Any number of objects to modify (may be from different models).

.. attribute:: condition

Each object will only be modified if the condition holds for that object. Defaults to None.

.. attribute:: atomic

Whether or not to use an atomic condition for this operation. When True, DynamoDB and the local state must match
to perform the operation (in addition to any other condition). Defaults to ``engine.config["atomic"]``

==================
Conditions, Atomic
Expand Down
133 changes: 93 additions & 40 deletions docs/user/03_query_scan.rst
View file
Original file line number Diff line number Diff line change
Expand Up @@ -104,56 +104,109 @@ Scan and Query have the same interface:
obj: Union[bloop.BaseModel, bloop.Index],
consistent: Optional[bool]=None) -> bloop.Filter
**obj**
| *(required)*
| This is either an instance of a model, or an index on a model.
**consistent**
| *(default is None)*
| If None, ``engine.config["consistent"]`` is used.
| The default engine config does not enable consistent operations.
| If True, `Strongly Consistent`_ Reads will be used.
.. attribute:: obj

This is either an instance of a model, or an index on a model. From the example above, this can
be the ``File`` model, or any of its indexes ``Filter.on_created``, ``Filter.by_owner``, or ``Filter.by_size``.

.. attribute:: consistent

See the :ref:`consistent property <property-consistent>` below.

----------
Properties
----------

Of the following, only ``key`` is required, and only for Queries.

**key**
| *(required for Queries)*
| *(ignored by Scans)*
**select**
|
**filter**
|
**consistent**
| *(not available for GSIs)*
**forward**
| *(ignored by Queries)*
**limit**
|
**prefetch**
|
.. attribute:: key

Queries require a key :ref:`condition <conditions>`. Scans do not use key conditions.

A key condition must always include an equality condition (``==``) against the hash key of the object (Model
or Index) being queried. You may optionally include one condition against the range key of the object.

The available conditions for a range key are[0]::

<, <=, ==, >=, >, begins_with, between

To use a hash key and range key condition together, join them with ``&``:

.. code-block:: python
in_home = File.path == "~"
start_with_a = File.name.begins_with("a")
query.key = in_home & starts_with_a
.. attribute:: select

The columns to load. One of ``"all"``, ``"projected"``, ``"count"``, or a list of columns.
When select is "count", no objects will be returned, but the ``count`` and ``scanned`` properties
will be set on the result iterator (see below). If the Query or Scan is against a Model, you cannot
use "projected". Defaults to "all" for Models and "projected" for Indexes.

.. attribute:: filter

A server-side filter :ref:`condition <conditions>` that DynamoDB applies to objects before returning them.
Only objects that match the filter will be returned. Defaults to None.

.. _property-consistent:

.. attribute:: consistent

Whether or not `strongly consistent reads`_ should be used. Keep in mind that Strongly Consistent Reads
consume twice as many read units as Eventually Consistent Reads. This setting has no effect when used
with a GSI, since strongly consistent reads `can't be used with a Global Secondary Index`__.
Defaults to ``engine.config["consistent"]``

__ http://docs.aws.amazon.com/amazondynamodb/latest/APIReference/API_Query.html#DDB-Query-request-ConsistentRead

.. attribute:: forward

Whether to scan in ascending order (see `ScanIndexForward`_). When True, scans are ascending.
When False, scans are descending. This setting is not used for Queries. Defaults to True.

.. attribute:: limit

The maximum number of objects that will be returned. This is **NOT** the same as DynamoDB's `Limit`__, which
is the maximum number of objects evaluated per continuation token. Once the iterator has returned ``limit``
object, it will not return any more (even if the internal buffer is not empty). Defaults to None.

__ http://docs.aws.amazon.com/amazondynamodb/latest/APIReference/API_Query.html#DDB-Query-request-Limit

.. attribute:: prefetch

The number of objects to buffer from DynamoDB before the iterator will yield objects. Setting this to a high value
can be useful if you want to use your read capacity in bursts. Otherwise, the iterator will only follow
continuation tokens when the buffer is empty and another object is requested. Defaults to 0.

After you have finished defining the Query or Scan, you can use ``first()``, ``one()``, or ``build()`` to
retrieve results.
retrieve results. If there are no matching objects, ``first`` will raise a ``ConstraintViolation``. If
there is not exactly one matching object, ``one`` will raise a ``ConstraintViolation``.

You can use ``build`` to return an iterable, which fetches objects depending on ``prefetch`` and ``limit``.
The object returned by ``build`` does not cache objects. You can start the iterable over at any time by calling
``reset()``. The iterator has the following properties for inspecting the state of the scan or query:

.. attribute:: count

The number of objects loaded from DynamoDB so far. This includes objects still in the iterator's buffer, which
may not have been yielded yet.

.. attribute:: scanned

If there are no matching results, ``first`` will raise a ``ConstraintViolation``.
The number of objects that DynamoDB has scanned so far. If you are not using a filter, this is equal
to ``count``. Otherwise, the difference ``scanned - count`` is the number of objects that so far have
not met the filter condition. See `Counting Items`_.

If there is not exactly one matching result, ``one`` will raise a ``ConstraintViolation``.
.. attribute:: exhausted

Finally, ``build`` returns an iterable, fetching results within the constraints of ``prefetch`` and ``limit``.
The object returned by ``build`` does not cache results. You can start the iterable over by calling ``reset()``.
The iterator has the following properties for inspecting the state of the scan or query:
If there is no limit, this will be True when the buffer is empty and DynamoDB stops returning ContinuationTokens
to follow.

**count**
| Number of items loaded so far.
| Some items may still be in the iterable's buffer.
**scanned**
| Number of items DynamoDB has seen so far.
| This number may be greater than ``count``. See `Counting Items`_.
**exhausted**
| True if there are no more results to fetch.
If there is a limit, this will be True when the iterator has yielded ``limit`` objects, or the above;
whichever happens first. With a limit, there may be objects in the internal buffer when the
iterator is exhausted.

.. _Strongly Consistent: http://docs.aws.amazon.com/amazondynamodb/latest/developerguide/HowItWorks.ReadConsistency.html
.. _strongly consistent reads: http://docs.aws.amazon.com/amazondynamodb/latest/developerguide/HowItWorks.ReadConsistency.html
.. _ScanIndexForward: http://docs.aws.amazon.com/amazondynamodb/latest/APIReference/API_Query.html#DDB-Query-request-ScanIndexForward
.. _Counting Items: http://docs.aws.amazon.com/amazondynamodb/latest/developerguide/QueryAndScan.html#Count

0 comments on commit c107ac3

Please sign in to comment.