Skip to content
Permalink
Browse files
IGNITE-14851 Enable partition awareness by default, fix unnecessary d…
…ouble connect - Fixes #42.
  • Loading branch information
ivandasch committed Jun 11, 2021
1 parent c64fb10 commit fbe61f3779a8da3d2218b969cb92d7515312de69
Showing 15 changed files with 129 additions and 62 deletions.
@@ -84,6 +84,7 @@ File: `transactions.py`_.

Client transactions are supported for caches with
:py:attr:`~pyignite.datatypes.cache_config.CacheAtomicityMode.TRANSACTIONAL` mode.
**Supported only python 3.7+**

Let's create transactional cache:

@@ -97,28 +98,28 @@ Let's start a transaction and commit it:
.. literalinclude:: ../examples/transactions.py
:language: python
:dedent: 8
:lines: 36-41
:lines: 36-42

Let's check that the transaction was committed successfully:

.. literalinclude:: ../examples/transactions.py
:language: python
:dedent: 8
:lines: 44-45
:lines: 45-46

Let's check that raising exception inside `async with` block leads to transaction's rollback

.. literalinclude:: ../examples/transactions.py
:language: python
:dedent: 8
:lines: 48-59
:lines: 49-61

Let's check that timed out transaction is successfully rolled back

.. literalinclude:: ../examples/transactions.py
:language: python
:dedent: 8
:lines: 62-73
:lines: 64-75

See more info about transaction's parameters in a documentation of :py:meth:`~pyignite.aio_client.AioClient.tx_start`

@@ -219,35 +219,35 @@ Let's create transactional cache:
.. literalinclude:: ../examples/transactions.py
:language: python
:dedent: 8
:lines: 82-85
:lines: 84-87

Let's start a transaction and commit it:

.. literalinclude:: ../examples/transactions.py
:language: python
:dedent: 8
:lines: 88-92
:lines: 90-96

Let's check that the transaction was committed successfully:

.. literalinclude:: ../examples/transactions.py
:language: python
:dedent: 8
:lines: 94-95
:lines: 98-99

Let's check that raising exception inside `with` block leads to transaction's rollback

.. literalinclude:: ../examples/transactions.py
:language: python
:dedent: 8
:lines: 98-108
:lines: 102-113

Let's check that timed out transaction is successfully rolled back

.. literalinclude:: ../examples/transactions.py
:language: python
:dedent: 8
:lines: 111-121
:lines: 116-126

See more info about transaction's parameters in a documentation of :py:meth:`~pyignite.client.Client.tx_start`

Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
@@ -22,6 +22,7 @@ Welcome to Apache Ignite binary client Python API documentation!

readme
modules
partition_awareness
examples
async_examples

@@ -0,0 +1,63 @@
.. Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
this work for additional information regarding copyright ownership.
The ASF licenses this file to You under the Apache License, Version 2.0
(the "License"); you may not use this file except in compliance with
the License. You may obtain a copy of the License at
.. http://www.apache.org/licenses/LICENSE-2.0
.. Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
===================
Partition Awareness
===================

Partition awareness allows the thin client to send query requests directly to the node that owns the queried data.

Without partition awareness, an application that is connected to the cluster via a thin client executes all queries and operations via a single server node that acts as a proxy for the incoming requests. These operations are then re-routed to the node that stores the data that is being requested. This results in a bottleneck that could prevent the application from scaling linearly.

.. image:: images/partitionawareness01.png
:alt: Without partition awareness

Notice how queries must pass through the proxy server node, where they are routed to the correct node.

With partition awareness in place, the thin client can directly route queries and operations to the primary nodes that own the data required for the queries. This eliminates the bottleneck, allowing the application to scale more easily.

.. image:: images/partitionawareness02.png
:alt: With partition awareness

Partition awareness can be enabled or disabled by setting `partition_aware` parameter in
:meth:`pyignite.client.Client.__init__` or :meth:`pyignite.aio_client.AioClient.__init__` to `True` (by default)
or `False`.

Also, it is recommended to pass list of address and port pairs of all server nodes
to :meth:`pyignite.client.Client.connect` or to :meth:`pyignite.aio_client.AioClient.connect`.

For example:

.. code-block:: python3

from pyignite import Client

client = Client(
partition_awareness=True
)
nodes = [('10.128.0.1', 10800), ('10.128.0.2', 10800),...]
with client.connect(nodes):
....

.. code-block:: python3

from pyignite import AioClient

client = AioClient(
partition_awareness=True
)
nodes = [('10.128.0.1', 10800), ('10.128.0.2', 10800),...]
async with client.connect(nodes):
....
@@ -14,7 +14,7 @@
limitations under the License.
pyignite.transaction module
=========================
===========================

.. automodule:: pyignite.transaction
:members:
@@ -35,19 +35,21 @@ async def async_example():
# starting transaction
key = 1
async with client.tx_start(
isolation=TransactionIsolation.REPEATABLE_READ, concurrency=TransactionConcurrency.PESSIMISTIC
isolation=TransactionIsolation.REPEATABLE_READ,
concurrency=TransactionConcurrency.PESSIMISTIC
) as tx:
await cache.put(key, 'success')
await tx.commit()

# key=1 value=success
val = await cache.get(key)
print(f"key=1 value={val}")
print(f"key={key} value={val}")

# rollback transaction.
try:
async with client.tx_start(
isolation=TransactionIsolation.REPEATABLE_READ, concurrency=TransactionConcurrency.PESSIMISTIC
isolation=TransactionIsolation.REPEATABLE_READ,
concurrency=TransactionConcurrency.PESSIMISTIC
):
await cache.put(key, 'fail')
raise RuntimeError('test')
@@ -56,7 +58,7 @@ async def async_example():

# key=1 value=success
val = await cache.get(key)
print(f"key=1 value={val}")
print(f"key={key} value={val}")

# rollback transaction on timeout.
try:
@@ -70,7 +72,7 @@ async def async_example():

# key=1 value=success
val = await cache.get(key)
print(f"key=1 value={val}")
print(f"key={key} value={val}")

# destroy cache
await cache.destroy()
@@ -85,40 +87,43 @@ def sync_example():
})

# starting transaction
key = 1
with client.tx_start(
isolation=TransactionIsolation.REPEATABLE_READ, concurrency=TransactionConcurrency.PESSIMISTIC
isolation=TransactionIsolation.REPEATABLE_READ,
concurrency=TransactionConcurrency.PESSIMISTIC
) as tx:
cache.put(1, 'success')
cache.put(key, 'success')
tx.commit()

# key=1 value=success
print(f"key=1 value={cache.get(1)}")
print(f"key={key} value={cache.get(key)}")

# rollback transaction.
try:
with client.tx_start(
isolation=TransactionIsolation.REPEATABLE_READ, concurrency=TransactionConcurrency.PESSIMISTIC
isolation=TransactionIsolation.REPEATABLE_READ,
concurrency=TransactionConcurrency.PESSIMISTIC
):
cache.put(1, 'fail')
cache.put(key, 'fail')
raise RuntimeError('test')
except RuntimeError:
pass

# key=1 value=success
print(f"key=1 value={cache.get(1)}")
print(f"key={key} value={cache.get(key)}")

# rollback transaction on timeout.
try:
with client.tx_start(timeout=1.0, label='long-tx') as tx:
cache.put(1, 'fail')
cache.put(key, 'fail')
time.sleep(2.0)
tx.commit()
except CacheError as e:
# Cache transaction timed out: GridNearTxLocal[...timeout=1000, ... label=long-tx]
print(e)

# key=1 value=success
print(f"key=1 value={cache.get(1)}")
print(f"key={key} value={cache.get(key)}")

# destroy cache
cache.destroy()
@@ -60,21 +60,18 @@ class AioClient(BaseClient):
Asynchronous Client implementation.
"""

def __init__(self, compact_footer: bool = None, partition_aware: bool = False, **kwargs):
def __init__(self, compact_footer: bool = None, partition_aware: bool = True, **kwargs):
"""
Initialize client.
:param compact_footer: (optional) use compact (True, recommended) or
full (False) schema approach when serializing Complex objects.
Default is to use the same approach the server is using (None).
Apache Ignite binary protocol documentation on this topic:
https://apacheignite.readme.io/docs/binary-client-protocol-data-format#section-schema
https://ignite.apache.org/docs/latest/binary-client-protocol/data-format#schema
:param partition_aware: (optional) try to calculate the exact data
placement from the key before to issue the key operation to the
server node:
https://cwiki.apache.org/confluence/display/IGNITE/IEP-23%3A+Best+Effort+Affinity+for+thin+clients
The feature is in experimental status, so the parameter is `False`
by default. This will be changed later.
server node, `True` by default.
"""
super().__init__(compact_footer, partition_aware, **kwargs)
self._registry_mux = asyncio.Lock()
@@ -494,7 +491,7 @@ def tx_start(self, concurrency: TransactionConcurrency = TransactionConcurrency.
isolation: TransactionIsolation = TransactionIsolation.REPEATABLE_READ,
timeout: Union[int, float] = 0, label: Optional[str] = None) -> 'AioTransaction':
"""
Start async thin client transaction.
Start async thin client transaction. **Supported only python 3.7+**
:param concurrency: (optional) transaction concurrency, see
:py:class:`~pyignite.datatypes.transactions.TransactionConcurrency`
@@ -17,7 +17,7 @@
This module contains functions, that are (more or less) directly mapped to
Apache Ignite binary protocol operations. Read more:
https://apacheignite.readme.io/docs/binary-client-protocol#section-client-operations
https://ignite.apache.org/docs/latest/binary-client-protocol/binary-client-protocol#client-operations
When the binary client protocol changes, these functions also change. For
stable end user API see :mod:`pyignite.client` module.
@@ -335,31 +335,21 @@ def __exit__(self, exc_type, exc_val, exc_tb):

class Client(BaseClient):
"""
This is a main `pyignite` class, that is build upon the
:class:`~pyignite.connection.Connection`. In addition to the attributes,
properties and methods of its parent class, `Client` implements
the following features:
* cache factory. Cache objects are used for key-value operations,
* Ignite SQL endpoint,
* binary types registration endpoint.
Synchronous Client implementation.
"""

def __init__(self, compact_footer: bool = None, partition_aware: bool = False, **kwargs):
def __init__(self, compact_footer: bool = None, partition_aware: bool = True, **kwargs):
"""
Initialize client.
:param compact_footer: (optional) use compact (True, recommended) or
full (False) schema approach when serializing Complex objects.
Default is to use the same approach the server is using (None).
Apache Ignite binary protocol documentation on this topic:
https://apacheignite.readme.io/docs/binary-client-protocol-data-format#section-schema
https://ignite.apache.org/docs/latest/binary-client-protocol/data-format#schema
:param partition_aware: (optional) try to calculate the exact data
placement from the key before to issue the key operation to the
server node:
https://cwiki.apache.org/confluence/display/IGNITE/IEP-23%3A+Best+Effort+Affinity+for+thin+clients
The feature is in experimental status, so the parameter is `False`
by default. This will be changed later.
server node, `True` by default.
"""
super().__init__(compact_footer, partition_aware, **kwargs)

0 comments on commit fbe61f3

Please sign in to comment.