Merge pull request #201 from aerospike/3.0.0-release-candidate

3.0.0 release candidate

BUILD.md

@@ -1,5 +1,7 @@
 # Manually Building the Python Client for Aerospike
 
+First clone this repository to get the necessary files.
+
 ## Dependencies
 
 The Python client for Aerospike works with Python 2.7, 3.4, 3.5, 3.6 running on
@@ -24,6 +26,8 @@ The following are dependencies for:
 sudo yum install openssl-devel
 sudo yum install python26-devel # on CentOS 6 and similar
 sudo yum install python-devel # on CentOS 7
+# Possibly needed
+sudo yum install python-setuptools
 ```
 
 To get `python26-devel` on older distros such as CentOS 5, see [Stack Overflow](http://stackoverflow.com/a/11684053/582436).
@@ -107,6 +111,12 @@ export AEROSPIKE_LUA_PATH=/path/to/aerospike-lua-core/src
 python setup.py build --force
 ```
 
+If using sudo, you may need to set the values inline with the command:
+
+```
+sudo DOWNLOAD_C_CLIENT=0 AEROSPIKE_C_HOME=/path/to/aerospike-c-client AEROSPIKE_LUA_PATH=/path/to/aerospike-lua-core/src python setup.py build --force
+```
+
 
 ## Install
 

VERSION

@@ -1 +1 @@
-2.2.3
+3.0.0

api-changes.md

@@ -0,0 +1,104 @@
+# API Changes
+
+## Version 3.0.0
+
+
+### Additional Features
+
+* Updated to C client `4.3.1`
+* Added a new list increment operation OP_LIST_INCREMENT . It can be used to increase an element of a list by a provided amount.
+* Added the option to specify a specific node to run on a scan on, via a new optional nodename parameter.
+* Added the option to specify that Query.results and Query.foreach should not return bins, via a new optional parameter options to both methods.
+* Added aerospike.info_all() to allow sending of an info command to all nodes in the current cluster.
+* Added linearize_read option to read policies. Requires Enterprise server >= 4.0.0
+
+### Deprecations
+* `client#info` has been deprecated. In order to send requests to the entire cluster, the new method `client#info_all` should be used. In order to send requests to hosts specified in a list, we recommend a  loop invoking multiple calls to aerospike.info_node. See below for an example implementation:
+
+```python
+
+def info_to_host_list(client, request, hosts, policy=None):
+	output = {}
+	for host in hosts:
+	    try:
+	        response = client.info_node(request, host, policy)
+	        output[host] = response
+	    except Exception as e:
+	        #  Handle the error gracefully here
+	        output[host] = e
+	return output
+```
+
+* Setting of global policy defaults via top level entries in the 'policies' dictionary in the constructor config dict has been deprecated. See the constructor documentation for the new recommended method of specifying defaults.
+
+
+### Backwards Incompatible API changes
+#### LDT Removal
+Removed LDT (`client#llist`) methods and support. Server version 3.14 is the last version of the Aerospike server to support the functionality.
+
+
+#### Index Methods
+
+* Methods which create indexes, ( `index_string_create`, `index_integer_create`, `index_list_create`, `index_map_keys_create`, `index_map_values_create` and, `index_geo2dsphere_create` ),  will now raise an `IndexFoundError` if the specified bin has already been indexed, or if an index with the same name already exists.
+
+* Methods which drop indexes (`index_remove`), will now raise an `IndexNotFound` if the named index does not exist.
+
+#### Shared memory layout change
+Shared memory layout has changed, and accordingly the default SHM key has changed from `0xA6000000` to `0xA7000000` . If manually specifiying an
+SHM key, it is crucial to ensure that a separate key is used in order to prevent this version's client from sharing memory with a previous version.
+
+#### Changed and removed policy field names:
+In all policies, (except for `info` and `admin`), `timeout` has been split into `total_timeout` and `socket_timeout`
+`total_timeout` is an int representing total transaction timeout in milliseconds. The `total_timeout` is tracked on the client and sent to the server along with the transaction in the wire protocol. The client will most likely timeout first, but the server also has the capability to timeout the transaction. If `total_timeout` is not zero and `total_timeout` is reached before the transaction completes, the transaction will return error `TimeoutError`. If `total_timeout` is zero, there will be no total time limit. See the documentation for individual policies for the default values.
+
+`socket_timeout` is an int representing socket idle timeout in milliseconds when processing a database command. If `socket_timeout` is not zero and the socket has been idle for at least `socket_timeout`, both max_retries and `total_timeout` are checked. If `max_retries` and `total_timeout` are not exceeded, the transaction is retried. If both `socket_timeout` and `total_timeout` are non-zero and `socket_timeout > total_timeout`, then `socket_timeout` will be set to `total_timeout`. If `socket_timeout` is zero, there will be no socket idle limit. See the documentation for individual policies for the default values.
+
+`retry` has beeen removed.
+
+`max_retries` has been added, it is an integer specifying the number of times a transaction should be retried before aborting. If `max_retries` is exceeded, `TimeoutError` will be raised.
+
+WARNING: Database writes that are not idempotent (such as `client#increment`) should not be retried because the write operation may be performed multiple times if the client timed out previous transaction attempts. It’s important to use a distinct write policy for non-idempotent writes which sets `max_retries` = 0;
+
+The default value for `max_retries` is 2.
+
+#### Changes in policy defaults for `aerospike.client` constructor
+In this version, individual config dictionaries (`read`, `write`, `apply`, `operate`, `scan`, `query`, `batch`, `remove`) for method types should be used inside of the top level `policies` dictionary for setting policies. In previous versions, individual policies were set at the top level `policies` dictionary rather than in per method type dictionaries. The type of policy which affects each method can be found in the documenation. See the main documentation for the keys and values available for these new configuration dictionaries. See below for an example of the change in constructor usage:
+
+```python
+#  Pre Python client 3.0.0
+
+hosts = [('localhost', 3000)]
+
+timeout = 2345
+key_policy = aerospike.POLICY_KEY_SEND
+
+# This changes defaults for all methods, requiring a config dictionary to be passed in to all methods which
+# should use different values.
+policies = {timeout': timeout, 'key': key_policy, 'retry': aerospike.POLICY_RETRY_ONCE}
+config = {'hosts': hosts, 'policies': policies}
+
+client = aerospike.client(config)
+```
+
+```python
+#  Post Python client 3.0.0
+
+hosts = [('localhost', 3000)]
+
+write_total_timeout = 4321
+read_total_timeout = 1234
+write_key_policy = aerospike.POLICY_KEY_SEND
+
+read_policy = {'total_timeout': read_total_timeout, 'max_retries': 1}
+write_policy = {'total_timeout': write_total_timeout, 'key': write_key_policy, 'max_retries': 1}
+
+# operate policies affect methods such as client#increment, so these should not be retried since they are not idempotent.
+operate_policy = {'max_retries': 0}
+
+# Change the defaults for read, write, and operate methods, all other methods will use builtin defaults.
+policies = {'read': read_policy, 'write': write_policy, 'operate': operate_policy}
+
+config = {'hosts': hosts, 'policies': policies}
+
+client = aerospike.client(config)
+```

doc/aerospike.rst

@@ -53,19 +53,26 @@ in an in-memory primary index.
                 * **system_path** the location of the system modules such as ``aerospike.lua`` (default: ``/usr/local/aerospike/lua``)
                 * **user_path** the location of the user's record and stream UDFs . (default: ``./``)
             * **policies** a :class:`dict` of policies
-                * **timeout** default connection timeout in milliseconds
-                * **key** default key policy, with values such as :data:`aerospike.POLICY_KEY_DIGEST`
-                * **exists** default exists policy, with values such as :data:`aerospike.POLICY_EXISTS_CREATE`
-                * **gen** default generation policy, with values such as :data:`aerospike.POLICY_GEN_IGNORE`
-                * **retry** default retry policy, with values such as :data:`aerospike.POLICY_RETRY_NONE`
-                * **consistency_level** default consistency level policy, with values such as :data:`aerospike.POLICY_CONSISTENCY_ONE`
-                * **replica** default replica policy, with values such as :data:`aerospike.POLICY_REPLICA_MASTER`
-                * **commit_level** default commit level policy, with values such as :data:`aerospike.POLICY_COMMIT_LEVEL_ALL`
+                * **read** A dictionary containing read policies. See :ref:`aerospike_read_policies` for available policy fields and values.
+                * **write** A dictionary containing write policies. See :ref:`aerospike_write_policies` for available policy fields and values.
+                * **apply** A dictionary containing apply policies. See :ref:`aerospike_apply_policies` for available policy fields and values.
+                * **operate** A dictionary containing operate policies. See :ref:`aerospike_operate_policies` for available policy fields and values.
+                * **remove** A dictionary containing remove policies. See :ref:`aerospike_remove_policies` for available policy fields and values.
+                * **query** A dictionary containing query policies. See :ref:`aerospike_query_policies` for available policy fields and values.
+                * **scan** A dictionary containing scan policies. See :ref:`aerospike_scan_policies` for available policy fields and values.
+                * **batch** A dictionary containing batch policies. See :ref:`aerospike_batch_policies` for available policy fields and values.
+                * **total_timeout** default connection timeout in milliseconds (**Deprecated**: set this the individual policy dictionaries)
+                * **key** default key policy, with values such as :data:`aerospike.POLICY_KEY_DIGEST` (**Deprecated**: set this individually in the 'read', 'write', 'apply', 'operate', 'remove' policy dictionaries)
+                * **exists** default exists policy, with values such as :data:`aerospike.POLICY_EXISTS_CREATE` (**Deprecated**: set in the 'write' policies dictionary)
+                * **max_retries** a :class:`int` representing the number of times to retry a transaction (**Deprecated**: set this the individual policy dictionaries)
+                * **consistency_level** default consistency level policy, with values such as :data:`aerospike.POLICY_CONSISTENCY_ONE` (**Deprecated**: set this individually as needed in the 'read','operate', 'batch' policy dictionaries)
+                * **replica** default replica policy, with values such as :data:`aerospike.POLICY_REPLICA_MASTER` (**Deprecated**: set this in one or all of the 'read', 'write', 'apply', 'operate', 'remove' policy dictionaries)
+                * **commit_level** default commit level policy, with values such as :data:`aerospike.POLICY_COMMIT_LEVEL_ALL` (**Deprecated**: set this as needed individually in the 'write', 'apply', 'operate', 'remove' policy dictionaries)
             * **shm** a :class:`dict` with optional shared-memory cluster tending parameters. Shared-memory cluster tending is on if the :class:`dict` is provided. If multiple clients are instantiated talking to the same cluster the *shm* cluster-tending should be used.
                 * **max_nodes** maximum number of nodes allowed. Pad so new nodes can be added without configuration changes (default: 16)
                 * **max_namespaces** similarly pad (default: 8)
                 * **takeover_threshold_sec** take over tending if the cluster hasn't been checked for this many seconds (default: 30)
-                * **shm_key** explicitly set the shm key for this client. If **use_shared_connection** is not set, or set to `False`, the user must provide a value for this field in order for shared memory to work correctly. If , and only if, **use_shared_connection** is set to `True`, the key will be implicitly evaluated per unique hostname, and can be inspected with :meth:`~aerospike.Client.shm_key` . It is still possible to specify a key when using **use_shared_connection** = `True`. (default: 0xA5000000)
+                * **shm_key** explicitly set the shm key for this client. If **use_shared_connection** is not set, or set to `False`, the user must provide a value for this field in order for shared memory to work correctly. If , and only if, **use_shared_connection** is set to `True`, the key will be implicitly evaluated per unique hostname, and can be inspected with :meth:`~aerospike.Client.shm_key` . It is still possible to specify a key when using **use_shared_connection** = `True`. (default: 0xA7000000)
             * **use_shared_connection** :class:`bool` indicating whether this instance should share its connection to the Aerospike cluster with other client instances in the same process. (default: ``False``)
             * **tls** a :class:`dict` of optional TLS configuration parameters. **TLS usage requires Aerospike Enterprise Edition**
                 * **enable** a :class:`bool` indicating whether tls should be enabled or not. Default: ``False``
@@ -76,23 +83,21 @@ in an in-memory primary index.
                 * **keyfile** :class:`str` Path to the client's key for mutual authentication. By default mutual authentication is disabled.
                 * **cert_blacklist** :class:`str` Path to a certificate blacklist file. The file should contain one line for each blacklisted certificate. Each line starts with the certificate serial number expressed in hex. Each entry may optionally specify the issuer name of the certificate (serial numbers are only required to be unique per issuer). Example records: 867EC87482B2 /C=US/ST=CA/O=Acme/OU=Engineering/CN=Test Chain CA E2D4B0E570F9EF8E885C065899886461
                 * **certfile** :class:`str` Path to the client's certificate chain file for mutual authentication. By default mutual authentication is disabled.
-                * **encrypt_only** :class:`bool` If ``True`` Only encrypt connections; do not verify certificates. By default TLS will verify certificates.
                 * **crl_check** :class:`bool` Enable CRL checking for the certificate chain leaf certificate. An error occurs if a suitable CRL cannot be found. By default CRL checking is disabled.
                 * **crl_check_all** :class:`bool` Enable CRL checking for the entire certificate chain. An error occurs if a suitable CRL cannot be found. By default CRL checking is disabled.
                 * **log_session_info** :class:`bool` Log session information for each connection.
             * **serialization** an optional instance-level :py:func:`tuple` of (serializer, deserializer). Takes precedence over a class serializer registered with :func:`~aerospike.set_serializer`.
             * **thread_pool_size** number of threads in the pool that is used in batch/scan/query commands (default: 16)
-            * **max_threads** size of the synchronous connection pool for each server node (default: 300) *DEPRECATED*
             * **max_socket_idle** Maximum socket idle time in seconds.  Connection pools will discard sockets that have
 			been idle longer than the maximum.  The value is limited to 24 hours (86400).
 	 		It's important to set this value to a few seconds less than the server's proto-fd-idle-ms
 			(default 60000 milliseconds or 1 minute), so the client does not attempt to use a socket
 			that has already been reaped by the server.
 			Default: 0 seconds (disabled) for non-TLS connections, 55 seconds for TLS connections.
             * **max_conns_per_node** maximum number of pipeline connections allowed for each node 
-            * **batch_direct** whether to use the batch-direct protocol (default: ``False``, so will use batch-index if available)
+            * **batch_direct** whether to use the batch-direct protocol (default: ``False``, so will use batch-index if available) (**Deprecated**: set 'use_batch_direct' in the batch policy dictionary)
             * **tend_interval** polling interval in milliseconds for tending the cluster (default: 1000)
-            * **compression_threshold** compress data for transmission if the object size is greater than a given number of bytes (default: 0, meaning 'never compress')
+            * **compression_threshold** compress data for transmission if the object size is greater than a given number of bytes (default: 0, meaning 'never compress') (**Deprecated**, set this in the 'write' policy dictionary)
             * **cluster_name** only server nodes matching this name will be used when determining the cluster
 
     :return: an instance of the :py:class:`aerospike.Client` class.
@@ -111,7 +116,7 @@ in an in-memory primary index.
         # which is appropriate for a multi-process context, such as a webserver
         config = {
             'hosts':    [ ('127.0.0.1', 3000) ],
-            'policies': {'timeout': 1000},
+            'policies': {'read': {total_timeout': 1000}},
             'shm':      { }}
         client = aerospike.client(config)
 
@@ -134,11 +139,10 @@ in an in-memory primary index.
         tls_host_tuple = (tls_ip, tls_port, tls_name)
         hosts = [tls_host_tuple]
 
-        # Example configuration which will use TLS to encrypt only
+        # Example configuration which will use TLS with the specifed cafile
         tls_config = {
             "cafile": "/path/to/cacert.pem",
-            "enable": True,
-            "encrypt_only": True,
+            "enable": True
         }
 
         client = aerospike.client({
@@ -540,6 +544,19 @@ Operators for the multi-ops method :py:meth:`~aerospike.Client.operate`.
             "val": [ 123, 456 ]
         }
 
+.. data:: OP_LIST_INCREMENT
+
+    Increment the value of an item at the given index in a list stored in the specified bin
+
+    .. code-block:: python
+
+        {
+            "op": aerospike.OP_LIST_INCREMENT,
+            "bin": "bin_name",
+            "index": 2,
+            "val": 5
+        }
+
 .. data:: OP_LIST_POP
 
     Remove and return the element at a specified index of a bin with :class:`list` type data