Skip to content

Commit

Permalink
Merge pull request #233 from aerospike/3.6.0-in-progress
Browse files Browse the repository at this point in the history
3.6.0 in progress
  • Loading branch information
aerospikerobertmarks committed Jan 9, 2019
2 parents 02f7993 + d1e7761 commit 7676097
Show file tree
Hide file tree
Showing 22 changed files with 209 additions and 276 deletions.
65 changes: 22 additions & 43 deletions BUILD.md
Original file line number Diff line number Diff line change
Expand Up @@ -13,16 +13,15 @@ The client depends on:
- OpenSSL
- The Aerospike C client


### RedHat 6+ and CentOS 6+

The following are dependencies for:

- RedHat Enterprise (RHEL) 6 or newer
- CentOS 6 or newer
- Related distributions which use the `yum` package manager
- RedHat Enterprise (RHEL) 6 or newer
- CentOS 6 or newer
- Related distributions which use the `yum` package manager

```
```sh
sudo yum install openssl-devel
sudo yum install python26-devel # on CentOS 6 and similar
sudo yum install python-devel # on CentOS 7
Expand All @@ -32,21 +31,19 @@ sudo yum install python-setuptools

To get `python26-devel` on older distros such as CentOS 5, see [Stack Overflow](http://stackoverflow.com/a/11684053/582436).


### Debian 6+ and Ubuntu 14.04+

The following are dependencies for:

- Debian 6 or newer
- Ubuntu 14.04 or newer
- Related distributions which use the `apt` package manager
- Debian 6 or newer
- Ubuntu 14.04 or newer
- Related distributions which use the `apt` package manager

```
```sh
sudo apt-get install libssl-dev
sudo apt-get install build-essential python-dev
```


### OS X

By default OS X will be missing command line tools. On Mavericks (OS X 10.9)
Expand All @@ -58,31 +55,29 @@ The dependencies can be installed through the OS X package manager [Homebrew](ht

brew install openssl


## Build

To build the library:

git submodule update --init
python setup.py build --force


The helper `scripts/aerospike-client-c.sh` is triggered by `setup.py` to
download the appropriate C client. However, if one is present this will not
happen, and a build may fail against an old client. At that point you should
remove the directory `aerospike-client-c` and run the build command again.


### Troubleshooting OS X builds

Building on OS X versions >= 10.11 , may cause a few additional errors to be generated. If the build command fails with an
error similar to: `error: could not create '/usr/local/aerospike/lua': Permission denied` there are a couple of options:

- Rerun the build command with the additional command line flags `--user --prefix=` *Note that there are no charcters after the '='.* This will cause the library to only be installed for the current user, and store the library's data files in a user specific location.
- rerun the command with sudo.

If an error similar to `ld: targeted OS version does not support use of thread local variables` appears, it can be fixed by temporarily setting the `MACOSX_DEPLOYMENT_TARGET` environment variable to `'10.12'` e.g.

```
```sh
MACOSX_DEPLOYMENT_TARGET=10.12 python setup.py build --force
MACOSX_DEPLOYMENT_TARGET=10.12 python setup.py install --force
```
Expand All @@ -96,15 +91,15 @@ you will need to first build the C client manually.
2. Install the dependencies. See the [README](https://github.com/aerospike/aerospike-client-c/blob/master/README.md).
3. Change directory to the C client, and build it.

```
```sh
git submodule update --init
make
```

4. Clone the [aerospike/aerospike-lua-core](https://github.com/aerospike/aerospike-lua-core) repo from GitHub.
5. Change directory to the Python client and build it.

```
```sh
export DOWNLOAD_C_CLIENT=0
export AEROSPIKE_C_HOME=/path/to/aerospike-c-client
export AEROSPIKE_LUA_PATH=/path/to/aerospike-lua-core/src
Expand All @@ -113,65 +108,49 @@ python setup.py build --force

If using sudo, you may need to set the values inline with the command:

```
```bash
sudo DOWNLOAD_C_CLIENT=0 AEROSPIKE_C_HOME=/path/to/aerospike-c-client AEROSPIKE_LUA_PATH=/path/to/aerospike-lua-core/src python setup.py build --force
```


## Install

To install the library:

python setup.py install --force



### Troubleshooting OS X Installation

Installing on OS X versions >= 10.11 , may cause a few additional errors to be generated. If the install command fails with an
error similar to: `error: could not create '/usr/local/aerospike/lua': Permission denied` there are a couple of options:

- Rerun the install command with the additional command line flags `--user --prefix=` *Note that there are no charcters after the '='.* This will cause the library to only be installed for the current user, and store the library's data files in a user specific location.
- rerun the command with sudo.

### Lua System Modules

Stream UDF functionality requires a local copy of the system Lua modules.
By default, those Lua files are copied to an `aerospike` directory inside of Python's' installation path for system dependent packages. This directory can be viewed by running `python -c "import sys; print(sys.prefix);" `


**Note** The default search location for the lua system files is `/usr/local/aerospike/lua`. If the .lua files are stored somewhere besides `/usr/local/aerospike/lua`. and you wish to perform Stream UDF operations it will be necessary to specify the locations of the system modules as a configuration parameter to the Aerospike client constructor:

config = {'hosts': [('127.0.0.1', 3000)], 'lua': {'system_path': '/path/to/lua'} ...}
my_client = aerospike.client(config)

## Examples

**Note** If you did not install the library, then you will need to setup your `PYTHONPATH` environment variable. The `PYTHONPATH` should contain an entry for the directory where the Python module is stored. This is usually in `build/lib.*`.


Examples are in the `examples` directory. The following examples are available:

* `kvs.py` — Key-Value Store API Example
* `query.py` — Query API Example
* `scan.py` — Scan API Example
* `info.py` — Info API Example
* `simple.lua` — Simple UDF Example
- `kvs.py` — Key-Value Store API Example
- `query.py` — Query API Example
- `scan.py` — Scan API Example
- `info.py` — Info API Example
- `simple.lua` — Simple UDF Example

Each example provides help/usage information when you specify the `--help` option. For example, for help on the `kvs.py` example, then run:

python examples/client/kvs.py --help


### Running Examples

Simply call `python` with the path to the example

python examples/client/kvs.py


## License

The Aerospike Python Client is made availabled under the terms of the Apache License, Version 2, as stated in the file `LICENSE`.

Individual files may be made available under their own specific license,
Individual files may be made available under their own specific license,
all compatible with Apache License, Version 2. Please see individual files for details.
15 changes: 0 additions & 15 deletions README.rst
Original file line number Diff line number Diff line change
Expand Up @@ -91,21 +91,6 @@ using an outdated ``pip``.
Versions of ``pip`` older than 7.0.0 should be upgraded, as well as versions of
``setuptools`` older than 18.0.0.

Lua files
~~~~~~~~~~

The system .lua files used for client side aggregation will be installed.
By default pip will install the .lua files in a subdirectory named `aerospike/lua/` inside of the Python
installations directory for platform specific files. The location of the files can be found by running:

``pip show -f aerospike``


**Note** If the .lua files are stored somewhere besides `/usr/local/aerospike/lua`. and you wish to perform Stream UDF operations it will be necessary to specify the locations of the system modules as a configuration parameter to the Aerospike client constructor:

config = {'hosts': [('127.0.0.1', 3000)], 'lua': {'system_path': '/path/to/lua'} ...}
my_client = aerospike.client(config)


OS X Installation
~~~~~~~~~~~~~~~~~~
Expand Down
2 changes: 1 addition & 1 deletion VERSION
Original file line number Diff line number Diff line change
@@ -1 +1 @@
3.5.0
3.6.0
145 changes: 3 additions & 142 deletions api-changes.md
Original file line number Diff line number Diff line change
@@ -1,143 +1,4 @@
# API Changes
# Backward Incompatible API Changes


## Version 3.2.0
### Additional Features
* Updated to C client `4.3.11`
* Added `client.map_get_by_value_list` and `client.map_get_by_key_list` These methods require Aerospike Server version >= 3.16.0.1 .


## Version 3.1.0

### Additional Features
* Updated to C client `4.3.6`
* Added `exists` policy field to operate policies

### Backwards Incompatible API changes
* Updated the args passed to AerospikeError constructor internally to contain 5 arguments.
The arguments previously were `error code`, `error message`, `error file`, `error line`.
A fifth argument `in_doubt` has been added to the internal calls. so the arguments passed to the constructor are now : `error_code`, `error_message`, `error_file`, `error_line`, `in_doubt`

This means that code such as the following will now raise a ValueError

```python
try:
client.get(key)
except AerospikeError as e:
code, msg, file, line = e.args
print(code, msg, file, line)
```

This can be fixed by unpacking the fifth value from the Error's `args` tuple

```python
try:
client.get(key)
except AerospikeError as e:
code, msg, file, line, in_doubt = e.args
print(code, msg, file, line)
```


## Version 3.0.0


### Additional Features

* Updated to C client `4.3.1`
* Added a new list increment operation OP_LIST_INCREMENT . It can be used to increase an element of a list by a provided amount.
* Added the option to specify a specific node to run on a scan on, via a new optional nodename parameter.
* Added the option to specify that Query.results and Query.foreach should not return bins, via a new optional parameter options to both methods.
* Added aerospike.info_all() to allow sending of an info command to all nodes in the current cluster.
* Added linearize_read option to read policies. Requires Enterprise server >= 4.0.0

### Deprecations
* `client#info` has been deprecated. In order to send requests to the entire cluster, the new method `client#info_all` should be used. In order to send requests to hosts specified in a list, we recommend a loop invoking multiple calls to aerospike.info_node. See below for an example implementation:

```python

def info_to_host_list(client, request, hosts, policy=None):
output = {}
for host in hosts:
try:
response = client.info_node(request, host, policy)
output[host] = response
except Exception as e:
# Handle the error gracefully here
output[host] = e
return output
```

* Setting of global policy defaults via top level entries in the 'policies' dictionary in the constructor config dict has been deprecated. See the constructor documentation for the new recommended method of specifying defaults.


### Backwards Incompatible API changes
#### LDT Removal
Removed LDT (`client#llist`) methods and support. Server version 3.14 is the last version of the Aerospike server to support the functionality.


#### Index Methods

* Methods which create indexes, ( `index_string_create`, `index_integer_create`, `index_list_create`, `index_map_keys_create`, `index_map_values_create` and, `index_geo2dsphere_create` ), will now raise an `IndexFoundError` if the specified bin has already been indexed, or if an index with the same name already exists.

* Methods which drop indexes (`index_remove`), will now raise an `IndexNotFound` if the named index does not exist.

#### Shared memory layout change
Shared memory layout has changed, and accordingly the default SHM key has changed from `0xA6000000` to `0xA7000000` . If manually specifiying an
SHM key, it is crucial to ensure that a separate key is used in order to prevent this version's client from sharing memory with a previous version.

#### Changed and removed policy field names:
In all policies, (except for `info` and `admin`), `timeout` has been split into `total_timeout` and `socket_timeout`
`total_timeout` is an int representing total transaction timeout in milliseconds. The `total_timeout` is tracked on the client and sent to the server along with the transaction in the wire protocol. The client will most likely timeout first, but the server also has the capability to timeout the transaction. If `total_timeout` is not zero and `total_timeout` is reached before the transaction completes, the transaction will return error `TimeoutError`. If `total_timeout` is zero, there will be no total time limit. See the documentation for individual policies for the default values.

`socket_timeout` is an int representing socket idle timeout in milliseconds when processing a database command. If `socket_timeout` is not zero and the socket has been idle for at least `socket_timeout`, both max_retries and `total_timeout` are checked. If `max_retries` and `total_timeout` are not exceeded, the transaction is retried. If both `socket_timeout` and `total_timeout` are non-zero and `socket_timeout > total_timeout`, then `socket_timeout` will be set to `total_timeout`. If `socket_timeout` is zero, there will be no socket idle limit. See the documentation for individual policies for the default values.

`retry` has beeen removed.

`max_retries` has been added, it is an integer specifying the number of times a transaction should be retried before aborting. If `max_retries` is exceeded, `TimeoutError` will be raised.

WARNING: Database writes that are not idempotent (such as `client#increment`) should not be retried because the write operation may be performed multiple times if the client timed out previous transaction attempts. It’s important to use a distinct write policy for non-idempotent writes which sets `max_retries` = 0;

The default value for `max_retries` is 2.

#### Changes in policy defaults for `aerospike.client` constructor
In this version, individual config dictionaries (`read`, `write`, `apply`, `operate`, `scan`, `query`, `batch`, `remove`) for method types should be used inside of the top level `policies` dictionary for setting policies. In previous versions, individual policies were set at the top level `policies` dictionary rather than in per method type dictionaries. The type of policy which affects each method can be found in the documenation. See the main documentation for the keys and values available for these new configuration dictionaries. See below for an example of the change in constructor usage:

```python
# Pre Python client 3.0.0

hosts = [('localhost', 3000)]

timeout = 2345
key_policy = aerospike.POLICY_KEY_SEND

# This changes defaults for all methods, requiring a config dictionary to be passed in to all methods which
# should use different values.
policies = {timeout': timeout, 'key': key_policy, 'retry': aerospike.POLICY_RETRY_ONCE}
config = {'hosts': hosts, 'policies': policies}

client = aerospike.client(config)
```

```python
# Post Python client 3.0.0

hosts = [('localhost', 3000)]

write_total_timeout = 4321
read_total_timeout = 1234
write_key_policy = aerospike.POLICY_KEY_SEND

read_policy = {'total_timeout': read_total_timeout, 'max_retries': 1}
write_policy = {'total_timeout': write_total_timeout, 'key': write_key_policy, 'max_retries': 1}

# operate policies affect methods such as client#increment, so these should not be retried since they are not idempotent.
operate_policy = {'max_retries': 0}

# Change the defaults for read, write, and operate methods, all other methods will use builtin defaults.
policies = {'read': read_policy, 'write': write_policy, 'operate': operate_policy}

config = {'hosts': hosts, 'policies': policies}

client = aerospike.client(config)
```
The documentation of backward incompatible API changes has been moved to
<https://www.aerospike.com/docs/client/python/usage/incompatible.html>.
23 changes: 22 additions & 1 deletion doc/aerospike.rst
Original file line number Diff line number Diff line change
Expand Up @@ -74,7 +74,12 @@ in an in-memory primary index.
* **max_nodes** maximum number of nodes allowed. Pad so new nodes can be added without configuration changes (default: 16)
* **max_namespaces** similarly pad (default: 8)
* **takeover_threshold_sec** take over tending if the cluster hasn't been checked for this many seconds (default: 30)
* **shm_key** explicitly set the shm key for this client. If **use_shared_connection** is not set, or set to `False`, the user must provide a value for this field in order for shared memory to work correctly. If , and only if, **use_shared_connection** is set to `True`, the key will be implicitly evaluated per unique hostname, and can be inspected with :meth:`~aerospike.Client.shm_key` . It is still possible to specify a key when using **use_shared_connection** = `True`. (default: 0xA7000000)
* **shm_key**
| explicitly set the shm key for this client.
| If **use_shared_connection** is not set, or set to `False`, the user must provide a value for this field in order for shared memory to work correctly.
| If , and only if, **use_shared_connection** is set to `True`, the key will be implicitly evaluated per unique hostname, and can be inspected with :meth:`~aerospike.Client.shm_key` .
| It is still possible to specify a key when using **use_shared_connection** = `True`.
| (default: 0xA8000000)
* **use_shared_connection** :class:`bool` indicating whether this instance should share its connection to the Aerospike cluster with other client instances in the same process. (default: ``False``)
* **tls** a :class:`dict` of optional TLS configuration parameters. **TLS usage requires Aerospike Enterprise Edition**
* **enable** a :class:`bool` indicating whether tls should be enabled or not. Default: ``False``
Expand Down Expand Up @@ -102,6 +107,16 @@ in an in-memory primary index.
* **tend_interval** polling interval in milliseconds for tending the cluster (default: 1000)
* **compression_threshold** compress data for transmission if the object size is greater than a given number of bytes (default: 0, meaning 'never compress') (**Deprecated**, set this in the 'write' policy dictionary)
* **cluster_name** only server nodes matching this name will be used when determining the cluster
* **rack_id**
| An integer. Rack where this client instance resides.
| **rack_aware** and **POLICY_REPLICA_PREFER_RACK** and server rack configuration must also be set to enable this functionality.
| Default 0.
* **rack_aware**
| Boolean. Track server rack data.
| This field is useful when directing read commands to the server node that contains the key and exists on the same rack as the client.
| This serves to lower cloud provider costs when nodes are distributed across different racks/data centers.
| **rack_id** and **POLICY_REPLICA_PREFER_RACK** and server rack configuration must also be set to enable this functionality.
| Default False
:return: an instance of the :py:class:`aerospike.Client` class.

Expand Down Expand Up @@ -1490,6 +1505,12 @@ Specifies which partition replica to read from.
Distribute reads across nodes containing key's master and replicated partition in round-robin fashion. Currently restricted to master and one prole.
.. data:: POLICY_REPLICA_PREFER_RACK
Try node on the same rack as the client first. If there are no nodes on the same rack, use POLICY_REPLICA_SEQUENCE instead.
**rack_aware** and **rack_id** must be set in the config argument of the client constructor in order to enable this functionality
.. rubric:: Retry Policy Options
Specifies the behavior of failed operations.
Expand Down

0 comments on commit 7676097

Please sign in to comment.