-
Notifications
You must be signed in to change notification settings - Fork 0
Server side transforms in Voldemort
Server-side transforms or predicate pushdowns are a way of transforming or filtering the values mapped to a key before transferring it to the client. This functionality is currently unsupported in Voldemort since there is no way to “pass-down” filter queries to server side. In absence of this feature, to filter out values, one needs to fetch it through the ‘get()’ API to client side, and then apply the filter queries there. There is unnecessary data exchanged over the network leading to wastage of network bandwidth and increased latency of the get API.
Similarly for the put() API, if some filters need to be applied on (existing values in the Voldemort store + new values specified through put()), before writing the filtered data back into the store, we need some way to transmit the ‘filter queries’ along with the new values for the key.
If an application needs to support server-side transforms on a store, then it can define a ‘view’ on the store. For filtering in the get() API, the storeToView() method will apply the transforms on the values fetched from the store. For filtering in the put() API, the viewToStore() method will apply the transforms on the old+new data before writing it to the store.
If server-side transforms are implemented in the following way, the rest of the stores, current StoreClients are untouched.
Both viewToStore() and storeToView() APIs will take in transforms
- public V storeToView(Store<K, S> targetStore, K k, S s, T t) throws UnsupportedViewOperationException;
- public S viewToStore(Store<K, S> targetStore, K k, V v, T t) throws UnsupportedViewOperationException;
Define a new interface – ViewStoreClient<K, V, T> just like StoreClient<K, V> with following API additions
- public Versioned get(K key, T transforms);
- public Map<K, Versioned> getAll(Iterable keys, Map<K, T> transforms);
- public void put(K key, V value, T transforms);
Define a new interface – ViewStorageEngine<K, V, T> with following API changes to StorageEngine<K, V>
- public List<Versioned> get(K key, T transforms) throws VoldemortException;
- public Map<K, List<Versioned>> getAll(Iterable keys, Map<K, T> transforms) throws VoldemortException;
- public void put(K key, Versioned value, T transforms) throws VoldemortException;
Change the StorageConfiguration interface to add a new method –
- public StorageEngine<ByteArray, byte[], byte[]> getViewStore(String name);
The above get(), getAll(), put() APIs will apply the transforms on the values as defined by the storeToView() and viewToStore() methods in the view. And since there is a different StorageEngine interface for views, it will have to be handled in a special way in internal code
If server-side transforms are implemented in the following way, all the stores AND current StoreClients need to be changed. Since the transforms can only be applied in the presence of a view over a store, all the changes to the stores (except the ViewStorageEngine) and store clients are redundant.
Both viewToStore() and storeToView() APIs will take in transforms
- public V storeToView(Store<K, S> targetStore, K k, S s, T t) throws UnsupportedViewOperationException;
- public S viewToStore(Store<K, S> targetStore, K k, V v, T t) throws UnsupportedViewOperationException;
Change the existing StoreClient<K, V> interface to StoreClient<K, V, T> with following API additions -
- public Versioned get(K key, T transforms);
- public Map<K, Versioned> getAll(Iterable keys, Map<K, T> transforms);
- public void put(K key, V value, T transforms);
Make the following API changes to Store<K, V>
- public List<Versioned> get(K key, T transforms) throws VoldemortException;
- public Map<K, List<Versioned>> getAll(Iterable keys, Map<K, T> transforms) throws VoldemortException;
- public void put(K key, Versioned value, T transforms) throws VoldemortException;
Except the ViewStorageEngine, the transforms are useless for all stores/storage engines and Store Clients.