FS-37 Cache Algebra And Implementations #70

diesalbla · 2017-02-21T17:06:45Z

This PR fulfills ticket #37. We introduce two new sbt modules, freestyle-cache and freestyle-cache-redis, that define an algebra of cache operations, and two interpreters for it. In this ticket, we understand by cache an efficient, high-load, and potentially parallel, key-value store.

Algebra

In the sbt project freestyle-cache, in the package freestyle.cache, we define an algebra named @free trait CacheM[F[_]], that describes a type-class of carrier types F[_] that support cache operations. This algebra gives the usual operations for Key-Value store: get, put, delete, hasKey, keys (for key-set), clear.
Because we want the CacheM trait to be generic on the type of indexing keys and on the type of stored values, we wrap the @free algebra inside an empty KeyValueProvider[K,V] class. Besides the algebra we provide an implicit method to build an interpreter for the algebra (with the same K and V types). This interpreter is built out of two pieces.

One is a raw interpreter to an intermediary carrier type F[_], as an object that inherits from trait KeyValueMap[F,K,V]. The reason for this is that in some implementations the raw format F is fixed.
A natural transformation from the raw format F to the desired one G.

HashMap

In the same project, in the package freestyle.cache.hashmap, we provide an implementation for an interpreter of the algebra, in which we use as a store a ConcurrentHashMap from the java.util library.

Hasher: the ConcurrentHashMap is based on the use of the methods hashcode and equals, which unfortunately is not always properly implemented for all types. We introduce a type-class Hashable, to indicate that the Key type must provide a hashing method.

Redis

In the project freestyle-cache-redis, in the package freestyle.cache.redis, we implement an interpreter for the algebra based on an external Redis data-store. For the implementation, we use the Redis client library rediscala, which is based in akka.

In this interpreter, the notion of parallelization is based on the use of Redis transactions, which for our purposes allows us to group a sequence of data-independent operations and get them sent and executed atomically by the redis server. This parallelization is implemented using the Applicative instance of Kleisli, where the Kleisli is used to defer the communications to the client.

@raulraja ¿Could you have it reviewed?

We add the simple algebra of Cache Operations, to represent key-value stores. We provide a simple implementation, based on a java Concurrent Hash Map.

We add an implementation of the cache algebra, where the cache is backed in a Redis data-store. For the connection with the data store, we need a client library for Scala. We use Scredis.

We replace the previous ad-hoc type ScredisOps with cats.data.Kleisli, so as to reuse the functions already defined for this type.

We add a simple test example, to show the combination of applicative ( (parallel) fragments and the sequential ones. We also rename the package for the scredis subfolder.x

We replace Scredis by Rediscala as our Scala library client for Redis. Scredis does not seem to be actively maintained.

We refactor the `freestyle-cache` module. We remove from the `freestyle.cache` package any implementation details based on a Java Concurrent HashMap, and we move those to the `hashmap` subpackage. In the `freestyle.cache.implicits`, we leave a simple constructor for interpreters, based on two steps: a raw implementation, and a natural transformation.

After the previous refactoring, the Cache Module already separates between generating a _raw_ format (in the Redis it's Future) and the Natural Transformation to another carrier `F[_]`.

codecov-io · 2017-02-21T17:56:44Z

Codecov Report

Merging #70 into master will increase coverage by 4.51%.
The diff coverage is 70.68%.

@@            Coverage Diff             @@
##           master      #70      +/-   ##
==========================================
+ Coverage   57.23%   61.75%   +4.51%     
==========================================
  Files          20       27       +7     
  Lines         159      217      +58     
  Branches        0        1       +1     
==========================================
+ Hits           91      134      +43     
- Misses         68       83      +15

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 706a6af...5582973. Read the comment docs.

raulraja

@diesalbla Excellent work! This is very exciting! Added a few ideas but we can add a new ticket for those improvements 👍 🎉

raulraja · 2017-02-21T20:26:33Z

freestyle-cache/shared/src/main/scala/cache.scala

+      def has(key: Key): FreeS.Par[F, Boolean]
+
+      // Returns the set of keys in the store
+      def keys: FreeS.Par[F, Seq[Key]]


Since the default Seq imported in Predef is mutable and pattern matching is easier over List can this return something else that is not Seq?

raulraja · 2017-02-21T20:31:32Z

freestyle-cache/shared/src/main/scala/hashmap/Hasher.scala

+  def hashCode(a: A): Int
+}
+
+object Hasher {


We can include here automatic instances for any product type such as case classes, HList, TupleN etc with shapeless generic derivation given there is also an implicit Hasher for the contained elements.

raulraja · 2017-02-21T20:33:22Z

freestyle-cache/shared/src/test/scala/CacheTests.scala

+import freestyle.implicits._
+import org.scalatest._
+
+class CacheTests extends WordSpec with Matchers with CacheTestContext {


raulraja · 2017-02-21T20:43:57Z

freestyle-cache/shared/src/main/scala/cache.scala

+      def get(key: Key): FreeS.Par[F, Option[Val]]
+
+      // Sets the value of a key to a newValue.
+      def put(key: Key, newVal: Val): FreeS.Par[F, Unit]


I would add the following methods if we can support them both on the InMemory and Redis impl.

def putAll(keyValues: Map[Key, Val]): FreeS.Par[F, Unit] def putIfAbsent(key: Key, newVal: Val): FreeS.Par[F, Unit] def replace(key: Key, newVal: Val): FreeS.Par[F, Unit] def isEmpty: FreeS.Par[F, Boolean] def containsKey(key: Key): FreeS.Par[F, Boolean] def containsValue(v: Val): FreeS.Par[F, Boolean]

The methods putAll and putIfAbsent are straight-forward. AAMOF, the latter is directly supported in redis. The command isEmpty can be supported efficiently through the use of the scan redis command. The containsValue is the one that could be less efficient to implement.

In any case, this can be left to a low-hanging-fruit ticket.

Sounds good, since you seem to have more insight, would you mind creating one and tagging it as such? thanks! @anamariamv Is getting started with Freestyle and she could take on that task as well.

raulraja · 2017-02-21T20:45:11Z

freestyle/shared/src/main/scala/freestyle/Capture.scala

@@ -5,6 +5,13 @@ import simulacrum.typeclass

 import scala.util.Try

+/*


The implementation of the `keys` operation in the Hashmap based was using a `2.12` API operation, unavailable in `2.11` for crossbuild.

raulraja · 2017-02-22T10:57:14Z

freestyle-cache-redis/shared/src/main/scala/rediscala/RedisMap.scala

-  override def keys: Ops[M, Seq[Key]] =
-    RediscalaCont.keys.transform(toM).map(_.map(parser).flatten)
+  override def keys: Ops[M, List[Key]] =
+    RediscalaCont.keys.transform(toM).map(_.map(parser).flatten.toList)


Can this be simplified to flatMap?

diesalbla added 9 commits February 20, 2017 11:40

FS-37 Cache Algebra

e0d8194

We add the simple algebra of Cache Operations, to represent key-value stores. We provide a simple implementation, based on a java Concurrent Hash Map.

FS-37 Add Scredis Cache Implementation

c648e5a

We add an implementation of the cache algebra, where the cache is backed in a Redis data-store. For the connection with the data store, we need a client library for Scala. We use Scredis.

FS-37 Kleisli

a054c01

We replace the previous ad-hoc type ScredisOps with cats.data.Kleisli, so as to reuse the functions already defined for this type.

FS-37 Add Example

d35e45b

We add a simple test example, to show the combination of applicative ( (parallel) fragments and the sequential ones. We also rename the package for the scredis subfolder.x

FS-37 Replace Scredis by Rediscala

f7878ed

We replace Scredis by Rediscala as our Scala library client for Redis. Scredis does not seem to be actively maintained.

FS-37 Cache-Refactoring in Redis Module

c198357

After the previous refactoring, the Cache Module already separates between generating a _raw_ format (in the Redis it's Future) and the Natural Transformation to another carrier `F[_]`.

FS-37 Add Keys Operation

3dfaf68

Cache HashMap Tests

ec55dec

diesalbla requested a review from raulraja February 21, 2017 17:06

diesalbla self-assigned this Feb 21, 2017

raulraja approved these changes Feb 21, 2017

View reviewed changes

FS-37 Resolve Error

289d500

The implementation of the `keys` operation in the Hashmap based was using a `2.12` API operation, unavailable in `2.11` for crossbuild.

diesalbla force-pushed the diesalbla-37-Cache_Algebras branch from 22b6ed3 to 289d500 Compare February 22, 2017 09:58

FS-37 Address some Code-Review Comments

ccfbae5

raulraja reviewed Feb 22, 2017

View reviewed changes

FS-37 Extra comment

c541799

diesalbla mentioned this pull request Feb 22, 2017

Extend Cache Operations #76

Closed

Merge branch 'master' into diesalbla-37-Cache_Algebras

5582973

raulraja merged commit 4a55ae3 into master Feb 23, 2017

raulraja deleted the diesalbla-37-Cache_Algebras branch February 23, 2017 13:34

diesalbla removed their assignment Jun 18, 2018

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

FS-37 Cache Algebra And Implementations #70

FS-37 Cache Algebra And Implementations #70

diesalbla commented Feb 21, 2017

codecov-io commented Feb 21, 2017 •

edited

raulraja left a comment

raulraja Feb 21, 2017

raulraja Feb 21, 2017

raulraja Feb 21, 2017

raulraja Feb 21, 2017

diesalbla Feb 22, 2017 •

edited

raulraja Feb 22, 2017

raulraja Feb 21, 2017

raulraja Feb 22, 2017

		@@ -5,6 +5,13 @@ import simulacrum.typeclass

		import scala.util.Try

		/*

FS-37 Cache Algebra And Implementations #70

FS-37 Cache Algebra And Implementations #70

Conversation

diesalbla commented Feb 21, 2017

Algebra

HashMap

Redis

codecov-io commented Feb 21, 2017 • edited

Codecov Report

raulraja left a comment

Choose a reason for hiding this comment

raulraja Feb 21, 2017

Choose a reason for hiding this comment

raulraja Feb 21, 2017

Choose a reason for hiding this comment

raulraja Feb 21, 2017

Choose a reason for hiding this comment

raulraja Feb 21, 2017

Choose a reason for hiding this comment

diesalbla Feb 22, 2017 • edited

Choose a reason for hiding this comment

raulraja Feb 22, 2017

Choose a reason for hiding this comment

raulraja Feb 21, 2017

Choose a reason for hiding this comment

raulraja Feb 22, 2017

Choose a reason for hiding this comment

codecov-io commented Feb 21, 2017 •

edited

diesalbla Feb 22, 2017 •

edited