Redesign #155

purrgrammer · 2018-08-31T12:20:29Z

This is still a WIP but I managed to rewrite the core Fetch execution model using an approach much closer to the original paper. The goal was to modernize the library and use the primitives from cats-effect.

I removed the Query type and I'm instead using ConcurrentEffect in data sources, it looks like:

import cats.temp.par._
import cats.effect._

trait DataSource[I, A]{
  def name: String

  def fetch[F[_] : ConcurrentEffect : Par](id: I): F[Option[A]]

  /* automatically implemented in terms of `fetch` and parTraverse, user can override */
  def batch[F[_] : ConcurrentEffect : Par](ids: NonEmptyList[I]): F[Map[I, A]] 
}

Par is a typeclass that eases using cats.Parallel, taking only one type parameter instead of two. It comes from cats-par.

I've gotten rid of Free in our code, since it didn't allow us to explore both sides of a computation in the way this encoding does. Now a description of a Fetch is encoded in the FetchResult type which can be Done or Blocked waiting for a computation to finish before continuing. Having our own representation of the primitives we use from Free (pure, flatMap) allows us to explore (and combine) both sides of a computation and group their requests.

DataSourceCache is also implemented in terms of ConcurrentEffect now:

trait DataSourceCache {
  def lookup[F[_] : ConcurrentEffect, I, A](i: I, ds: DataSource[I, A]): F[Option[A]]
  def insert[F[_] : ConcurrentEffect, I, A](i: I, v: A, ds: DataSource[I, A]): F[DataSourceCache]
}

which should solve #148 and allow users to use caches that perform IO operations. I still need to write a proof-of-concept Redis cache integration to make sure the API makes sense.

Since I've moved away from Free and a state-based interpreter, the new interpreter runs a Fetch instance into a ConcurrentEffect.

There are some features that I need to add before finishing, namely:

fedefernandez · 2018-09-03T15:39:48Z

shared/src/main/scala/fetch.scala

+      Unfetch(for {
+        a <- fa.run
+        b <- fb.run
+        result = (a, b) match {


Can this be expressed with an apply product? If it's possible I think it makes more sense.

(fa.run, fb.run).tupled.map { case (Done(a), Done(b)) => Done((a, b)) //...

fedefernandez · 2018-09-03T15:46:20Z

shared/src/main/scala/cache.scala

-case class InMemoryCache(state: Map[DataSourceIdentity, Any]) extends DataSourceCache {
-  override def get[A](k: DataSourceIdentity): Option[A] =
-    state.get(k).asInstanceOf[Option[A]]
+case class InMemoryCache(state: Map[(String, Any), Any]) extends DataSourceCache {


Maybe we could implement the InMemoryCache with Ref

case class InMemoryCache(state: Ref[IO, Map[(String, Any), Any]) extends DataSourceCache

But in this case, you would need to find the place to instantiate this cache as a side effect (Ref.of[IO, Map[(String, Any), Any](Map.empty) returns IO

fedefernandez · 2018-09-03T15:48:57Z

shared/src/main/scala/cache.scala

-case class InMemoryCache(state: Map[DataSourceIdentity, Any]) extends DataSourceCache {
-  override def get[A](k: DataSourceIdentity): Option[A] =
-    state.get(k).asInstanceOf[Option[A]]
+case class InMemoryCache(state: Map[(String, Any), Any]) extends DataSourceCache {


(String, Any) represents the DataSource name and the "key", right?

Can we define both with a value classes?

final class DataSourceName(val name: String) extends AnyVal final class DataSourceKey(val key: Any) extends AnyVal

raulraja

Excellent work and great step fwd toward a better runtime for Fetch. Kudos! 👏 👏 . Once we have a final impl merged in master we will redo https://github.com/47deg/kollect in the same way now that Arrow has an Effect library.

raulraja · 2018-09-11T11:35:56Z

README.md

-    })
+
+  override def fetch(id: String): IO[Option[Int]] = {
+    IO.delay(println(s"--> [${Thread.currentThread.getId}] One Length $id")) >>


Nothing wrong with IO.delay but the IO.apply already delegates to delay so you can express these just as:

IO(println(...))

Indeed, I changed to IO#delay to be more explicit. I'll use ConcurrentEffect when I add #156 changes to this branch.

raulraja · 2018-09-11T11:53:11Z

docs/src/main/tut/index.md


 trait DataSource[Identity, Result]{
  def name: String
-  def fetchOne(id: Identity): Query[Option[Result]]
-  def fetchMany(ids: NonEmptyList[Identity]): Query[Map[Identity, Result]]
+  def fetch(id: Identity): IO[Option[Result]]


Even if Fetch internal implementation is on IO I believe there is no need to force users to use concrete data types if we define data similar to:

abstract class DataSource[F[_]: Effect, G[_], Identity, Result](implicit P: Parallel[F, G]) { def name: String def fetch(id: Identity): F[Option[Result]] def batch(ids: NEL[Identity]): F[Option[Result]] }

This would allow users to implement their data sources and consume Fetch with other datatypes in that provide instances for Effect beside IO because Effect#runAsync returns an IO value that you can use internally and back to the context of the user to F via LiftIO

Done in #156

raulraja · 2018-09-11T11:59:39Z

shared/src/test/scala/FetchSyntaxTests.scala


-  implicit override def executionContext = ExecutionContext.Implicits.global
+  implicit def ioToFuture[A](io: IO[A]): Future[A] = io.unsafeToFuture()


We have been bitten before by this kind of implicit conversion and usually prefer explicit calls to unsafetoFuture in tests that expect a future back and it's explicit when things run in tests.

I'll use that instead, good point 👍

purrgrammer · 2018-09-14T14:29:50Z

All checks passed

raulraja · 2018-09-14T20:02:10Z

This is great @purrgrammer, outstanding work. Merge at will!

peterneyens · 2018-09-17T11:55:42Z

Do we plan on bumping the version to 2?

purrgrammer · 2018-09-17T13:06:34Z

I was planning to bump the version to 1.0.0 when we merge these changes, they are pretty substantial and deserve a major version change.

Hide implementation

codecov · 2018-09-18T19:23:03Z

Codecov Report

Merging #155 into master will increase coverage by 12.41%.
The diff coverage is 90.9%.

@@            Coverage Diff             @@
##           master    #155       +/-   ##
==========================================
+ Coverage   78.09%   90.5%   +12.41%     
==========================================
  Files          16       5       -11     
  Lines         315     158      -157     
  Branches        2       5        +3     
==========================================
- Hits          246     143      -103     
+ Misses         69      15       -54

Impacted Files	Coverage Δ
shared/src/main/scala/syntax.scala	`100% <100%> (+33.33%)`	⬆️
shared/src/main/scala/datasource.scala	`100% <100%> (+33.33%)`	⬆️
shared/src/main/scala/env.scala	`66.66% <66.66%> (+16.66%)`	⬆️
shared/src/main/scala/cache.scala	`80% <85.71%> (-4.62%)`	⬇️
shared/src/main/scala/fetch.scala	`91.36% <91.36%> (+3.13%)`	⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 4b85a21...3fe15b3. Read the comment docs.

codecov · 2018-09-18T19:23:55Z

Codecov Report

Merging #155 into master will increase coverage by 12.41%.
The diff coverage is 90.9%.

@@            Coverage Diff             @@
##           master    #155       +/-   ##
==========================================
+ Coverage   78.09%   90.5%   +12.41%     
==========================================
  Files          16       5       -11     
  Lines         315     158      -157     
  Branches        2       5        +3     
==========================================
- Hits          246     143      -103     
+ Misses         69      15       -54

Impacted Files	Coverage Δ
shared/src/main/scala/syntax.scala	`100% <100%> (+33.33%)`	⬆️
shared/src/main/scala/datasource.scala	`100% <100%> (+33.33%)`	⬆️
shared/src/main/scala/env.scala	`66.66% <66.66%> (+16.66%)`	⬆️
shared/src/main/scala/cache.scala	`80% <85.71%> (-4.62%)`	⬇️
shared/src/main/scala/fetch.scala	`91.36% <91.36%> (+3.13%)`	⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 4b85a21...3fe15b3. Read the comment docs.

codecov · 2018-09-18T19:24:01Z

Codecov Report

Merging #155 into master will increase coverage by 12.41%.
The diff coverage is 90.9%.

@@            Coverage Diff             @@
##           master    #155       +/-   ##
==========================================
+ Coverage   78.09%   90.5%   +12.41%     
==========================================
  Files          16       5       -11     
  Lines         315     158      -157     
  Branches        2       5        +3     
==========================================
- Hits          246     143      -103     
+ Misses         69      15       -54

Impacted Files	Coverage Δ
shared/src/main/scala/syntax.scala	`100% <100%> (+33.33%)`	⬆️
shared/src/main/scala/datasource.scala	`100% <100%> (+33.33%)`	⬆️
shared/src/main/scala/env.scala	`66.66% <66.66%> (+16.66%)`	⬆️
shared/src/main/scala/cache.scala	`80% <85.71%> (-4.62%)`	⬇️
shared/src/main/scala/fetch.scala	`91.36% <91.36%> (+3.13%)`	⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 4b85a21...3fe15b3. Read the comment docs.

purrgrammer · 2018-09-18T19:24:38Z

codecov · 2018-09-18T19:24:47Z

Codecov Report

Merging #155 into master will increase coverage by 12.41%.
The diff coverage is 90.9%.

@@            Coverage Diff             @@
##           master    #155       +/-   ##
==========================================
+ Coverage   78.09%   90.5%   +12.41%     
==========================================
  Files          16       5       -11     
  Lines         315     158      -157     
  Branches        2       5        +3     
==========================================
- Hits          246     143      -103     
+ Misses         69      15       -54

Impacted Files	Coverage Δ
shared/src/main/scala/syntax.scala	`100% <100%> (+33.33%)`	⬆️
shared/src/main/scala/datasource.scala	`100% <100%> (+33.33%)`	⬆️
shared/src/main/scala/env.scala	`66.66% <66.66%> (+16.66%)`	⬆️
shared/src/main/scala/cache.scala	`80% <85.71%> (-4.62%)`	⬇️
shared/src/main/scala/fetch.scala	`91.36% <91.36%> (+3.13%)`	⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 4b85a21...3fe15b3. Read the comment docs.

codecov · 2018-09-18T19:25:37Z

Codecov Report

Merging #155 into master will increase coverage by 12.41%.
The diff coverage is 90.9%.

@@            Coverage Diff             @@
##           master    #155       +/-   ##
==========================================
+ Coverage   78.09%   90.5%   +12.41%     
==========================================
  Files          16       5       -11     
  Lines         315     158      -157     
  Branches        2       5        +3     
==========================================
- Hits          246     143      -103     
+ Misses         69      15       -54

Impacted Files	Coverage Δ
shared/src/main/scala/syntax.scala	`100% <100%> (+33.33%)`	⬆️
shared/src/main/scala/datasource.scala	`100% <100%> (+33.33%)`	⬆️
shared/src/main/scala/env.scala	`66.66% <66.66%> (+16.66%)`	⬆️
shared/src/main/scala/cache.scala	`80% <85.71%> (-4.62%)`	⬇️
shared/src/main/scala/fetch.scala	`91.36% <91.36%> (+3.13%)`	⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 4b85a21...3fe15b3. Read the comment docs.

Redesign

purrgrammer added 27 commits August 23, 2018 13:54

👶 steps

3975fc5

🔥

9d2f20f

🔥 env in error

b1f5221

🔥 FetchMonadError, interpret Fetch to IO

e02c9ac

🔥

a21c1c7

🔥 🔥 🔥

4adb52c

High-level API working, now onto the execution model

543f875

👶 run Fetch with an environment

658574e

Use Deferred and IO internally for storing intermediate results

c9bbba6

🔥

3eb78ff

Failing test to implement batching

2098a62

Time requests

1d3fa96

Test execution model

d8ff3b8

🔥

713a95d

🔥 monix subproject

09aa43a

Async queries

5de8412

Caching

28a98e4

Tweaks to caching

c89a8c7

Error handling

4a5f42f

Test batching

82a6d24

wip environment debug code

e5a3b92

Use IO in DataSource

12719b9

🔥 cleanup

1c4a479

wip docs

edffae2

Introduce parallelism

346358f

Progress on documentation

3223dcc

Update cache after fetching a batch

917fcb1

fedefernandez reviewed Sep 3, 2018

View reviewed changes

purrgrammer added 2 commits September 3, 2018 21:54

Run both sides of a fetch with .tupled

62a3f0d

Use value classes for the in-memory cache

04dcfdf

🔥 just a poc, does not compile

c29d8a0

raulraja approved these changes Sep 11, 2018

View reviewed changes

purrgrammer added 6 commits September 12, 2018 12:32

Fetch implementation parameterised to F[_]

bd422e0

Minor cleanup

cb32e20

Update examples

01fe2aa

Update doc examples

0897ed4

Add Par to DataSource#fetch implicits

184e8a9

Add tests to validation commands

3e90ff5

Update README

f7dd3bc

peterneyens added 3 commits September 17, 2018 11:32

Hide implementation

fee19d6

Replace >> with functor syntax

a26faf0

Some refactoring

554ab7a

Simplify flatMap

e907a69

Merge pull request #157 from 47deg/redesign-peter

b09f1ed

Hide implementation

This was referenced Sep 18, 2018

Redesign 47degrees/kollect#13

Closed

Kollect 0.1 - Mirrors Fetch redesign 47degrees/kollect#14

Merged

Bump 1.0.0-RC1 release

3fe15b3

purrgrammer merged commit 6ecea36 into master Sep 18, 2018

purrgrammer deleted the redesign branch September 18, 2018 19:24

bijancn pushed a commit to bijancn/fetch that referenced this pull request Sep 19, 2019

Merge pull request xebia-functional#155 from 47deg/redesign

7b3546d

Redesign

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Redesign #155

Redesign #155

purrgrammer commented Aug 31, 2018 •

edited

fedefernandez Sep 3, 2018

fedefernandez Sep 3, 2018

fedefernandez Sep 3, 2018

raulraja left a comment

raulraja Sep 11, 2018

purrgrammer Sep 11, 2018

raulraja Sep 11, 2018

purrgrammer Sep 11, 2018

raulraja Sep 11, 2018

purrgrammer Sep 11, 2018

purrgrammer commented Sep 14, 2018

raulraja commented Sep 14, 2018

peterneyens commented Sep 17, 2018

purrgrammer commented Sep 17, 2018

codecov bot commented Sep 18, 2018

codecov bot commented Sep 18, 2018

codecov bot commented Sep 18, 2018

purrgrammer commented Sep 18, 2018

codecov bot commented Sep 18, 2018

codecov bot commented Sep 18, 2018


		implicit override def executionContext = ExecutionContext.Implicits.global
		implicit def ioToFuture[A](io: IO[A]): Future[A] = io.unsafeToFuture()

Redesign #155

Redesign #155

Conversation

purrgrammer commented Aug 31, 2018 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

raulraja left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

purrgrammer commented Sep 14, 2018

raulraja commented Sep 14, 2018

peterneyens commented Sep 17, 2018

purrgrammer commented Sep 17, 2018

codecov bot commented Sep 18, 2018

Codecov Report

codecov bot commented Sep 18, 2018

Codecov Report

codecov bot commented Sep 18, 2018

Codecov Report

purrgrammer commented Sep 18, 2018

codecov bot commented Sep 18, 2018

Codecov Report

codecov bot commented Sep 18, 2018

Codecov Report

purrgrammer commented Aug 31, 2018 •

edited