Add F[_] type parameter to DataSource #171

purrgrammer · 2018-11-21T16:44:38Z

In the old version of Fetch, one needed to define the data source as an
implicit object. This way, we ensured that each DataSource had only one instance using reference equality.

A data source used to look like the following:

implicit object Users extends DataSource[UserId, User]{
  // ...
}

Now that we've introduced an F[_] type parameter in DataSource we no
longer can define it as an object. Thus, we need a way to identify
requests to a data source to be able to perform optimizations.

Now there's a new trait called Data, that identifies requests for a
particular piece of data, and is used to identify and group requests for
the same data.

A Data definition looks like the following:

object Users extends Data[Int, User] {
  def name = "Users"
}

Then, when defining the data source for users, we have to specify which
Data the DataSource is fetching:

def usersSource[F[_] : ConcurrentEffect]: DataSource[F, Int, User] = new DataSource[F, Int,
User] {
  def data = Users

  def CF = ConcurrentEffect[F]

  // ...
}

Now that we have Data and DataSource we can create a Fetch por a User:

def fetchUser[F[_] : ConcurrentEffect](id: Int): Fetch[F, User] =
  Fetch(id, usersSource)

See the examples directory and the tests for plenty of examples.

shared/src/main/scala/datasource.scala

diesalbla · 2018-11-27T17:48:47Z

examples/src/test/scala/DoobieExample.scala

-      q.query[Author].list
-    }
+    def transactor[F[_]: ConcurrentEffect]: F[Transactor[F]] =
+      for {


Suggested change

for {

createTransactor.flatTap(_.trans(dropTable *> createTable *> authors.traverse(addAuthor) )

The flatTap allows you to tee and use the output for a side-effectful computation.

I tried it but I got a type mismatch between doobie.h2.H2Transactor and doobie.Transactor.

diesalbla · 2018-11-27T18:32:12Z

shared/src/main/scala/datasource.scala


  /** Fetch many identities, returning a mapping from identities to results. If an
   * identity wasn't found, it won't appear in the keys.
   */
-  def batch[F[_] : ConcurrentEffect](ids: NonEmptyList[I]): F[Map[I, A]] =
+  def batch(ids: NonEmptyList[I])(implicit C: ConcurrentEffect[F]): F[Map[I, A]] =


So, one problem with using implicit parameters is that you can not override them or remove them from a method in the subclass: the subclasses must keep a similar signature to the parent.

We could leave here no implementation, or add one that only used an Applicative constraint (to use with List.traverse), and then move the implementation using ConcurrentEffect to a different mix-in trait.

trait ConcurrentBatch[F[_]] extends DataSource[F] { implicit protected ConcF: ConcurrentEffect[F] def batch(ids: NonEmptyList[I]): F[Map[I, A]] = /****/ }

This would allow you to decouple the tests from the ConcurrentEffect type class.

The ConcurrentEffect type class is something we can't escape from since it'll be required by Fetch when running or creating fetches. Having the implicit ConcurrentEffect in scope is something that you want when defining data fetches, since very few of those will require only Applicative.

Remember that Fetch is a library for reordering and optimizing IO, in particular reading data. In practice, you will be using the ConcurrentEffect API to express resource allocation, asynchronous effects and lifting IO computations to F so I think it's fine.

codecov · 2019-01-31T13:37:18Z

Codecov Report

Merging #171 into master will increase coverage by 1.34%.
The diff coverage is 100%.

@@            Coverage Diff             @@
##           master     #171      +/-   ##
==========================================
+ Coverage    95.2%   96.55%   +1.34%     
==========================================
  Files           6        6              
  Lines         167      174       +7     
  Branches        7        4       -3     
==========================================
+ Hits          159      168       +9     
+ Misses          8        6       -2

Impacted Files	Coverage Δ
shared/src/main/scala/datasource.scala	`100% <100%> (ø)`	⬆️
shared/src/main/scala/fetch.scala	`96.79% <100%> (+1.4%)`	⬆️
shared/src/main/scala/cache.scala	`100% <100%> (ø)`	⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 06556e7...59a45b8. Read the comment docs.

codecov · 2019-01-31T13:37:18Z

Codecov Report

Merging #171 into master will increase coverage by 1.36%.
The diff coverage is 100%.

@@            Coverage Diff             @@
##           master     #171      +/-   ##
==========================================
+ Coverage    95.2%   96.57%   +1.36%     
==========================================
  Files           6        6              
  Lines         167      175       +8     
  Branches        7        6       -1     
==========================================
+ Hits          159      169      +10     
+ Misses          8        6       -2

Impacted Files	Coverage Δ
shared/src/main/scala/datasource.scala	`100% <100%> (ø)`	⬆️
shared/src/main/scala/fetch.scala	`96.79% <100%> (+1.4%)`	⬆️
shared/src/main/scala/cache.scala	`100% <100%> (ø)`	⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 06556e7...61a28be. Read the comment docs.

In the old version of Fetch, one needed to define the data source as an implicit object. This way, we ensured that each DataSource had only one instance using reference equality. A data source used to look like the following: ```scala implicit object Users extends DataSource[UserId, User]{ // ... } ``` Now that we've introduced an `F[_]` type parameter in `DataSource` we no longer can define it as an `object`. Thus, we need a way to identify requests to a data source to be able to perform optimizations. Now there's a new trait called `Data`, that identifies requests for a particular piece of data, and is used to identify and group requests for the same data. A `Data` definition looks like the following: ```scala object Users extends Data[Int, User] { def name = "Users" } ``` Then, when defining the data source for users, we have to specify which `Data` the `DataSource` is fetching: ```scala def usersSource[F[_]]: DataSource[F, Int, User] = new DataSource[F, Int, User] { def data = Users // ... } ``` Now that we have `Data` and `DataSource` we can create a `Fetch` por a `User`: ```scala def fetchUser(id: Int): Fetch[F, User] = Fetch(id, usersSource) ``` See the `examples` directory and the tests for plenty of examples.

diesalbla · 2019-02-03T18:19:45Z

shared/src/main/scala/fetch.scala

+          val combined = acc.get(dsId).fold(
+            (ds, blocked)
+          )({
+            case (ds, req) => {


The ds variable here shadows the one from line 136. Should they always be the same?

In that case, perhaps we should use back-quotes? As it is right now, the second ds is a completely different variable, that just happens to have the same name.

README.md

JorgeCastilloPrz · 2019-02-05T13:31:57Z

@purrgrammer can you open an issue on Kollect to replicate this in case it makes sense?

Add F[_] type parameter to DataSource

purrgrammer requested review from peterneyens and diesalbla November 21, 2018 16:44

diesalbla reviewed Nov 21, 2018

View reviewed changes

shared/src/main/scala/datasource.scala Outdated Show resolved Hide resolved

diesalbla reviewed Nov 27, 2018

View reviewed changes

purrgrammer force-pushed the ds-identity branch from f11f027 to d0330b6 Compare January 31, 2019 11:49

purrgrammer force-pushed the ds-identity branch from 59a45b8 to 4e0f1c6 Compare January 31, 2019 15:18

purrgrammer requested a review from raulraja January 31, 2019 15:32

Add implicit 'ConcurrentEffect' to 'DataSource' trait

dca5ee6

diesalbla approved these changes Feb 3, 2019

View reviewed changes

juanpedromoreno approved these changes Feb 4, 2019

View reviewed changes

raulraja approved these changes Feb 4, 2019

View reviewed changes

README.md Outdated Show resolved Hide resolved

README.md Outdated Show resolved Hide resolved

README.md Outdated Show resolved Hide resolved

Rename variable

61a28be

purrgrammer merged commit 02c3738 into master Feb 5, 2019

purrgrammer deleted the ds-identity branch February 5, 2019 11:43

purrgrammer mentioned this pull request Feb 5, 2019

F[_] type parameter to DataSource and Data trait for identifying requests 47degrees/kollect#17

Open

bijancn pushed a commit to bijancn/fetch that referenced this pull request Sep 19, 2019

Merge pull request xebia-functional#171 from 47deg/ds-identity

2419638

Add F[_] type parameter to DataSource

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add F[_] type parameter to DataSource #171

Add F[_] type parameter to DataSource #171

purrgrammer commented Nov 21, 2018 •

edited

diesalbla Nov 27, 2018 •

edited

purrgrammer Jan 31, 2019

diesalbla Nov 27, 2018 •

edited

purrgrammer Jan 31, 2019

codecov bot commented Jan 31, 2019

codecov bot commented Jan 31, 2019 •

edited

diesalbla Feb 3, 2019

purrgrammer Feb 4, 2019

diesalbla Feb 4, 2019

JorgeCastilloPrz commented Feb 5, 2019 •

edited

	for {
	createTransactor.flatTap(_.trans(dropTable > createTable > authors.traverse(addAuthor) )

Add F[_] type parameter to DataSource #171

Add F[_] type parameter to DataSource #171

Conversation

purrgrammer commented Nov 21, 2018 • edited

diesalbla Nov 27, 2018 • edited

Choose a reason for hiding this comment

purrgrammer Jan 31, 2019

Choose a reason for hiding this comment

diesalbla Nov 27, 2018 • edited

Choose a reason for hiding this comment

purrgrammer Jan 31, 2019

Choose a reason for hiding this comment

codecov bot commented Jan 31, 2019

Codecov Report

codecov bot commented Jan 31, 2019 • edited

Codecov Report

diesalbla Feb 3, 2019

Choose a reason for hiding this comment

purrgrammer Feb 4, 2019

Choose a reason for hiding this comment

diesalbla Feb 4, 2019

Choose a reason for hiding this comment

JorgeCastilloPrz commented Feb 5, 2019 • edited

purrgrammer commented Nov 21, 2018 •

edited

diesalbla Nov 27, 2018 •

edited

diesalbla Nov 27, 2018 •

edited

codecov bot commented Jan 31, 2019 •

edited

JorgeCastilloPrz commented Feb 5, 2019 •

edited