New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Split ecto into ecto and ecto_sql #2558

Closed
josevalim opened this Issue May 21, 2018 · 8 comments

Comments

Projects
None yet
4 participants
@josevalim
Member

josevalim commented May 21, 2018

Ecto 2.0 pushed many improvements to decouple the schema/changeset <-> repo relationship, allowing Ecto work generally as a data validation library. This has pushed some users to ask us to split Ecto apart, allowing developers to bring a small part of Ecto when they didn't need a database.

Most of the discussion around splitting Ecto apart was always about extracting schemas and changesets out of Ecto. However, this introduced an issue. The changeset module also has conveniences for working with constraints (repository) and associations. Breaking it apart meant that we would need to break the changeset API into multiple modules, making discovery much harder (and learning Ecto already has its challenges, as you need to learn about schemas, changesets, queries and repos).

After a while, we realized we were simply trying to slice Ecto in the wrong way. Repo, schemas, changesets and queries are an essential part of Ecto, they belong together. So the goal is to move everything else elsewhere.

Our proposal is to introduce two projects in Ecto v3.0: ecto_sql and ecto. ecto_sql will depend on ecto and our goal is to eventually host all SQL implementations in ecto_sql (more about it later).

Users don't need to worry. Updating your project from Ecto 2.0 to Ecto 3.0 should be as easy as:

-{:ecto, "~> 2.0"}
+{:ecto_sql, "~> 3.0"}

What goes where

The following will be moved to ecto_sql:

  1. Everything in lib/ecto/adapters plus support code
  2. Ecto Migration behaviour, module API and mix tasks
  3. Ecto Structure behaviour, module API and mix tasks

Everything else will remain in ecto. The name of the moved modules will be kept as is. The only pending task to do this is to figure out a public API for Mix.Ecto, but we can do this as we migrate.

The future

With the addition of ecto_sql, our plan is to actually support the major SQL TCP-based adapters directly in the ecto_sql application. The reason for this is the following: postgres and mysql are in a privileged situation because everytime there is a new feature in Ecto, they are immediately changed to leverage this new feature. The other drivers/adapters go through a longer development cycle which is detrimental to users, maintainers and the ecto ecosystem.

Note however we don't plan to do those changes now but rather incorporate them in future ecto/ecto_sql releases.

Summary

The goal of this proposal is to reduce Ecto's default footprint and provide a core library that focuses exclusively on Ecto four pillars: schemas, changesets, queries and repositories. All of the SQL specific behaviour will be moved to a separate library.

@josevalim

This comment has been minimized.

Show comment
Hide comment
@josevalim

josevalim May 29, 2018

Member

I believe most of the prep work for this to happen has been done. Adapters no longer depend on private Mix APIs, so it is a matter of moving the code out. 👍 I will start with it soon. I just plan to clean up the issues tracker a bit more to avoid having to sync two repos constantly.

Member

josevalim commented May 29, 2018

I believe most of the prep work for this to happen has been done. Adapters no longer depend on private Mix APIs, so it is a matter of moving the code out. 👍 I will start with it soon. I just plan to clean up the issues tracker a bit more to avoid having to sync two repos constantly.

@josevalim josevalim modified the milestone: v3.0 Jun 2, 2018

@ringods

This comment has been minimized.

Show comment
Hide comment
@ringods

ringods Aug 16, 2018

@josevalim does this split also brings ecto closer to support other types of datastores, like noSQL or graph databases?

ringods commented Aug 16, 2018

@josevalim does this split also brings ecto closer to support other types of datastores, like noSQL or graph databases?

@josevalim

This comment has been minimized.

Show comment
Hide comment
@josevalim

josevalim Aug 16, 2018

Member

@ringods yes and no. The query aspect in Ecto is still very tied to SQL and I don't see that changing. But there is nothing stopping you from implementing Ecto.Repo for those data stores and leveraging them only for the insert/update/delete bits and bringing in your own query language.

This split definitely reduces the perceived burden to introducing an adapter though.

Member

josevalim commented Aug 16, 2018

@ringods yes and no. The query aspect in Ecto is still very tied to SQL and I don't see that changing. But there is nothing stopping you from implementing Ecto.Repo for those data stores and leveraging them only for the insert/update/delete bits and bringing in your own query language.

This split definitely reduces the perceived burden to introducing an adapter though.

@solnic

This comment has been minimized.

Show comment
Hide comment
@solnic

solnic Aug 31, 2018

The query aspect in Ecto is still very tied to SQL and I don't see that changing

As in, not in the near future, or never-ever?

solnic commented Aug 31, 2018

The query aspect in Ecto is still very tied to SQL and I don't see that changing

As in, not in the near future, or never-ever?

@josevalim

This comment has been minimized.

Show comment
Hide comment
@josevalim

josevalim Aug 31, 2018

Member

Never say never. :)

Although I should clarify. Today, you can totally use parts of Ecto.Query to map it to simpler domains. What I meant to say is that we likely won't ever support something graph queries because I believe the only way to do it properly is by introducing a new query language appropriate for graph DBs. So I don't think the issue is that Ecto.Query is coupled to SQL, since making it more generic would likely make it a poor fit for both SQL and NoSQL, but rather that to support other databases properly we would need to introduce other query languages.

I will be totally glad to be proven wrong though. :)

Member

josevalim commented Aug 31, 2018

Never say never. :)

Although I should clarify. Today, you can totally use parts of Ecto.Query to map it to simpler domains. What I meant to say is that we likely won't ever support something graph queries because I believe the only way to do it properly is by introducing a new query language appropriate for graph DBs. So I don't think the issue is that Ecto.Query is coupled to SQL, since making it more generic would likely make it a poor fit for both SQL and NoSQL, but rather that to support other databases properly we would need to introduce other query languages.

I will be totally glad to be proven wrong though. :)

@solnic

This comment has been minimized.

Show comment
Hide comment
@solnic

solnic Sep 1, 2018

I agree that a generic query dsl is not a good idea. So now I wonder why it’s Ecto.Query instead of Ecto.SQL.Query, this would make it more clear that this is a dsl for SQL adapter.

solnic commented Sep 1, 2018

I agree that a generic query dsl is not a good idea. So now I wonder why it’s Ecto.Query instead of Ecto.SQL.Query, this would make it more clear that this is a dsl for SQL adapter.

@josevalim

This comment has been minimized.

Show comment
Hide comment
@josevalim

josevalim Sep 4, 2018

Member

I have broken the Ecto.Adapter specification into smaller chunks in master, so it is now possible to have an adapter that cares only about the schema functions and does not interface with Ecto.Query at all (and vice-versa).

The docs regarding this split have also been updated in master. Now it is mostly a matter of moving the code to a new repository, which we will do before the release candidate that should still happen this september.

Member

josevalim commented Sep 4, 2018

I have broken the Ecto.Adapter specification into smaller chunks in master, so it is now possible to have an adapter that cares only about the schema functions and does not interface with Ecto.Query at all (and vice-versa).

The docs regarding this split have also been updated in master. Now it is mostly a matter of moving the code to a new repository, which we will do before the release candidate that should still happen this september.

@josevalim

This comment has been minimized.

Show comment
Hide comment
@josevalim

josevalim Oct 3, 2018

Member

In master. Ecto.SQL will be pushed to github soon!

Member

josevalim commented Oct 3, 2018

In master. Ecto.SQL will be pushed to github soon!

@josevalim josevalim closed this Oct 3, 2018

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment