Split ecto into ecto and ecto_sql

[josevalim]

Ecto 2.0 pushed many improvements to decouple the schema/changeset <-> repo relationship, allowing Ecto work generally as a data validation library. This has pushed some users to ask us to split Ecto apart, allowing developers to bring a small part of Ecto when they didn't need a database.

Most of the discussion around splitting Ecto apart was always about extracting schemas and changesets out of Ecto. However, this introduced an issue. The changeset module also has conveniences for working with constraints (repository) and associations. Breaking it apart meant that we would need to break the changeset API into multiple modules, making discovery much harder (and learning Ecto already has its challenges, as you need to learn about schemas, changesets, queries and repos).

After a while, we realized we were simply trying to slice Ecto in the wrong way. Repo, schemas, changesets and queries are an essential part of Ecto, they belong together. So the goal is to move everything else elsewhere.

Our proposal is to introduce two projects in Ecto v3.0: ecto_sql and ecto. ecto_sql will depend on ecto and our goal is to eventually host all SQL implementations in ecto_sql (more about it later).

Users don't need to worry. Updating your project from Ecto 2.0 to Ecto 3.0 should be as easy as:

-{:ecto, "~> 2.0"}
+{:ecto_sql, "~> 3.0"}

What goes where

The following will be moved to ecto_sql:

1. Everything in lib/ecto/adapters plus support code
2. Ecto Migration behaviour, module API and mix tasks
3. Ecto Structure behaviour, module API and mix tasks

Everything else will remain in ecto. The name of the moved modules will be kept as is. The only pending task to do this is to figure out a public API for Mix.Ecto, but we can do this as we migrate.

The future

With the addition of ecto_sql, our plan is to actually support the major SQL TCP-based adapters directly in the ecto_sql application. The reason for this is the following: postgres and mysql are in a privileged situation because everytime there is a new feature in Ecto, they are immediately changed to leverage this new feature. The other drivers/adapters go through a longer development cycle which is detrimental to users, maintainers and the ecto ecosystem.

Note however we don't plan to do those changes now but rather incorporate them in future ecto/ecto_sql releases.

Summary

The goal of this proposal is to reduce Ecto's default footprint and provide a core library that focuses exclusively on Ecto four pillars: schemas, changesets, queries and repositories. All of the SQL specific behaviour will be moved to a separate library.

[josevalim]

I believe most of the prep work for this to happen has been done. Adapters no longer depend on private Mix APIs, so it is a matter of moving the code out. 👍 I will start with it soon. I just plan to clean up the issues tracker a bit more to avoid having to sync two repos constantly.

[ringods]

@josevalim does this split also brings ecto closer to support other types of datastores, like noSQL or graph databases?

[josevalim]

@ringods yes and no. The query aspect in Ecto is still very tied to SQL and I don't see that changing. But there is nothing stopping you from implementing Ecto.Repo for those data stores and leveraging them only for the insert/update/delete bits and bringing in your own query language.

This split definitely reduces the perceived burden to introducing an adapter though.

[solnic]

The query aspect in Ecto is still very tied to SQL and I don't see that changing

As in, not in the near future, or never-ever?

[josevalim]

Never say never. :)

Although I should clarify. Today, you can totally use parts of Ecto.Query to map it to simpler domains. What I meant to say is that we likely won't ever support something graph queries because I believe the only way to do it properly is by introducing a new query language appropriate for graph DBs. So I don't think the issue is that Ecto.Query is coupled to SQL, since making it more generic would likely make it a poor fit for both SQL and NoSQL, but rather that to support other databases properly we would need to introduce other query languages.

I will be totally glad to be proven wrong though. :)

[solnic]

I agree that a generic query dsl is not a good idea. So now I wonder why it’s Ecto.Query instead of Ecto.SQL.Query, this would make it more clear that this is a dsl for SQL adapter.

[josevalim]

I have broken the Ecto.Adapter specification into smaller chunks in master, so it is now possible to have an adapter that cares only about the schema functions and does not interface with Ecto.Query at all (and vice-versa).

The docs regarding this split have also been updated in master. Now it is mostly a matter of moving the code to a new repository, which we will do before the release candidate that should still happen this september.

[josevalim]

In master. Ecto.SQL will be pushed to github soon!

[feymartynov]

I agree with @solnic. Shouldn't we move Ecto.Query to ecto_sql as Ecto.SQL.Query before 3.0 is released?

[josevalim]

I think there is some confusion on what extracting Ecto.SQL means. It does not mean Ecto will now focus on graph or document databases. Because if your goal is to support graph databases, then changing Ecto.Query is likely not enough, because you most likely want to change how you define schemas, handle associations or even changesets.

So at its core, Ecto is still relational, which is where the Ecto.Query is rooted and it has other uses besides SQL. LINQ is a good example of applying the relational constructs to other areas, although maybe too much. At the end of the day, the goal of having ecto_sql is to mostly move concretions out of Ecto for the users who don't care about those concretions.

And while this change does make it easier for you to run a graph database with Ecto, a tool that was designed from day one with graph databases in mind, from schema to queries, will likely do a better job than Ecto ever will (and that's alright because Ecto shouldn't do everything). Case in point: we have refused to add abstractions that made Ecto better for document databases in the past because they were a poor fit for relational ones.

I know people have a lot of strong opinions on this topic but with Ecto 3.0 RC out and the fact that we announced stable API, Ecto is what it is. There are likely better solutions out there and everyone is welcome to explore them.