embeds_one won't dump; says it's an invalid type

[jdewar]

I am trying sqlite3 out for the first time in Livebook. I believe that I have parity with how an app would use it, but I cannot get an embeds_one field to be inserted. I've appended an example of the failure on a 'buildings' table that has an embeds_one 'size' field. The error occurs in dump_field!:

(Ecto.ChangeError) value `%Satis.Model.BuildingSize{height: 0, length: 0, width: 0}` for `Satis.Model.Building.size` 
in `insert` does not match type #Ecto.Embedded<%Ecto.Embedded{cardinality: :one, field: :size, owner: 
Satis.Model.Building, related: Satis.Model.BuildingSize, on_cast: nil, on_replace: :raise, unique: true, ordered: true}>

I've used embeds before with postgres, and it feels like i'm missing something obvious, but I don't know what.
Below is the Livebook file (.livemd file not attachable).

---

embeds in sqlite3 won't dump

Mix.install([:kino, :ecto_sqlite3, {:ecto, "~> 3.12"}])

Section

sqlite3_db_file = "Documents/codex.sqlite3"

Application.put_env(:satis, Satis.Repo,
  # Cannot use :memory sqlite3 bc you can't migrate it
  database: sqlite3_db_file,
  # Turn off foreign key constraints to ease data importing
  foreign_keys: :off,
  log: false
)

defmodule Satis.Repo do
  @otp_app :satis
  use Ecto.Repo, otp_app: @otp_app, adapter: Ecto.Adapters.SQLite3
end

defmodule Satis.Model do
  defmacro __using__(_) do
    quote do
      use Ecto.Schema
      import Ecto.Changeset
      alias unquote(__MODULE__)
      @before_compile unquote(__MODULE__)
    end
  end

  defmacro __before_compile__(_) do
    quote do
      def changeset(params \\ %{}) do
        changeset(%__MODULE__{}, params)
      end

      def changeset(model, params) do
        fields = __MODULE__.__schema__(:fields)
        associations = __MODULE__.__schema__(:associations)
        embeds = __MODULE__.__schema__(:embeds)

        cs = cast(model, params, fields -- (embeds ++ associations))
        cs = Enum.reduce(embeds, cs, &cast_embed(&2, &1))
        Enum.reduce(associations, cs, &cast_assoc(&2, &1))
      end
    end
  end
end

defmodule Satis.Model.BuildingSize do
  use Satis.Model
  @primary_key false
  embedded_schema do
    field(:height, :integer)
    field(:length, :integer)
    field(:width, :integer)
  end
end

defmodule Satis.Model.Building do
  use Satis.Model
  @primary_key {:class_name, :string, autogenerate: false}
  schema "buildings" do
    embeds_one(:size, Model.BuildingSize)
  end

  defmodule Migration do
    use Ecto.Migration

    def change do
      create table("buildings", primary_key: false) do
        add(:class_name, :string, primary_key: true)
        add(:size, :map)
      end
    end
  end
end

{:module, Satis.Model.Building, <<70, 79, 82, 49, 0, 0, 61, ...>>,
 {:module, Satis.Model.Building.Migration, <<70, 79, 82, ...>>, {:change, 0}}}

Ready Database

alias Satis.Model
File.rm(sqlite3_db_file) |> dbg

Kino.start_child!(Satis.Repo) |> dbg

[
  Model.Building
]
|> Enum.with_index(1)
|> Enum.each(fn {migration, idx} ->
  :ok = Ecto.Migrator.up(Satis.Repo, idx, Module.concat([migration, "Migration"]))
end)

:ok

#PID<0.265.0>

09:42:48.019 [info] == Running 1 Satis.Model.Building.Migration.change/0 forward

09:42:48.021 [info] create table buildings

09:42:48.022 [info] == Migrated 1 in 0.0s

:ok

data =
  %{
    "class_name" => "Desc_QuarterPipeMiddleOutCorner_Ficsit_4x1_C",
    "size" => %{"height" => 0, "length" => 0, "width" => 0},
  }

Model.Building.changeset(data)
|> Satis.Repo.insert!()

** (Ecto.ChangeError) value `%Satis.Model.BuildingSize{height: 0, length: 0, width: 0}` for `Satis.Model.Building.size` in `insert` does not match type #Ecto.Embedded<%Ecto.Embedded{cardinality: :one, field: :size, owner: Satis.Model.Building, related: Satis.Model.BuildingSize, on_cast: nil, on_replace: :raise, unique: true, ordered: true}>
    (ecto 3.12.5) lib/ecto/repo/schema.ex:1115: Ecto.Repo.Schema.dump_field!/6
    (ecto 3.12.5) lib/ecto/repo/schema.ex:1124: anonymous fn/6 in Ecto.Repo.Schema.dump_fields!/5
    (stdlib 6.2) maps.erl:860: :maps.fold_1/4
    (ecto 3.12.5) lib/ecto/repo/schema.ex:1122: Ecto.Repo.Schema.dump_fields!/5
    (ecto 3.12.5) lib/ecto/repo/schema.ex:1055: Ecto.Repo.Schema.dump_changes!/7
    (ecto 3.12.5) lib/ecto/repo/schema.ex:406: anonymous fn/15 in Ecto.Repo.Schema.do_insert/4
    (ecto 3.12.5) lib/ecto/repo/schema.ex:1099: anonymous fn/3 in Ecto.Repo.Schema.wrap_in_transaction/6
    #cell:zrd25i7t5zceldyc:8: (file)

[warmwaffles]

What's the changeset function defined as?

[jdewar]

The given example is clipped from about 6 more tables / migrations, so to save space in the livebook they are all being auto generated in the __before_compile__ hook of Satis.Model. I tried manually defining the changsets as below:

Model.BuildingSize

  def changeset(params) do
    %__MODULE__{} |> changeset(params)
  end

  def changeset(data, params) do
    data
    |> cast(params, [:height, :length, :width])
  end

and Model.Building

  def changeset(params) do
    %__MODULE__{} |> changeset(params)
  end

  def changeset(data, params) do
    data
    |> cast(params, [:class_name])
    |> cast_embed(:size)
  end

and received the same error

[warmwaffles]

I know this is a stretch, but have you tried this against the postgres adapter?

Edit: I am asking because I think you may be running into an issue with Ecto itself, and not this adapter.

[jdewar]

I had been avoiding it (was trying out sqlite3), but thankfully it wasn't too hard to change adapters, and it worked with postgres (i've attached below for completeness), but that also showed me the actual error!

Punchline: I was missing Jason. Here was Postgres' error without :jason being installed:

** (UndefinedFunctionError) function Jason.encode_to_iodata!/1 is undefined (module Jason is not available). Make sure the module name is correct and has been specified in full (or that an alias has been defined)
    Jason.encode_to_iodata!(%{length: 0, width: 0, height: 0})
    (postgrex 0.20.0) lib/postgrex/type_module.ex:1084: Postgrex.DefaultTypes.encode_params/3
    (postgrex 0.20.0) lib/postgrex/query.ex:75: DBConnection.Query.Postgrex.Query.encode/3
    (db_connection 2.7.0) lib/db_connection.ex:1449: DBConnection.encode/5
    (db_connection 2.7.0) lib/db_connection.ex:1549: DBConnection.run_prepare_execute/5
    (db_connection 2.7.0) lib/db_connection.ex:772: DBConnection.parsed_prepare_execute/5
    (db_connection 2.7.0) lib/db_connection.ex:764: DBConnection.prepare_execute/4
    #cell:icoyuvuwifcyj22c:8: (file)

Once I added :jason to the livebook in the original post, it worked like expected. Perhaps a more clear error could be surfaced, but this issue seems closed as is, thanks for the assistance!

---

embeds in sqlite3 won't dump

Mix.install([:kino, :jason, :ecto_sql, :postgrex, {:ecto, "~> 3.12"}])

Section

sqlite3_db_file = ""

Application.put_env(:satis, Satis.Repo,
  hostname: "localhost",
  username: "postgres",
  password: "postgres",
  database: "codex-test",
  log: false
)

defmodule Satis.Repo do
  @otp_app :satis
  use Ecto.Repo, otp_app: @otp_app, adapter: Ecto.Adapters.Postgres
end

defmodule Satis.Model do
  defmacro __using__(_) do
    quote do
      use Ecto.Schema
      import Ecto.Changeset
      alias unquote(__MODULE__)
    end
  end
end

defmodule Satis.Model.BuildingSize do
  use Satis.Model
  @primary_key false
  embedded_schema do
    field(:height, :integer)
    field(:length, :integer)
    field(:width, :integer)
  end

  def changeset(params) do
    %__MODULE__{} |> changeset(params)
  end

  def changeset(data, params) do
    data
    |> cast(params, [:height, :length, :width])
  end
end

defmodule Satis.Model.Building do
  use Satis.Model
  @primary_key {:class_name, :string, autogenerate: false}
  schema "buildings" do
    embeds_one(:size, Model.BuildingSize)
  end

  def changeset(params) do
    %__MODULE__{} |> changeset(params)
  end

  def changeset(data, params) do
    data
    |> cast(params, [:class_name])
    |> cast_embed(:size)
  end

  defmodule Migration do
    use Ecto.Migration

    def change do
      create table("buildings", primary_key: false) do
        add(:class_name, :string, primary_key: true)
        add(:size, :map)
      end
    end
  end
end

{:module, Satis.Model.Building, <<70, 79, 82, 49, 0, 0, 57, ...>>,
 {:module, Satis.Model.Building.Migration, <<70, 79, 82, ...>>, {:change, 0}}}

Ready Database

alias Satis.Model

repo = Satis.Repo
repo.__adapter__().storage_up(repo.config()) |> dbg

Kino.start_child!(Satis.Repo) |> dbg

[
  Model.Building
]
|> Enum.with_index(1)
|> Enum.each(fn {migration, idx} ->
   Ecto.Migrator.up(Satis.Repo, idx, Module.concat([migration, "Migration"])) |> dbg
end)

{:error, :already_up}

#PID<0.1827.0>

:already_up

:ok

data =
  %{
    "class_name" => "Desc_QuarterPipeMiddleOutCorner_Ficsit_4x1_C",
    "size" => %{"height" => 0, "length" => 0, "width" => 0},
  }

Model.Building.changeset(data)
|> Satis.Repo.insert!()

%Satis.Model.Building{
  __meta__: #Ecto.Schema.Metadata<:loaded, "buildings">,
  class_name: "Desc_QuarterPipeMiddleOutCorner_Ficsit_4x1_C",
  size: %Satis.Model.BuildingSize{height: 0, length: 0, width: 0}
}

[jdewar]

As an aside, I tried adding the below to Mix.install:

config: [ecto_sqlite3: [json_library: :json]]

to see if i could get around Jason (no strong reason, just trying new things in livebook), but i got the same error yet again. I could not get the native :json library to be ecto_sqlite's json library after a few other random attempts. I was not installing ':json' because i think that's a hex.pm package that's not the new erlang module. I'm on Livebook 15.2 (Elixir 1.18.2, and erlang 27.something, which should have :json)

Just gonna fall back to Jason. This is always the push-pull of hobby projects, you wanna try new things, but how much time do you wanna spend debugging new problems.

[warmwaffles]

You gotta set the config to the module to use

ecto_sqlite3/lib/ecto/adapters/sqlite3/codec.ex

Line 95 in a04c249

{:ok, Application.get_env(:ecto_sqlite3, :json_library, Jason).encode!(value)}

So in your case

config: [ecto_sqlite3: [json_library: JSON]]

[jdewar]

Worked! Oh, I had no idea the JSON was in the Elixir space; I'd used :json.decode elsewhere in the Livebook, so I assumed it was the usual 'when it's available to both erlang and elixir, you use the lowercase name, eg. :telemetry'. I see it here https://hexdocs.pm/elixir/1.18.2/JSON.html now. Thanks much!

[warmwaffles]

The JSON module is backed by :json, just has some elixir'y things added.