Problem with encoding Ecto model after upgrading to 0.14

[chinh7]

My Phoenix app had been running fine until recently when I upgraded Ecto to v0.14
I noticed Ecto metadata source was changed from string to tuple, resulting in a json encoding problem
Any advice to fix this?

Postgrex 0.8.4 & Ecto 0.12.1 __meta__: %Ecto.Schema.Metadata{source: "users", state: :loaded}
Postgrex 0.9.1 & Ecto 0.14.3 __meta__: %Ecto.Schema.Metadata{source: {nil, "users"}, state: :loaded}

** (exit) an exception was raised:
    ** (Poison.EncodeError) unable to encode value: {nil, "users"}
        (poison) lib/poison/encoder.ex:213: Poison.Encoder.Any.encode/2
        (poison) lib/poison/encoder.ex:156: anonymous fn/4 in Poison.Encoder.Map.encode/2
        (stdlib) lists.erl:1261: :lists.foldl/3
        (poison) lib/poison/encoder.ex:157: Poison.Encoder.Map.encode/2
        (poison) lib/poison/encoder.ex:156: anonymous fn/4 in Poison.Encoder.Map.encode/2
        (stdlib) lists.erl:1261: :lists.foldl/3
        (poison) lib/poison/encoder.ex:157: Poison.Encoder.Map.encode/2
        (poison) lib/poison/encoder.ex:156: anonymous fn/4 in Poison.Encoder.Map.encode/2

[josevalim]

This is actually a good error to have. You were encoding private metadata information to the client. Please check how to customize your encoders in Poison or provide a default function for such that removes fields like __struct__ and __meta__ as this is not an Ecto bug.

[estevaoam]

@chinh7 if you or someone else still having this issue, I posted a quick coderwall post with the solution for it: https://coderwall.com/p/fhsehq/fix-encoding-issue-with-ecto-and-poison

[radar]

I think this should be made more obvious for newbies somewhere. It is not immediately clear from reading the Ecto or Phoenix docs that I need to add code to my application from an obscure blog post.

[daveslutzkin]

Thanks @estevaoam, very helpful. I agree with @radar, there's got to be a better way to communicate this. render conn, users: users into JSON seems to be pretty common in an API endpoint and this error is very hard to debug inside Phoenix.

[leighhalliday]

Just ran into this too... it's an extremely confusing error. This came up in Phoenix... maybe Phoenix should implement a method to easily remove fields that shouldn't be encoded in the JSON response.

[josevalim]

Poison already provides this:

@derive {Poison.Encoder, only: [:foo, :bar]}

Plus Phoenix recommendation is to not rely on JSON encoding but instead to
use Phoenix views. Check mix phoenix.gen.json

José Valimwww.plataformatec.com.br
http://www.plataformatec.com.br/Founder and Director of R&D

[leighhalliday]

Thank you @josevalim :)

[sebbean]

i c u @radar

+1 @josevalim on the mix phoenix.gen.json that's how i figured it out.

[sebbean]

@estevaoam why not this syntax (just a curiosity):

  def encode(%{__struct__: _} = struct, options) do
    struct
      |> Map.from_struct
      |> sanitize_map
      |> Poison.Encoder.Map.encode(options)
  end

[sebbean]

ok last thing I swear:

@estevaoam your blog post doesn't quite allude to the fact that you can implement a custom implementation per model.

This one breaks it down:
http://www.cultivatehq.com/posts/serialisation-of-ecto-models-in-phoenix-channels-and-views/

That being said, I do think I prefer the mix phoenix.gen.json Phoenix Views method that @josevalim mentioned; it's closer to Rails' ActiveModelSerializers.

[sorentwo]

The View method of serialization has some advantages, but I think flexibility is the biggest. If you ever want to change which attributes are serializer based on the current scope (user) it will be easy to control within the view.

[estevaoam]

@sebbean thank you for your comment.

Your syntax is much better, that final result happened after many tries and fails. 😄
Thanks for the link, indeed is much more detailed.

Just a info: I'm an experienced ruby developer learning some bits of Elixir, thank you for your help. 😃

[jcortesg]

I use this an work:

defmodule App.ChangesetView do
  use App.Web, :view

  def render("error.json", %{changeset: changeset}) do
    errors = Enum.map(changeset.errors, fn {field, detail} ->
      %{
        source: %{ pointer: "/data/attributes/#{field}" },
        title: "Invalid Attribute",
        detail: render_detail(detail)
      }
    end)

    %{errors: errors}
  end

  def render_detail({message, values}) do
    Enum.reduce values, message, fn {k, v}, acc ->
      String.replace(acc, "%{#{k}}", to_string(v))
    end
  end

  def render_detail(message) do
    message
  end
end

ref: here

[tcoopman]

I'm just a beginner with elixir and phoenix, so maybe this is a trivial question. I don't get how to use the mix phoenix.gen.json phoenix views method with channels. Does someone have an example or a pointer to some place with documentation how to use this?

[stevedomin]

@tcoopman can you post this on the Phoenix mailing-list? GH Issues are reserved for bugs plus I'm sure you'll get more response over there.

[tcoopman]

@stevedomin thanks for the suggestion. Just posted a question. Sorry for the noise!

[lucidstack]

Just leaving a tip for those coming here in the future: poison 2.1.0 now has an :except option in the @derive attribute, which is the exact reverse of :only.

So, if you are positive that you want to encode all your fields but __meta__, this is the way:

defmodule Model do
  @derive {Poison.Encoder, except: [:__meta__]}
  schema "models" do
     # blah blah blah
  end
end

Hope this helps!

[rubik]

@lucidstack Good tip, but how can you use Poison 2.1.0 when Ecto locks it at 1.5.2?

[lucidstack]

@rubik that is a very good point.

Fortunately, the current Ecto version, 2.0.0-beta.2 doesn't lock to Poison 1.5.2 anymore, but {:poison, "~> 1.5 or ~> 2.0", optional: true}. So, if your are okay with/can use the latest version, you can use Poison 2.1.0 with it. 😀

[sebbean]

to clarify the last coment:
replace the phoenix_ecto dep in mix.exs:
{:phoenix_ecto, git: "git@github.com:phoenixframework/phoenix_ecto.git"}

and run mix deps.get

[Dania02525]

I like having poison handle this by default, as it prevents having to specify an implementation with derive in each model (as model fields change, etc). Also, would be nice for poison to handle associations not loaded- heres what I'm using:

      defimpl Poison.Encoder, for: Ecto.Association.NotLoaded do
        def encode(struct, options) do
          case struct.__cardinality__ do
            :many -> "[]"
            _ -> "{}"
          end
        end
      end

not a hard workaround, as this code and the implementation for ecto models in general just gets stuck in web.ex, but I could see how it not working out of the box could be frustrating for people used to as_json working pretty much no matter what

[michalmuskala]

The problem with that implementation is that it's plain wrong in many situations. We cannot use something like that as a default.

[josevalim]

The problem with that implementation is that it's plain wrong in many situations.

Can you please expand?

[michalmuskala]

I meant the Ecto.Association.NotLoaded implementation of the poison protocol that returns the empty collections. The collection is not empty when it's not loaded - we don't know if it's empty or not. Returning an empty value for a not loaded association is not correct.

[renanvalentin]

Here is not working 😢 , I'm using Poison ~> 1.5.2 and I've tried to use the @derive on my model, but it keeps throwing the same error:

defmodule Momsfood.Sale do
   use Ecto.Model
   import Ecto.Changeset

   @derive {Poison.Encoder, only: [:price, :sale_time]}
    schema "sales" do
      field :price, :float
      field :sale_time, Ecto.DateTime
      timestamps
    end

   def changeset(sale, params \\ %{}) do
     sale
     |> cast(params, [:price, :sale_time])
   end
 end

And here's my route:

post "/sales" do
   sale = Momsfood.Sale.changeset(%Momsfood.Sale{}, conn.params)

   if sale.valid? do
        Momsfood.Repo.insert(sale)
        send_resp(conn, 201, "")
    else
        send_resp(conn, 400, Poison.encode!(sale.errors))
end

** (Poison.EncodeError) unable to encode value: {:sale_time, "is invalid"}

Any idea ?

Thanks guys!!

[Alamoz]

I just experienced this problem after adding a new model to an existing Phoenix app (upgraded to Phoenix 1.2.0, Elixir 1.3.) The solution shown by @estevaoam solved the problem, with an app warning of: "warning: redefining module Poison.Encoder.Any (current version loaded from _build/dev/lib/poison/ebin/Elixir.Poison.Encoder.Any.beam) lib/myapp/encoder.ex:1"

I find it strange that my original model doesn't experience this problem when served as json with its own (original) controller, yet the formats for the schemas, views and controllers are the same for both the older and newer models. 😕

[Alamoz]

OK, the solution shown by @estevaoam worked for me only in local development mode, but then caused an error preventing a release from building. The way to solve this problem is answered by @josevalim on Stack Overflow:

http://stackoverflow.com/questions/32549712/encoding-a-ecto-model-to-json-in-elixir/32553676#32553676

Using Phoenix as an API seems to be a popular use case. @chrismccord - Perhaps the guide at http://www.phoenixframework.org/docs/ecto-models can be extended to include an example of serving json. Looks like an easy addition. Do you accept PRs for that guide?

[michalmuskala]

A great example of how to deal with json in phoenix are the generated json endpoints with phoenix.gen.json task. As far as I understand the generators are considered mainly as a learning resource, and part of guides.

[elbow-jason]

I've used the defimpl example provided by @Dania02525 with a small change to just return "null" instead.

defimpl Poison.Encoder, for: Ecto.Association.NotLoaded do
  def encode(_struct, _options) do
    "null"
  end
end

And it has worked like a charm so far.

[LoneStar1994]

Привет, товарищи!!

I came across with a solution with a little help from "sjoconnor" @ slack/phoenix.

I hade the same issue "Poison.EncodeError" So what we propose is to tear down from the sending list the ":meta" and ":struct":

def handle_in("show_all", _payload, socket) do
    info = Repo.all(Frutas) #we load the list into "Info" 
            |> Enum.map(fn(info) ->  #then we pipe it through an anon function
                Map.drop(info, [:__meta__, :__struct__]) end) #We tear down the invalid information
  	{:reply, {:ok, %{basket: info}},socket} #And finally we send the information 
  end

That is a pretty "Cave Man" way to do the things, but it works just good!!

Happy programming

[caldempsey]

Just ran into this too when forced to use old versions of Ecto!

[lloydaf]

I guess one should follow the entire tutorial to figure this, or use mix to generate models. I kept thinking I was doing something wrong, but normal maps worked well, lol. Wish poison handles this internally. Is there any reason why we would not want dropping meta to be a standard encoding feature for poison?