Add support for integration testing

[stevedomin]

The current TestAdapter breaks with integration testing using hound or wallaby.

The best option seem to be implementing some kind of ownership-based mechanism similar to what Ecto 2 does (elixir-ecto/ecto#1237)

@barisbalic @hubertlepicki @lau I would love to get your thoughts on that

[victorsolis]

👍

Prior to this getting implemented, is there another way to test asynchronous deliveries, like from within a Task?

[stevedomin]

Depending on how we decide to implement this, we might that tackle both use case in one go. I'll keep your comment in mind when we work on it.

[monicao]

A possible workaround is to use the Swoosh.Adapters.Local and get Hound to navigate to /dev/mailbox.

I set up a separate mix environment called 'acceptance' that uses Swoosh.Adapters.Local. The test environment still uses Swoosh.Adapters.Test.

Code snippets: https://gist.github.com/monicao/0e31545ea3d37cab342504bd8eeb0a87

[stevedomin]

Thanks for sharing this @monicao

[manukall]

I'm using a custom adapter for this, following a pattern that I just published a blog post about:

defmodule MyApp.SwooshAdapter.Test do
  use Swoosh.Adapter
  use GenServer

  def start_link do
    GenServer.start_link(__MODULE__, [], name: __MODULE__)
  end

  def subscribe do
    GenServer.call(__MODULE__, {:subscribe, self()})
  end

  def deliver(email, _config) do
    GenServer.call(__MODULE__, {:deliver, email})
  end


  # SERVER
  def handle_call({:subscribe, pid}, _from, listeners) do
    {:reply, :ok, [pid | listeners]}
  end

  def handle_call({:deliver, email}, _from, listeners) do
    send_to_listeners(listeners, {:email, email})
    {:reply, :ok, listeners}
  end

  defp send_to_listeners(listeners, message) do
    for listener <- listeners do
      send listener, message
    end
  end
end

[keown]

thanks a lot @manukall !!

[LostKobrakai]

This should work very much like the phoenix_ecto plug for the ecto sandbox: https://gist.github.com/LostKobrakai/9077143caacf534f9c4743cfc18a148e

[stevedomin]

@LostKobrakai apologies for not replying earlier. Would you like to submit a PR with that test adapter?

[LostKobrakai]

@stevedomin I've not tested it beyond my own needs and I'm currently quite busy. So if someone wants to make an official adapter work of of it feel free. I haven't got the time to make it more official/proper at the moment.

[stevedomin]

@LostKobrakai of course, makes sense! We will take care of it.

[hisapy]

This one is inspired on Bamboo.TestAdapter

defmodule Webapp.Mailer.TestAdapter do
  use Swoosh.Adapter

  def deliver(email, _config) do
    send(test_process(), {:email, email})
    {:ok, %{}}
  end

  defp test_process do
    Application.get_env(:swoosh, :shared_test_process) || self()
  end
end

# in your config/test.exs
config :webapp, Webapp.Mailer, adapter: Webapp.Mailer.TestAdapter

So if your Webapp is sending email inside Task.async or another async mechanism the integration test should begin with

setup do
   Application.put_env(:swoosh, :shared_test_process, self())
   :ok
end

# and somewhere in your assertions, something like
assert_email_sent(subject: subject, to: recipients)

# or 
receive do
  {:email, email} ->
    # assert other email fiels
    # _i.e. Mailgun: at the time of this writting there are no `assert_equal` support to `assert_email_sent` 
    # with `:provider_options` to assert email contains correct `recipient-variables` 
    assert email.subject == subject
    assert email.to == recipients
    assert email.provider_options == %{recipient_vars: recipient_vars}
after
  1_000 ->
    raise "No updates email delivered"
end

[lpil]

Would you be open to a pull request to the test adapter that adds this functionality?

[stevedomin]

@lpil yes, definitely!

[dustinfarris]

I tried to add this to the Swoosh Test Adapter, but had a hard time writing tests for it. The async process seems to get tangled up with the test runner.

I ended up copying @hisapy's example test adapter into my own project and it is working well.

[ivan-kolmychek]

We have also stumbled upon some problem with testing swoosh in our higher-level tests, as we test there interaction between few processes.

We have tried out an approach with "custom" adapter using Mox (https://github.com/plataformatec/mox). With minor workarounds it works quite all right, but I can't say for sure until enough time passes and no problems are discovered.

I'm leaving this message here as a tip for others, as I don't have time right now to provide more details, hope I can find time for that a bit later.

[ivan-kolmychek]

Here is a summary, without project-specific stuff.

In test helper (somewhere in test/support/):

defmodule BlahBlah.TestHelpers.BlahBlahEmails do
  alias BlahBlah.Mailer.AdapterMock, as: MailerMock
  
  # for multi-process high-level tests
  def allow_processes_to_send_mails(%{pids: pids})
  when is_list(pids) do
    pids |> Enum.each(fn pid ->
      :ok = allow_process_to_send_mails(%{pid: pid})
    end)
    :ok
  end

  # for single-process high-level tests
  def allow_process_to_send_mails(%{pid: pid}) do
    test_pid = self()
    MailerMock
    |> Mox.allow(test_pid, pid)
    |> Mox.stub(:validate_config, fn _ -> :ok end)
    |> Mox.stub(:deliver, fn email, _config ->
      # NOTE: self() here will not be the process of test.
      # This is why we need the test_pid that's set up outside.
      send(test_pid, {:email, email})

      {:ok, %{}}
    end)

    :ok
  end
end

Examples of/for error tests are omitted, as principle is pretty much the same, you control what fake adapter returns as result of :deliver.

If you already provide all required pids as either pid or pids in contexts, helpers can be used in setup.
Otherwise, nothing really prevents you from calling them directly, I did that in example below just to show it.

In test_helper.exs:

defmodule BlahBlah.Mailer.ValidateConfigAdapterBehaviour do
  @callback validate_config(any()) :: :ok
end

Mox.defmock(BlahBlah.Mailer.AdapterMock,
  for: [Swoosh.Adapter, BlahBlah.Mailer.ValidateConfigAdapterBehaviour])

Application.put_env(:admin, BlahBlah.Mailer,
  adapter: BlahBlah.Mailer.AdapterMock)

Unfortunately, Swoosh.Adapter does not directly provide @callback for the validate_config(), but Mox can accept multiple behaviours, so we just defined our own in-place and that solved the problem.

In test itself:

defmodule BlahBlah.SomeMultiProcessTest do
  ...
  import Mox
  import BlahBlah.TestHelpers.BlahBlahEmails
  import Swoosh.TestAssertions
  ...
  setup [ 
    ... set up processes under tests and such ...
    :verify_on_exit!
  ]
  ...
  test "something", %{pid_a: pid_a, pid_b: pid_b, ...} do
     ...
     allow_process_to_send_mails(%{pids: [pid_a, pid_b, ...]})
     
     # or
     allow_process_to_send_mails(%{pid: pid_a})
     allow_process_to_send_mails(%{pid: pid_b})
     ...
     do actual test here
     ...
     assert_email_sent BlahBlah.Emails.whatever()
     assert_email_sent BlahBlah.Emails.another()
     ...
  end
  ...
end

This way we let Mox handle details of isolating one test from another.

As for limitations, while we don't go over 3-4 processes in our tests, I wrote a quick dirty test with bunch of genservers sending fake emails and it looks like it does not break with 20+ processes as well.

Any feedback is welcome, especially if you will find any faults that we have not yet noticed.

[ivan-kolmychek]

A small note, just in case anyone will have same issue - after update today we got weird error in our tests:

warning: this clause cannot match because a previous clause at line 282 always matche
s                                                                                    
  deps/mox/lib/mox.ex:281   

Placing the IO.inspect([ info | body ]) a line above the deps/mox/lib/mox.ex:281 and running tests showed that validate_config was being defined twice:

...

Elixir.BlahBlah.Mailer.AdapterMock: [                                                   
  {:def, [context: Mox, import: Kernel],                                             
   [                                                                                 
     {:__mock_for__, [context: Mox], Mox},                                           
     [do: [Swoosh.Adapter, BlahBlah.Mailer.ValidateConfigAdapterBehaviour]]             
   ]},                                                                               
  {{:., [], [Swoosh.Adapter, :module_info]}, [], [:module]},                         
  {{:., [], [BlahBlah.Mailer.ValidateConfigAdapterBehaviour, :module_info]}, [],        
   [:module]},                                                                       
  {:def, [context: Mox, import: Kernel],                                             
   [                                                                                 
     {:validate_dependency, [context: Mox], []},                                     
     [                                                                               
       do: {{:., [], [{:__aliases__, [alias: false], [:Mox]}, :__dispatch__]},       
        [], [{:__MODULE__, [], Mox}, :validate_dependency, 0, []]}                   
     ]                                                                               
   ]},                                                                               
  {:def, [context: Mox, import: Kernel],                                             
   [                                                                                 
     {:validate_config, [context: Mox], [{:arg1, [], Elixir}]},                      
     [                                                                               
       do: {{:., [], [{:__aliases__, [alias: false], [:Mox]}, :__dispatch__]},       
        [],                                                                          
        [{:__MODULE__, [], Mox}, :validate_config, 1, [{:arg1, [], Elixir}]]}        
     ]                                                                               
   ]},                                                                               
  {:def, [context: Mox, import: Kernel],                                             
   [                                                                                 
     {:deliver, [context: Mox], [{:arg1, [], Elixir}, {:arg2, [], Elixir}]},         
     [                                                                               
       do: {{:., [], [{:__aliases__, [alias: false], [:Mox]}, :__dispatch__]},       
        [],                                                                          
        [                                                                            
          {:__MODULE__, [], Mox},                                                    
          :deliver,                                                                  
          2,                                                                         
          [{:arg1, [], Elixir}, {:arg2, [], Elixir}]                                 
        ]}                                                                           
     ]                                                                               
   ]},                                                                               
  {:def, [context: Mox, import: Kernel],
   [
     {:validate_config, [context: Mox], [{:arg1, [], Elixir}]},
     [
       do: {{:., [], [{:__aliases__, [alias: false], [:Mox]}, :__dispatch__]},
        [],
        [{:__MODULE__, [], Mox}, :validate_config, 1, [{:arg1, [], Elixir}]]}
     ]
   ]}
]

...

This is because validate_config was added to Swoosh.Adapter behaviour, and, as I've mentioned previously in this thread, we had to define our own behaviour to be able to mock it.

So changing

defmodule BlahBlah.Mailer.ValidateConfigAdapterBehaviour do
  @callback validate_config(any()) :: :ok
end

Mox.defmock(BlahBlah.Mailer.AdapterMock,
  for: [Swoosh.Adapter, BlahBlah.Mailer.ValidateConfigAdapterBehaviour])

to just

Mox.defmock(BlahBlah.Mailer.AdapterMock,
  for: [Swoosh.Adapter])

solves the issue.

[jc00ke]

What about an adapter where we can assert on the contents of a queue, not so unlike Oban.drain_queue/1. Seems like the message passing gets pretty sticky, but if there was a general "mailbox" that all emails went to, then finding your specific email would be O(n), which I'd assume to be tolerable.

[ivan-kolmychek]

@jc00ke I think you can achieve this with Mox-based setup by spinning a process to store the list of mails and sending message to it in Mox.stub(:deliver, fn email, _config -> ... end) part, instead of sending it to test process.

In this case passing messages around is still there, of course. In my (very limited) experience it's not a big problem, the spinning up a process per test may be a bigger one.

[jc00ke]

I used @hisapy's method but the assert_email_sent assertion doesn't work in all situations. Asserting in a receive/1 block works when assert_email_sent doesn't, specifically when I use Task.start/1. 🤷‍♂️