BE flake: metabase.pulse-test/multiple-recipients-test

[calherries]

example failure here

logs:

FAIL in metabase.pulse-test/multiple-recipients-test (pulse_test.clj:397)
Pulse should be sent to two recipients sent to :email channel 
with temporary :model/Card with attributes
{:creator_id 1,
 :database_id 1,
 :dataset_query
 {:database 1,
  :type :query,
  :query {:source-table 8, :aggregation [["count"]], :breakout [[:field 49 {:temporal-unit :day}]]}},
 :display :line,
 :name "Test card",
 :visualization_settings {:graph.dimensions ["DATE"], :graph.metrics ["count"]},
 :created_at #t "2024-05-21T19:11:19.102324Z[UTC]",
 :updated_at #t "2024-05-21T19:11:19.102325Z[UTC]"}

 
with temporary :model/Pulse with attributes
{:creator_id 1,
 :name "Pulse Name",
 :created_at #t "2024-05-21T19:11:19.122568Z[UTC]",
 :updated_at #t "2024-05-21T19:11:19.122569Z[UTC]"}

 
with temporary :model/PulseCard with attributes
{:position 0, :include_csv false, :include_xls false, :pulse_id 267, :card_id 1800}

 
with temporary :model/PulseChannel with attributes
{:channel_type :email,
 :details {},
 :schedule_type :daily,
 :schedule_hour 15,
 :created_at #t "2024-05-21T19:11:19.127194Z[UTC]",
 :updated_at #t "2024-05-21T19:11:19.127195Z[UTC]",
 :pulse_id 267}

 
with temporary :model/PulseChannelRecipient with attributes
{:user_id 1, :pulse_channel_id 167}

 
Setting :email-smtp-host = "fake_smtp_host"
 
Setting :email-smtp-port = 587
 
Setting :site-url = "https://metabase.com/testmb"
 
with temporary :model/PulseChannelRecipient with attributes
{:user_id 3, :pulse_channel_id 167}


expected: {"crowberto@metabase.com" [{:bcc #{"crowberto@metabase.com"
                                             "rasta@metabase.com"},
                                      :body [{"Pulse Name" true}
                                             {:content #<Class@1a6c5a9e java.net.URL>,
                                              :content-id true,
                                              :content-type "image/png",
                                              :type :inline}
                                             {:content #<Class@1a6c5a9e java.net.URL>,
                                              :content-id true,
                                              :content-type "image/png",
                                              :type :inline}],
                                      :from "notifications@metabase.com",
                                      :subject "Pulse: Pulse Name"}],
           "rasta@metabase.com" [{:bcc #{"crowberto@metabase.com"
                                         "rasta@metabase.com"},
                                  :body [{"Pulse Name" true}
                                         {:content #<Class@1a6c5a9e java.net.URL>,
                                          :content-id true,
                                          :content-type "image/png",
                                          :type :inline}
                                         {:content #<Class@1a6c5a9e java.net.URL>,
                                          :content-id true,
                                          :content-type "image/png",
                                          :type :inline}],
                                  :from "notifications@metabase.com",
                                  :subject "Pulse: Pulse Name"}]}
  actual: {}

The test fails because there is no matching email in the inbox.

The first flake happened on May 17. It has flaked at least 6 times since then (writing this on 24 May). Source here. It's interesting that the failures appear clustered on certain days, and not evenly distributed.

@ngoc because this might have something to do with the pulse changes you merged to master on May 7

This flake is very similar and likely related: #43321

[qnkhuat]

Tried to reproduce this today without any luck so far.

I don't think it relates to the recent change I made with pulse because those changes are purely about how pulses are scheduled. These tests are about pulse behavior.

It's interesting that both flaky tests are related to email tho.

[qnkhuat]

Fixed by reworking how test setup works in #43924