Getting Email to work via Docker

[dakotabenjamin]

I am hoping to push some more changes to get email working in the Docker environment, but I haven't yet come to a solution. I am running on an AWS EC2 using Amazon SES. Here are the changes I made so far (branched from PR #186):

1. Added port 25 to the web container in docker-compose (not actually sure if this is necessary, openstreetmap-website doesn't in their docker-compose file but I don't see a reason why it shouldn't be there)
2. make more changes to the settings.local.yml file to add the right envvars for email. I am fairly certain these are all correct, because the production logs in the web container show a successful message when an email is triggered:

I, [2021-03-09T20:41:38.898959 #289]  INFO -- : [a97570ab-b0a2-4d9a-ac85-375ed39bee08] Started GET "/user/osmseed1/confirm/resend" for 142.105.9.97 at 2021-03-09 20:41:38 +0000
I, [2021-03-09T20:41:38.900528 #289]  INFO -- : [a97570ab-b0a2-4d9a-ac85-375ed39bee08] Processing by UsersController#confirm_resend as HTML
I, [2021-03-09T20:41:38.900596 #289]  INFO -- : [a97570ab-b0a2-4d9a-ac85-375ed39bee08]   Parameters: {"display_name"=>"osmseed1"}
I, [2021-03-09T20:41:38.916996 #289]  INFO -- : [a97570ab-b0a2-4d9a-ac85-375ed39bee08] [ActiveJob] Enqueued ActionMailer::MailDeliveryJob (Job ID: 3c8884d1-338d-4560-a027-4d74aa185bea) to DelayedJob(mailers) with arguments: "UserMailer", "signup_confirm", "deliver_now", {:args=>[#<GlobalID:0x0000555fe04b4840 @uri=#<URI::GID gid://open-street-map/User/9>>, #<GlobalID:0x0000555fe0473f70 @uri=#<URI::GID gid://open-street-map/UserToken/56>>]}

Results from testing the host connection (Amazon EC2):

$ telnet email-smtp.us-east-1.amazonaws.com 25
Trying 34.236.168.54...
Connected to email-smtp.us-east-1.amazonaws.com.
Escape character is '^]'.
220 email-smtp.amazonaws.com ESMTP SimpleEmailService-d-WPOJ4FIL9 068Nx7FlW2eAQYaFKIZ8
451 4.4.2 Timeout waiting for data from client.
Connection closed by foreign host.

Detailed changes:

$ git diff
diff --git a/docker-compose.yml b/docker-compose.yml
index f16e50c..99b87ef 100644
--- a/docker-compose.yml
+++ b/docker-compose.yml
@@ -7,6 +7,7 @@ services:
       dockerfile: Dockerfile
     ports:
       - "80:80"
+      - "25:25"
     env_file: 
       - ./.env
     depends_on:
diff --git a/images/web/start.sh b/images/web/start.sh
index 378b8f6..5309d9c 100755
--- a/images/web/start.sh
+++ b/images/web/start.sh
@@ -14,11 +14,20 @@ production:
 # Setting up the SERVER_URL and SERVER_PROTOCOL
 # Rails is supposed to pick up env with OPENSTREETMAP prefix but for some reason this is not the case
 # So we'll just manually override the settings.local.yml for now
-sed -i -e 's/server_url: "localhost"/server_url: "'$OPENSTREETMAP_server_url'"/g' $workdir/config/settings.local.yml
+sed -i -e 's/server_url: "openstreetmap.example.com"/server_url: "'$OPENSTREETMAP_server_url'"/g' $workdir/config/settings.local.yml
 sed -i -e 's/server_protocol: "http"/server_protocol: "'$OPENSTREETMAP_server_protocol'"/g' $workdir/config/settings.local.yml
 
 # # Setting up the email
-sed -i -e 's/openstreetmap@example.com/'$MAILER_SENDER_EMAIL'/g' $workdir/config/settings.local.yml
+# sed -i -e 's/openstreetmap@example.com/'$MAILER_SENDER_EMAIL'/g' $workdir/config/settings.local.yml
+sed -i -e 's/smtp_address: "localhost"/smtp_address: "'$MAILER_ADDRESS'"/g' $workdir/config/settings.local.yml
+sed -i -e 's/smtp_user_name: null/smtp_user_name: "'$MAILER_USERNAME'"/g' $workdir/config/settings.local.yml
+sed -i -e 's/smtp_password: null/smtp_password: "'$MAILER_PASSWORD'"/g' $workdir/config/settings.local.yml
+sed -i -e 's/smtp_domain: "localhost"/smtp_domain: "'$MAILER_DOMAIN'"/g' $workdir/config/settings.local.yml
+sed -i -e 's/email_from: "OpenStreetMap <openstreetmap@example.com>"/email_from: "HOTOSM <'$MAILER_SENDER_EMAIL'>"/g' $workdir/config/settings.local.yml
+sed -i -e 's/email_return_path: "openstreetmap@example.com"/email_return_path: "'$MAILER_SENDER_EMAIL'"/g' $workdir/config/settings.local.yml
+sed -i -e 's/smtp_authentication: null/smtp_authentication: "login"/g' $workdir/config/settings.local.yml
+sed -i -e 's/smtp_enable_starttls_auto: false/smtp_enable_starttls_auto: true/g' $workdir/config/settings.local.yml
+
 
 # # Set up iD key
 sed -i -e 's/id-key/'$OPENSTREETMAP_id_key'/g' $workdir/config/settings.local.yml

[eternaltyro]

The issue seems to be the way email is setup in the openstreetmap-website container. By default, emails are not being delivered. ActionMailer's delivery array is not being populated. When I ran tcpdump, I didn't catch any connection attempt to AWS SES with the default smtp_settings.

I was however able to send an email from the ruby console using ActionMailer

Logs from interactive ruby console.

irb(main):045:0> class MyNewMailer < ActionMailer::Base
irb(main):046:1>   default :from => "xyz@hotosm.org"
irb(main):047:1>   def welcome_email
irb(main):048:2>     mail(:to => "xyz@hotosm.org", :subject => "hello from ActionMailer", :body => "Hopefully, this gets delivered" )
irb(main):049:2>   end
irb(main):050:1> end
=> :welcome_email
irb(main):051:0> MyNewMailer.welcome_email.deliver
MyNewMailer#welcome_email: processed outbound mail in 0.7ms
Delivered mail 6049f9c5e97a6_4822ad14d1b19a080776@0693fde7b2ae.mail (334.1ms)
Date: Thu, 11 Mar 2021 11:06:45 +0000
From: xyz@hotosm.org
Reply-To: noreply@hotosm.org
To: xyz@hotosm.org
Message-ID: <6049f9c5e97a6_4822ad14d1b19a080776@0693fde7b2ae.mail>
Subject: hello from ActionMailer
Mime-Version: 1.0
Content-Type: text/plain;
 charset=UTF-8
Content-Transfer-Encoding: 7bit
Hopefully, this gets delivered
=> #<Mail::Message:47078490503280, Multipart: false, Headers: <Date: Thu, 11 Mar 2021 11:06:45 +0000>, <From: xyz@hotosm.org>, <Reply-To: noreply@hotosm.org>, <To: xyz@hotosm.org>, <Message-ID: <6049f9c5e97a6_4822ad14d1b19a080776@0693fde7b2ae.mail>>, <Subject: hello from ActionMailer>, <Mime-Version: 1.0>, <Content-Type: text/plain>, <Content-Transfer-Encoding: 7bit>>
irb(main):052:0> MyNewMailer.deliveries
=> []
irb(main):053:0> MyNewMailer.smtp_settings
=> {:address=>"email-smtp.us-east-1.amazonaws.com", :port=>587, :domain=>"hotosm.org", :user_name=>"AKIAXXXXXXXXX", :password=>"SECRETPASSWORD", :authentication=>"plain", :enable_starttls_auto=>true}
irb(main):054:0> MyNewMailer.perform_deliveries
=> true
irb(main):055:0>

[geohacker]

@dakotabenjamin @eternaltyro you are right that this is because the openstreetmap-website isn't actually triggering those emails. I'm glad you were able to test through the Rails console.

I'm 99% sure that this is because the Rails jobs worker is not started. If you uncomment this like https://github.com/developmentseed/osm-seed/blob/develop/images/web/start.sh#L45 and deploy, that should queue the emails and trigger send.

That command starts the worker that looks for jobs submitted by the rails app. When an email request comes it's added to the worker queue.

[eternaltyro]

Thanks, @geohacker . That helped. I had to run the worker in the background since it doesn't exit and was blocking apache2 from starting.

[geohacker]

Glad that it helped!