Use nodemailer for sending email (with local email dev solution) (com…

…pdemocracy#433)

* Updated database schema to bugfix password reset process. [Fixes compdemocracy#273]

* Added maildev docker container for inspecting emails during dev.

* Ensured the proxied services are seeing the origin host.

* Small fixup from rebase.

* Added SMTP port exposure to maildev container.

* Migrated AWS_REGION config into envvar.

* Added mailgun nodemailer transport. Added fallback through multiple transport mechanisms.

* Added ability for cypress to check maildev inbox on another port.

* e2e: Fixed create_user test.

* e2e: Added checks of password reset flow.

* e2e: Added test stubs for types of emails sent.

* e2e: Run through whole password reset flow, and confirm new password.

* e2e: Added plugin to output more details to stdout.

* e2e: Make more clear when reporter prints to terminal.

* Added log command for troubleshooting GitHub Actions issue.

* e2e: Fixed issue with matching password reset token.

* Added testing of email transport failover.

* Adding docs for email transport configuration. [skip ci]

* Check maildev via API instead of UI.

* Improved documentation of cypress workflow for email transports.

* e2e: Added note about cypress-terminal-report in README.

* Removed straggling TODO.

* Set email transport defaults to match current production.

.github/workflows/cypress-tests.yml

@@ -50,6 +50,13 @@ jobs:
           echo GOOGLE_CREDENTIALS_BASE64=${{ secrets.GOOGLE_CREDENTIALS_BASE64 }} >> server/docker-dev.env
           echo SHOULD_USE_TRANSLATION_API=true >> server/docker-dev.env
 
+        - name: Set server configuration
+          run: |
+            # Test email transport failovers
+            # mailgun: unconfigured transport (will fail)
+            # nonexistent: nonexistent transport (will fail)
+            echo EMAIL_TRANSPORT_TYPES=mailgun,nonexistent,maildev >> server/docker-dev.env
+
       - name: Serve app via docker-compose
         run: docker-compose up --detach
 

docker-compose.yml

@@ -108,6 +108,16 @@ services:
       args:
         GIT_HASH: "${GIT_HASH}"
 
+  maildev:
+    image: maildev/maildev:1.1.0
+    networks:
+      - "polis-dev"
+    ports:
+      # User interface
+      - "1080:80"
+      # SMTP port
+      - "25:25"
+
 networks:
   polis-dev:
 

docs/deployment.md

@@ -107,6 +107,60 @@ We use Google to automatically translate submitted comments into the language of
    [base64-encoder]: https://codepen.io/bsngr/pen/awuDh
 
 
+## Email Transports
+
+We use [Nodemailer][] to send email. Nodemailer uses various built-in and
+packaged _email transports_ to send email via SMTP or API, either directly or
+via third-party platforms.
+
+Each transport needs a bit of hardcoded scaffold configuration to make it work,
+which we welcome via code contribution. But after this, others can easily use
+the same email transport by setting some configuration values via environment
+variable or otherwise.
+
+We use `EMAIL_TRANSPORT_TYPES` to set email transports and their fallback
+order. Each transport has a keyword (e.g., `maildev`). You may set one or more
+transports, separated by commas. If you set more than one, then each transport
+will "fallback" to the next on failure.
+
+For example, if you set `aws-ses,mailgun`, then we'll try to send via
+`aws-ses`, but on failure, we'll try to send via `mailgun`. If Mailgun fails,
+the email will not be sent.
+
+   [Nodemailer]: https://nodemailer.com/about/
+
+### Configuring transport: `maildev`
+
+Note: The [MailDev][] email transport is for **development purposes only**. Ensure it's disabled in production!
+
+1. Add `maildev` into the `EMAIL_TRANSPORT_TYPES` configuration.
+
+This transport will work automatically when running via Docker Compose, accessible on port 1080.
+
+   [MailDev]: https://github.com/maildev/maildev
+
+### Configuring transport: `aws-ses`
+
+1. Add `aws-ses` into the `EMAIL_TRANSPORT_TYPES` configuration.
+2. Set the `AWS_ACCESS_KEY_ID` and `AWS_SECRET_ACCESS_KEY` configuration.
+
+### Configuring transport: `mailgun`
+
+1. Add `mailgun` into the `EMAIL_TRANSPORT_TYPES` configuration.
+2. Set the `MAILGUN_API_KEY` and `MAILGUN_DOMAIN` configuration.
+
+### Adding a new transport
+
+1. [Find a transport for the service you require][transports] (or write your
+   own!)
+2. Add any new transport configuration to `getMailOptions(...)` in
+   [`server/email/senders.js`][mail-senders].
+3. Submit a pull request.
+
+   [transports]: https://github.com/search?q=nodemailer+transport
+   [mail-senders]: /server/email/senders.js
+
+
 ## Database Migrations
 
 When we need to update the Polis database, we use SQL migration files.

e2e/README.md

@@ -17,6 +17,16 @@ To run these tests:
 4. Click on tests to run
     - Alternatively, you can run all tests automatically: `npm test`
 
+## Debugging
+
+- We use [`cypress-terminal-report`][] to ensure that logs display not only in
+  Cypress's [Test Runner][test-runner] browser UI, but also in the console.
+    - These only print when a test has failed, to reduce noise.
+    - Logs of failed tests can be seen on CI server (GitHub Actions).
+
+   [`cypress-terminal-report`]: https://github.com/archfz/cypress-terminal-report#readme
+   [test-runner]: https://docs.cypress.io/guides/core-concepts/test-runner.html
+
 ## Notes
 
 - We keep some helper scripts in `package.json`.

e2e/cypress/integration/polis/client-admin/create_user.spec.js

@@ -1,8 +1,4 @@
 describe('Create User page', () => {
-  before(() => {
-    cy.fixture('users.json').as('users')
-  })
-
   beforeEach(() => {
     const [name, email, password] = ['Dummy', 'test@polis.test', 'testpassword']
 
@@ -21,11 +17,12 @@ describe('Create User page', () => {
   })
 
   it('does not create a new user with existing email address', function () {
-    const existingUser = this.users.moderator
-
     // Attempt to recreate existing user.
-    cy.get('input#createUserEmailInput').clear().type(existingUser.email)
-    cy.get('button#createUserButton').click()
+    cy.fixture('users.json').then(users => {
+      const existingUser = users.moderator
+      cy.get('input#createUserEmailInput').clear().type(existingUser.email)
+      cy.get('button#createUserButton').click()
+    })
 
     cy.wait('@authNew').then(xhr => {
       cy.wrap(xhr).its('status').should('eq', 403)

e2e/cypress/integration/polis/emails.spec.js

@@ -0,0 +1,111 @@
+describe('Emails', () => {
+  const MAILDEV_HTTP_PORT = '1080'
+  // See: https://github.com/maildev/maildev/blob/master/docs/rest.md
+  const MAILDEV_API_BASE = `${Cypress.config().baseUrl}:${MAILDEV_HTTP_PORT}`
+
+  beforeEach(() => {
+    cy.server()
+    cy.route('POST', Cypress.config().apiPath + '/auth/pwresettoken').as('resetPassword')
+
+    cy.request('DELETE', MAILDEV_API_BASE + '/email/all')
+  })
+
+  it('sends for failed password reset', function () {
+    const nonExistingEmail = 'nonexistent@polis.test'
+    cy.visit('/pwresetinit')
+    cy.get('input[placeholder="email"]').type(nonExistingEmail)
+    cy.contains('button', 'Send password reset email').click()
+    cy.wait('@resetPassword').its('status').should('eq', 200)
+    cy.location('pathname').should('eq', '/pwresetinit/done')
+
+    cy.request('GET', MAILDEV_API_BASE + '/email')
+      .then(resp => {
+        const email = resp.body.shift()
+        console.log(email)
+        cy.wrap(email).its('subject').should('contain', 'Password Reset Failed')
+        cy.wrap(email).its('to').its(0).its('address').should('contain', nonExistingEmail)
+      })
+  })
+
+  it('sends for successful password reset', function () {
+    // Create a new user account, so we can actually change password.
+    const randomInt = Math.floor(Math.random() * 10000)
+    const newUser = {
+      email: `user${randomInt}@polis.test`,
+      name: `Test User ${randomInt}`,
+      password: 'testpassword',
+      newPassword: 'newpassword',
+    }
+
+    const strictFail = true
+    cy.signup(newUser.name, newUser.email, newUser.password, strictFail)
+
+    cy.logout()
+
+    // Request password reset on new account
+    cy.visit('/pwresetinit')
+    cy.get('input[placeholder="email"]').type(newUser.email)
+    cy.contains('button', 'Send password reset email').click()
+    cy.wait('@resetPassword').its('status').should('eq', 200)
+    cy.location('pathname').should('eq', '/pwresetinit/done')
+
+    cy.request('GET', MAILDEV_API_BASE + '/email')
+      .then(resp => {
+        const email = resp.body.shift()
+        cy.wrap(email).its('subject').should('contain', 'Polis Password Reset')
+        cy.wrap(email).its('to').its(0).its('address').should('contain', newUser.email)
+
+        // Has password reset link with proper hostname.
+        cy.wrap(email).its('text').should('contain', `${Cypress.config().baseUrl}/pwreset/`)
+
+        const emailContent = email.text
+        console.log(email)
+        const tokenRegex = new RegExp('/pwreset/([a-zA-Z0-9]+)\n', 'g')
+        const match = tokenRegex.exec(emailContent)
+        // First "url" is email domain. Second url is the one we want.
+        cy.log(JSON.stringify(match))
+        const passwordResetToken = match[1]
+
+        // Submit password reset form with new password.
+        cy.visit(`/pwreset/${passwordResetToken}`)
+
+        cy.route('POST', Cypress.config().apiPath + '/auth/password').as('newPassword')
+
+        cy.get('form').within(() => {
+          cy.get('input[placeholder="new password"]').type(newUser.newPassword)
+          cy.get('input[placeholder="repeat new password"]').type(newUser.newPassword)
+          cy.get('button').click()
+        })
+
+        cy.wait('@newPassword').then((xhr) => {
+          expect(xhr.status).to.equal(200)
+        })
+      })
+
+    cy.logout()
+
+    // Login with new password.
+    cy.login(newUser.email, newUser.newPassword)
+
+    cy.url().should('eq', Cypress.config().baseUrl + '/')
+  })
+
+  // TODO: Re-enabled account verification.
+  it.skip('sends when new account requires verification', function () {
+  })
+
+  // TODO: Allow batch interval to be skipped or reduced for tests.
+  it.skip('sends when new statements arrive', function () {
+  })
+
+  // TODO: Fix data export.
+  it.skip('sends when data export is run', function () {
+  })
+
+  // TODO: Find way to test embedded iframe.
+  it.skip('sends when new conversation is auto-created', function () {
+  })
+
+  it.skip('sends when new statement available for moderation', function () {
+  })
+})

e2e/cypress/plugins/index.js

@@ -18,4 +18,5 @@
 module.exports = (on, config) => {
   // `on` is used to hook into various events Cypress emits
   // `config` is the resolved Cypress config
+  require('cypress-terminal-report/src/installLogsPrinter')(on)
 }

e2e/cypress/support/commands.js

@@ -34,7 +34,7 @@ Cypress.Commands.add("logout", () => {
     })
 })
 
-Cypress.Commands.add("signup", (name, email, password) => {
+Cypress.Commands.add("signup", (name, email, password, strictFail=false) => {
   cy.request({
     method: 'POST',
     url: Cypress.config().apiPath + '/auth/new',
@@ -44,7 +44,7 @@ Cypress.Commands.add("signup", (name, email, password) => {
       gatekeeperTosPrivacy: true,
       password: password
     },
-    failOnStatusCode: false
+    failOnStatusCode: strictFail
   }).then(resp => {
     // Expand success criteria to allow user already existing.
     // TODO: Be smarter with seeding users so we only create once.
@@ -110,3 +110,17 @@ Cypress.Commands.add('seedComment', (...args) => {
     is_seed: true
   })
 })
+
+// Allow visiting maildev inbox urls, to test sending of emails.
+// See: https://github.com/cypress-io/cypress/issues/944#issuecomment-651503805
+Cypress.Commands.overwrite(
+  'visit',
+  (originalFn, url, options) => {
+    if (url.includes(':1080')) {
+      cy.window().then(win => {
+        return win.open(url, '_self');
+      });
+    }
+    else { return originalFn(url, options); }
+  }
+);

e2e/cypress/support/index.js

@@ -28,3 +28,11 @@ before(() => {
     }
   })
 })
+
+// Register the log collector for logging activity to terminal.
+const reporterOptions = {
+  // When to print terminal logs for tests.
+  // Options: onFail, always
+  printLogs: 'onFail',
+}
+require('cypress-terminal-report/src/installLogsCollector')(reporterOptions)

e2e/package.json

@@ -15,6 +15,7 @@
   "author": "Benjamin Rosas <ben@aliencyb.org>",
   "devDependencies": {
     "cypress": "^4.9.0",
+    "cypress-terminal-report": "^1.4.1",
     "eslint": "^7.1.0",
     "eslint-config-prettier": "^6.11.0",
     "eslint-config-prettier-standard": "^3.0.1",

file-server/nginx.site.default.conf

@@ -4,6 +4,7 @@ server {
 
     location / {
         proxy_set_header X-Forwarded-Proto $scheme;
+        proxy_set_header Host $host;
         proxy_pass http://server:5000;
     }
 }

server/docker-dev.env

@@ -26,3 +26,10 @@ SHOULD_USE_TRANSLATION_API=false
 STATIC_FILES_ADMINDASH_PORT=8080
 STATIC_FILES_HOST=file-server
 STATIC_FILES_PORT=8080
+
+AWS_REGION=us-east-1
+
+# Options: maildev, aws-ses, mailgun
+# Example: `aws-ses,mailgun` would try sending via AWS SES first, and fallback to Mailgun on error.
+EMAIL_TRANSPORT_TYPES=maildev
+POLIS_FROM_ADDRESS="Example <team@example.com>"