AWS Cognito Auth Fails in Cypress automated test but works otherwise

[cxk280]

Description

I am deploying a Gatsby app with AWS Amplify and using AWS Cognito for user authentication. My login form works as expected when manually using the site. However, when I run a Cypress e2e test, the console displays the following on page load:

AuthError - 
            Error: Amplify has not been configured correctly. 
            The configuration object is missing required auth properties. 
            Did you run `amplify push` after adding auth via `amplify add auth`?

And then form submission action results in NoUserPoolError: Authentication Error.

To make things stranger, when I manually enter the very same credentials into the test runner Chrome browser, login succeeds!

Steps to reproduce

I built this with https://github.com/dabit3/gatsby-auth-starter-aws-amplify as boilerplate. The Login page and functionality are mostly unchanged. My package.json is below:

{
  "name": "frontend",
  "description": "frontend",
  "version": "1.0.0",
  "author": "Me",
  "dependencies": {
    "@stripe/stripe-js": "^1.8.0",
    "aws-amplify": "^3.0.20",
    "aws-amplify-react": "^4.1.19",
    "axios": "^0.19.2",
    "dotenv": "^8.2.0",
    "gatsby": "^2.13.45",
    "gatsby-image": "^2.2.7",
    "gatsby-plugin-manifest": "^2.2.4",
    "gatsby-plugin-offline": "^3.2.17",
    "gatsby-plugin-react-helmet": "^3.1.2",
    "gatsby-plugin-sharp": "^2.6.19",
    "gatsby-source-filesystem": "^2.1.7",
    "gatsby-source-graphql": "^2.1.3",
    "gatsby-transformer-sharp": "^2.5.11",
    "lodash": "^4.17.19",
    "prop-types": "^15.7.2",
    "react": "^16.8.6",
    "react-dom": "^16.8.6",
    "react-helmet": "^6.1.0"
  },
  "keywords": [
    "gatsby"
  ],
  "license": "UNLICENSED",
  "scripts": {
    "build": "gatsby build",
    "clean": "gatsby clean",
    "develop": "BUILD_ENV=development gatsby develop --https --cert-file ./server.crt --key-file ./server.key",
    "format": "prettier --write src/**/*.{js,jsx}",
    "serve": "gatsby serve -p 8000",
    "jest": "jest",
    "jest:watch": "jest --watch",
    "jest:us": "jest --updateSnapshot",
    "cy:run": "cypress run",
    "cy:open": "cypress open",
    "test:e2e:run": "start-test serve 8000 cy:run",
    "test:e2e:open": "start-test serve 8000 cy:open"
  },
  "pre-push": [
    "clean",
    "build",
    "test"
  ],
  "devDependencies": {
    "@testing-library/jest-dom": "^5.11.1",
    "@testing-library/react": "^10.4.7",
    "babel-jest": "^26.1.0",
    "babel-preset-gatsby": "^0.5.4",
    "cypress": "^4.11.0",
    "identity-obj-proxy": "^3.0.0",
    "jest": "^26.1.0",
    "jest-chain": "^1.1.5",
    "jest-localstorage-mock": "^2.4.3",
    "jest-mock-axios": "^4.2.1",
    "pre-push": "^0.1.1",
    "prettier": "^2.0.5",
    "react-test-renderer": "^16.13.1",
    "sinon": "^9.0.2",
    "start-server-and-test": "^1.11.2"
  },
  "repository": {
    "type": "git",
    "url": "https://github.com/gatsbyjs/gatsby-starter-default"
  }
}

In my Cypress for the login page, the following code leads to the error:

it('should go to Home page if correct credentials are entered', () => {
      cy.visit("/app/login")
        .reload(true)
      cy.get(logout)
        .should('not.exist')
      cy.get(username)
        .click()
        .clear()
        .type(Cypress.env('TEST_USERNAME'))
      cy.wait(5000)
      cy.get(password)
        .click()
        .clear()
        .type(Cypress.env('TEST_PASSWORD'))
      cy.wait(5000)
      cy.get(button)
        .click()
      cy.wait(5000)
      cy.url().should('eq', 'http://localhost:8000/app/home')
      cy.get(logout)
        .should('be.visible')
        .click()
      cy.url().should('eq', 'http://localhost:8000/app/login')
    })

Expected result

AWS Cognito authentication should succeed.

Actual result

AWS Cognito authentication fails.

Environment

System:
    OS: macOS 10.15.6
    CPU: (4) x64 Intel(R) Core(TM) i5-7267U CPU @ 3.10GHz
    Shell: 5.7.1 - /bin/zsh
  Binaries:
    Node: 14.3.0 - ~/.nvm/versions/node/v14.3.0/bin/node
    Yarn: 1.22.4 - /usr/local/bin/yarn
    npm: 6.14.5 - ~/.nvm/versions/node/v14.3.0/bin/npm
  Languages:
    Python: 2.7.16 - /usr/bin/python
  Browsers:
    Chrome: 84.0.4147.105
    Safari: 13.1.2
  npmPackages:
    gatsby: ^2.13.45 => 2.24.7
    gatsby-image: ^2.2.7 => 2.4.13
    gatsby-plugin-manifest: ^2.2.4 => 2.4.19
    gatsby-plugin-offline: ^3.2.17 => 3.2.19
    gatsby-plugin-react-helmet: ^3.1.2 => 3.3.10
    gatsby-plugin-sharp: ^2.6.19 => 2.6.20
    gatsby-source-filesystem: ^2.1.7 => 2.3.20
    gatsby-source-graphql: ^2.1.3 => 2.6.2
    gatsby-transformer-sharp: ^2.5.11 => 2.5.11
  npmGlobalPackages:
    gatsby-cli: 2.12.60

[sidharthachatterjee]

This seems unrelated to Gatsby?

[cxk280]

I am not sure. It could definitely be a Cypress issue, but after careful consideration I feel that the preponderance of the evidence points toward Gatsby since the latter is ultimately handling the page rendering and therefore keeping the elements attributes in order. It is worth noting that, crucially, I have seen the mis-assigned element attributes at least once totally outside of Cypress, just in a normal Chrome window. It is the same elements mentioned in the issue. This strongly points toward Gatsby instead of Cypress.

I'd also like to point out that the elements with mis-assigned attributes are the ones with (at least initially) the same field names as the corresponding fields in the sign in page. That is, username and password have mis-assigned attributes, but email does not, as the user does not have to enter an email address to sign in and so they are only asked to fill that field at sign up. But trying to change the input elements names to something different, like signupUsername and signupPassword, did not resolve the issue.