Added support for running E2E tests without seeded data #14285

nickytonline · 2021-07-20T03:13:40Z

What type of PR is this? (check all applicable)

Description

This makes changes to E2E testing infrastructure allowing folks to write tests that use seeded data (typical use case) or no seed data (atypical use case), e.g. admin onboarding.

Related Tickets & Documents

#14259

QA Instructions, Screenshots, Recordings

N/A This is testing infrastructure

UI accessibility concerns?

N/A This is testing infrastructure

Added/updated tests?

Yes
No, and this is why: N/A This is testing infrastructure, although I did add a test just so the new build node passes as there are no real tests for it yet.
I need help with writing tests

[Forem core team only] How will this change be communicated?

Will this PR introduce a change that impacts Forem members or creators, the
development process, or any of our internal teams? If so, please note how you
will share this change with the people who need to know about it.

I've updated the Developer Docs and/or
Admin Guide, or
Storybook (for Crayons components)
I've updated the README or added inline documentation
I will share this change in a Changelog
or in a forem.dev post
I will share this change internally with the appropriate teams
I'm not sure how best to communicate this change and need help
This change does not need to be communicated, and this is why not: please
replace this line with details on why this change doesn't need to be
shared

[optional] Are there any post deployment tasks we need to perform?

[optional] What gif best describes this PR or how it makes you feel?

…eds_e2e.rb

lib/tasks/e2e_seed.rake

nickytonline · 2021-07-20T03:20:58Z

Thinking about the seed file for admin onboarding @Ridhwana, and correct me if I'm wrong, but I don't think we need a seed file at all, just the initial DB state correct? If that's the case, it makes things even simpler.

Ridhwana · 2021-07-20T13:06:54Z

Thinking about the seed file for admin onboarding @Ridhwana, and correct me if I'm wrong, but I don't think we need a seed file at all, just the initial DB state correct? If that's the case, it makes things even simpler.

@nickytonline correct, just the initial db state is good for now.

…ue to seeds_e2e.rb" This reverts commit 6018ae7.

…ws folder.

…erent-seed-support

nickytonline

There are minimal changes in this PR. It looks like a lot of files, but it's because all current E2E tests have been moved to a seededFlows folder in Cypress' integration folder.

.travis.yml

config/initializers/cypress_rails.rb

cypress/integration/nonSeededFlows/test.spec.js

cypress/integration/seededFlows/loginFlows/userLogout.spec.js

cypress/integration/seededFlows/readingListFlows/readingList.spec.js

cypress/integration/seededFlows/readingListFlows/readingListArchive.spec.js

cypress/plugins/index.js

cypress/integration/seededFlows/loginFlows/userChangePassword.spec.js

nickytonline · 2021-07-20T23:06:26Z

Just looking into why the new E2E build node is failing even though the tests pass in Cypress. The error it's giving is related to the Bluepird Promise library, but I'm not sure why. And it's having difficulty with db:truncate_all. 🤔

====================================================================================================
  (Run Finished)
       Spec                                              Tests  Passing  Failing  Pending  Skipped  
  ┌────────────────────────────────────────────────────────────────────────────────────────────────┐
  │ ✔  nonSeededFlows/test.spec.js               69ms        1        1        -        -        - │
  └────────────────────────────────────────────────────────────────────────────────────────────────┘
    ✔  All specs passed!                         69ms        1        1        -        -        -  
(node:8819) Warning: a promise was created in a handler at internal/timers.js:461:21 but was not returned from it, see http://goo.gl/rRqMUw
    at Function.Promise.cast (/home/travis/.cache/Cypress/7.7.0/Cypress/resources/app/packages/server/node_modules/bluebird/js/release/promise.js:225:13)
(Use `Cypress --trace-warnings ...` to show where the warning was created)
Cleaning up and stopping server for end to end tests.
/home/travis/build/forem/forem/vendor/bundle/ruby/2.7.0/gems/rake-13.0.6/lib/rake/task_manager.rb:59:in `[]': Don't know how to build task 'db:truncate_all' (See the list of available tasks with `rake --tasks`) (RuntimeError)
	from /home/travis/build/forem/forem/vendor/bundle/ruby/2.7.0/gems/rake-13.0.6/lib/rake/task.rb:405:in `[]'
	from /home/travis/build/forem/forem/config/initializers/cypress_[secure].rb:28:in `block in <main>'
	from /home/travis/build/forem/forem/vendor/bundle/ruby/2.7.0/gems/cypress-[secure]-0.5.1/lib/cypress-[secure]/initializer_hooks.rb:38:in `block in run'
	from /home/travis/build/forem/forem/vendor/bundle/ruby/2.7.0/gems/cypress-[secure]-0.5.1/lib/cypress-[secure]/initializer_hooks.rb:37:in `each'
	from /home/travis/build/forem/forem/vendor/bundle/ruby/2.7.0/gems/cypress-[secure]-0.5.1/lib/cypress-[secure]/initializer_hooks.rb:37:in `run'
	from /home/travis/build/forem/forem/vendor/bundle/ruby/2.7.0/gems/cypress-[secure]-0.5.1/lib/cypress-[secure]/launches_cypress.rb:58:in `run_exit_hooks_if_necessary!'
	from /home/travis/build/forem/forem/vendor/bundle/ruby/2.7.0/gems/cypress-[secure]-0.5.1/lib/cypress-[secure]/launches_cypress.rb:43:in `block in set_exit_hooks!'
The command "SKIP_SEED_DATA=1 E2E_FOLDER=nonSeededFlows bin/e2e-ci" exited with 1.
cache.2
store build cache
Artifacts support disabled for pull requests
Done. Your build exited with 1.

… node." This reverts commit 0eb781a.

config/initializers/cypress_rails.rb

nickytonline · 2021-07-20T23:26:20Z

The only thing about these changes that I have on my mind is DX when folks need to write non-seeded E2E tests.

Are they OK running SKIP_SEED_DATA=1 E2E_FOLDER=nonSeededFlows bin/e2e from the command line for non-seeded E2E tests? That seems a bit clunky to me. We can definitely have another bin command that wraps all that, but not sure what to call it. bin/e2e-non-seeded? That name is terrible, lol.

djuber

I'm sure whatever DX pains we're going to have we'll understand better once we do have them.

nickytonline · 2021-07-21T19:06:35Z

I'm sure whatever DX pains we're going to have we'll understand better once we do have them.

It definitely can be done in a follow up PR once we exhibit the DX pains. 😎

rhymes · 2021-07-22T08:55:30Z

I'm waiting the US SRE team to be online to merge this because if we missed anything it'll block the CI pipeline for all :)

Ridhwana

🎉 Super excited for this! Thanks Nick, it looks good.

nickytonline · 2021-07-22T12:45:33Z

I'm going to go ahead and merge as we're almost 9am Eastern.

Added support for an E2E_SEED_FILE env var with a default value to se…

6018ae7

…eds_e2e.rb

pr-triage bot added the PR: draft bot applied label for PR's that are a work in progress label Jul 20, 2021

nickytonline commented Jul 20, 2021

View reviewed changes

lib/tasks/e2e_seed.rake Outdated Show resolved Hide resolved

nickytonline added 7 commits July 20, 2021 12:55

Revert "Added support for an E2E_SEED_FILE env var with a default val…

b50eb3e

…ue to seeds_e2e.rb" This reverts commit 6018ae7.

If SKIP_SEED_DATA env var is set, do not load seed data.

040c4b9

Separated E2E tests into seeded and non-seeded flows.

1cdac1d

Added support for choosing an E2E folder, defaulting to the seededFlo…

14ba7d0

…ws folder.

Added a non-seeded E2E tests build node.

cc1ba49

Merge remote-tracking branch 'origin/main' into nickytonline/e2e-diff…

48334b8

…erent-seed-support

Fixed a Jasmine test when it should have been chai based. 🙃

93ffb3a

nickytonline commented Jul 20, 2021

View reviewed changes

nickytonline changed the title ~~Added support for changing E2E seed file and test folders to run~~ Added support for runnin E2E tests without seeded data Jul 20, 2021

nickytonline marked this pull request as ready for review July 20, 2021 21:35

nickytonline requested a review from a team as a code owner July 20, 2021 21:35

pr-triage bot added PR: unreviewed bot applied label for PR's with no review and removed PR: draft bot applied label for PR's that are a work in progress labels Jul 20, 2021

nickytonline requested a review from Ridhwana July 20, 2021 21:35

nickytonline marked this pull request as draft July 20, 2021 21:40

pr-triage bot added PR: draft bot applied label for PR's that are a work in progress and removed PR: unreviewed bot applied label for PR's with no review labels Jul 20, 2021

Fixed glob for test files.

631cebe

nickytonline requested review from djuber and removed request for a team July 20, 2021 22:49

nickytonline marked this pull request as ready for review July 20, 2021 22:54

pr-triage bot added PR: unreviewed bot applied label for PR's with no review and removed PR: draft bot applied label for PR's that are a work in progress labels Jul 20, 2021

nickytonline changed the title ~~Added support for runnin E2E tests without seeded data~~ Added support for running E2E tests without seeded data Jul 20, 2021

nickytonline added 3 commits July 20, 2021 19:07

Temporarily removing all build nodes except the new E2E build node.

0eb781a

Revert "Temporarily removing all build nodes except the new E2E build…

e043fbb

… node." This reverts commit 0eb781a.

Fixed test crash for nonSeeded data flows.

bd05b60

nickytonline commented Jul 20, 2021

View reviewed changes

config/initializers/cypress_rails.rb Show resolved Hide resolved

Renamed build nodes based on PR feedback.

7dad856

djuber approved these changes Jul 21, 2021

View reviewed changes

pr-triage bot added PR: partially-approved bot applied label for PR's where a single reviewer approves changes and removed PR: unreviewed bot applied label for PR's with no review labels Jul 21, 2021

nickytonline added area: tests issues related to tests merge by any core This tag is a signal that you are okay with any core team member merging your code for you. labels Jul 22, 2021

citizen428 approved these changes Jul 22, 2021

View reviewed changes

rhymes approved these changes Jul 22, 2021

View reviewed changes

Ridhwana approved these changes Jul 22, 2021

View reviewed changes

pr-triage bot added PR: reviewed-approved bot applied label for PR's where reviewer approves changes and removed PR: partially-approved bot applied label for PR's where a single reviewer approves changes labels Jul 22, 2021

nickytonline merged commit 4858b75 into main Jul 22, 2021

nickytonline deleted the nickytonline/e2e-different-seed-support branch July 22, 2021 12:45

pr-triage bot added PR: merged bot applied label for PR's that are merged and removed PR: reviewed-approved bot applied label for PR's where reviewer approves changes labels Jul 22, 2021

nickytonline mentioned this pull request Jul 22, 2021

[15 min fix] Added support for the --no-seed argument to bin/e2e command #14307

Merged

15 tasks

nickytonline added this to DONE 🎉 in Current Cycle Work Jul 22, 2021

Ridhwana mentioned this pull request Jul 29, 2021

Minimal Seed Data for Admin Onboarding #14385

Merged

15 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Added support for running E2E tests without seeded data #14285

Added support for running E2E tests without seeded data #14285

nickytonline commented Jul 20, 2021 •

edited

nickytonline commented Jul 20, 2021 •

edited

Ridhwana commented Jul 20, 2021 •

edited

nickytonline left a comment

nickytonline commented Jul 20, 2021 •

edited

nickytonline commented Jul 20, 2021 •

edited

djuber left a comment

nickytonline commented Jul 21, 2021

rhymes commented Jul 22, 2021

Ridhwana left a comment •

edited

nickytonline commented Jul 22, 2021

Added support for running E2E tests without seeded data #14285

Added support for running E2E tests without seeded data #14285

Conversation

nickytonline commented Jul 20, 2021 • edited

What type of PR is this? (check all applicable)

Description

Related Tickets & Documents

QA Instructions, Screenshots, Recordings

UI accessibility concerns?

Added/updated tests?

[Forem core team only] How will this change be communicated?

[optional] Are there any post deployment tasks we need to perform?

[optional] What gif best describes this PR or how it makes you feel?

nickytonline commented Jul 20, 2021 • edited

Ridhwana commented Jul 20, 2021 • edited

nickytonline left a comment

Choose a reason for hiding this comment

nickytonline commented Jul 20, 2021 • edited

nickytonline commented Jul 20, 2021 • edited

djuber left a comment

Choose a reason for hiding this comment

nickytonline commented Jul 21, 2021

rhymes commented Jul 22, 2021

Ridhwana left a comment • edited

Choose a reason for hiding this comment

nickytonline commented Jul 22, 2021

nickytonline commented Jul 20, 2021 •

edited

nickytonline commented Jul 20, 2021 •

edited

Ridhwana commented Jul 20, 2021 •

edited

nickytonline commented Jul 20, 2021 •

edited

nickytonline commented Jul 20, 2021 •

edited

Ridhwana left a comment •

edited