Test Scenarios #1465

cannikin · 2020-11-10T22:56:46Z

After a good couple of hours working with @peterp and @mojombo we think we came up with a pretty clean way to integrate scenarios (formerly "fixtures") into the test workflow! This PR has a bunch of backstory and reads like a thriller so grab a warm drink and snuggle up with a blanket!

Definitions

scenario — data that is pre-set into the database at the start of your test, and removed afterwords

The Problem

Services are generally going to be accessing your database to retrieve/create/update/delete data. In most cases you'll want some data in the database at the start of the test. You could insert this data yourself:

it('returns a list of posts', async () => {
  const posts = [
    { title: 'First post',  body: 'Foobar' },
    { title: 'Second post',  body: 'Bazqux' }
  ]
  for (const post in posts) {
    await db.post.create({ data: post })
  }

  const dbPosts = db.post.findMany()
  expect(dbPosts.length).toEqual(posts.length)

  await db.post.deleteMany()
})

8 of the 10 lines in this test (80%) is just putting data into the database and cleaning up afterwords! And if you have another test that wants to use this data you have to write all the setup again. Also, the data in the posts variable is just the plain object that I created, not the actual data in the database which will include additional fields like id or createdAt which are set to default values in the database itself.

So the next logical step is to just put the data insertion into a beforeEach() and the cleanup into an afterEach() and actually return the data as it appears in the database:

const POSTS = [
  { title: 'First post',  body: 'Foobar' },
  { title: 'Second post',  body: 'Bazqux' }
]

const posts = []

beforeEach(() => {
  for (const post in POSTS) {
    posts.push(await db.post.create({ data: post }))
  }
})

afterEach(() => {
  await db.post.deleteMany()
})

it('returns a list of posts', async () => {
  const dbPosts = db.post.findMany()
  expect(dbPosts.length).toEqual(posts.length)
})

At least the test (in the it() function) is now dealing with just the test data, which is an improvement. But I'd still have to write these beforeEach and afterEach functions for each services test and their content is essentially the same:

Insert some data
Delete some data

The next logical step is to have Redwood do the insertion and cleanup for me. That's where this PR comes in.

The Solution

Here's what we came up with. What if the test suite, the entire thing, could look like this:

import { comments } from './comments'

describe('comments', () => {
  it('returns a list of comments', async (scenario) => {
    const list = await comments()
    expect(list.length).toEqual(scenario.comment.length)
  })
})

Where the scenario argument is the sample data you inserted into the database. Let Redwood worry about seeding it and removing it afterwords. Where does the actual scenario data live? Just like the mocks solution we came up with for stories, it lives in its own file, in this case comments.scenarios.ts:

export const standard = {
  comment: {
    rob: {
      name: 'Rob',
      body: 'This is my comment',
      post: {
        create: {
          title: 'First Post',
          body: 'Lorem ipsum',
        },
      },
    },
    tom: {
      name: 'Tom',
      body: 'Also a great comment',
      post: {
        create: {
          title: 'Second Post',
          body: 'Dolar sit amet',
        },
      },
    },
  },
}

The nested structure contains:

comment - model name
- rob - name of the fixture
  - name, body, etc. - the actual data that will be inserted into the comment model

You can insert data into nested models by using either the prisma create syntax (as in the example above) or declaring another model at the first level of the data structure:

export const standard = {
  comment: {
    rob: {
      // ...
    },
  },
  post: {
    first: {
      title: 'First Post',
      body: 'Lorem ipsum...'
    },
  },
}

When you get your scenario object in your test, it's an object with the same structure as what you created above, but after it's been inserted in the database, so it includes data in the fields that the database itself sets, like id and createdAt:

scenario.comment.rob.name
scenario.comment.rob.id
scenario.comment.rob.createdAt

To get this working, we create our own test function that:

Adds the scenario data to the database
Runs the test
Removes the records that were created

We do this in our custom function named scenario():

scenario('returns a list of comments', async (scenario) => {
  const list = await comments()
  expect(list.length).toEqual(scenario.comment.length)
})

Just like with the component mocks, we default to a scenario named standard. As long as you export that in your scenarios file, we'll use it by default. If you want to use another named scenario, export it from your *.scenarios.js file and pass that name, as a string, as the first argument to scenario():

scenario('withoutPosts', 'returns a list of comments', async (scenario) => {
  const list = await comments()
  expect(list[0].post).toEqual(null)
})

This lets us preserve the existing it() and test() functions and make it clear to users that if you want to use fixtures, use our new function instead.

Little history: the naming comes from a Ruby gem that @mojombo wrote years ago for Rails called fixture-scenarios.

Todo

Update the generators to include scneario files and tests that use it
VS Code hints aren't quite working with scenario...there were a couple changes made to some index.d.ts, global.d.ts and eslint-config files with Peter's help but it didn't fix the issue. He was going to look into that some more.

…s second parameter to test function

cannikin · 2020-11-14T00:34:55Z

@peterp After a couple of extensive coding sessions with @mojombo we made some updates! Trying to override it and test was causing all kinds of problems with async tests so we said screw it and just went with our own. We just named it scenario() and it accepts an optional first parameter, which is the name of the scenario to use, otherwise it defaults to one called "standard". I updated the code snippets above to use the function name.

This leaves out the typescript stuff though. The idea was that when declaring your scenario objects in *.fixtures.js you'd call a function instead of just create an object, and that function would worry about the types. Unless we were mistaken, Tom and I both thought we'd need to somehow re-generate the types on every prisma generate in case the user changed something about the schema somewhere. We sidestepped that problem and hoped you would have some idea of that would work. 😬

import { scenario } from '@redwoodjs/testing'

export const standard = scenario({
  comment: {
    rob: {
      name: 'Rob'
      // ...
    }
  }
})

peterp · 2020-11-16T12:34:30Z

@cannikin This is look great, I think I can dynamically create those prisma model types at run-time. So we would default to the standard scenario, but a user could export multiple scenarios and pick the scenario to run?

Could you include a comment of how to use the new function?

cannikin · 2020-11-16T17:51:47Z

So we would default to the standard scenario, but a user could export multiple scenarios and pick the scenario to run?

Yup! The scenario takes an optional first argument. Which seems weird, I know, but after @mojombo and I tried a bunch of different combinations, the first position had the most natural feel to it: the name was right next to the word "scenario" and conveys understanding better.

The two ways to call the method:

Without a scenario name, which makes it a drop-in replacement for it or test that will use the standard scenario
With a scenario name as the first argument, which is the same name as an export in the *.fixtures.js file

// users.fixtures.js

export const standard = {
  users: {
    rob: {
      name: 'Rob Cameron'
    }
  }
}

export const withProfiles = {
  users: {
    rob: {
      name: 'Rob Cameron',
      profile: {
        create: {
          username: 'cannikin'
        }
      }
    }
  }
}

// users.test.js

import { users } from './users'

// standard scenario
scenario('returns a list of users', async (fixtures) => {
  const usersList = await users()
  expect(usersList[0].name).toEqual(fixtures.users.rob.name)
})

// named scenario
scenario('withProfiles', 'returns a list of users and their profiles', (fixtures) => {
  const usersList = await users({ includeProfiles => true })
  expect(usersList[0].profile.username).toEqual(fixtures.users.rob.profile.username)
})

cannikin · 2020-11-16T21:53:15Z

@peterp I wasn't sure how to construct the type in https://github.com/redwoodjs/redwood/pull/1465/files#diff-dc00f5226046500fcbb89faee6cb2093feffa41a14bd9e644064267b4dd1818d since the second parameter could be a function OR a string, and if it's a string then the third parameter must be present, and be a function. TYPESCRIPT

cannikin · 2020-11-16T21:54:56Z

Maybe this code should go into a separate JS file somewhere, instead of everything being crammed into jest.setup.js?

github-actions · 2020-11-16T21:57:32Z

Install this PR by running yarn rw upgrade --pr 1465:0.21.0-d018ffb

Tobbe · 2020-11-16T22:28:16Z

@cannikin Don't know if it applies here, but generally, what you're asking would be solved like this

function foo(first: string, second: () => void): void;
function foo(first: string, second: string, third: () => void): void {
  // implementation
}

cannikin · 2020-11-17T18:32:01Z

@peterp I updated the eslint-config stuff to add the scenario function but I get that same any popup when VS Code tries to give a hint:

cannikin · 2020-11-17T21:52:47Z

Converted back to draft! Talked with @mojombo and we're going to drop the "fixtures" nomenclature and just stick with "scenarios." Jumping back and forth was getting confusing and I found myself making an aside block in the Tutorial II to try and explain the difference myself and having trouble.

So the files will become *.scenarios.js and the argument you pass to the scenario() function will also be called scenario so you can reference the fields inside as scenario.comment.rob.name. One word to rule them all!

peterp · 2020-11-18T04:29:07Z

@cannikin Let me know when you want me to try add some types to this!

cannikin · 2020-11-18T05:08:42Z

@peterp You should be able to go ahead? I'm working on the generator right now so that it creates the *.scenarios.js file along with the service. But the object I'm creating will match the types that Prisma creates, so you should be able to create that function that'll wrap the scenario JSON (also called scenario() I think?). In theory, the *.scenario.js files would end up looking something like:

import { scenario } from '@redwoodjs/scenario'

export const standard = scenario({
  post: {
    first: {
      title: 'First Post',
      body: 'Lorem ipsum dolar sit amet',
    },
  },
})

I think?

…rc-text-fixtures

…eate comment

…many)

cannikin · 2020-12-11T22:39:39Z

Holy cow @peterp I got Storybook to not blow up with hasRole()! See 8460e77

I don't understand why only the parts exported from AuthContext exist within Storybook, while everything in AuthContextInterface is what's actually available from useAuth() in the context of a real app?

Here's a console log of useAuth() within a story:

Here's the same output from the dev server:

…doesn't like it

cannikin · 2020-12-17T20:50:57Z

Oh boy @peterp you're either going to love or hate this one. ;)

After several hours of working with @mojombo, @Tobbe, @dthyresson and @clairefro we were able to mock the currentUser in a story!

mockCurrentUser({
  email: 'test@example.com',
  roles: ['moderator'],
})

Just insert that at the top of a story and you're good to go.

How we got that working...it's either genius or insanity. We wrapped the existing mock providers in an <AuthProvider> along with a new mock auth client because the fakeUseAuth didn't seem to do anything: https://github.com/redwoodjs/redwood/pull/1465/files#diff-a2640d09eed5d305774c16eadddcab67e7e61d3ed209a2c879f9b1359c777ce5

We mock the currentUser to be null in StorybookProvider and then if you call mockCurrentUser yourself it'll get overridden with whatever you want.

There was some weirdness with declaring a global that wasn't a const... typescript made us use var, but eslint doesn't like that, so we added an ignore: https://github.com/redwoodjs/redwood/pull/1465/files#diff-dc00f5226046500fcbb89faee6cb2093feffa41a14bd9e644064267b4dd1818d

I can walk you through the other stuff we did, let me know! I know you said you were looking into mocking currentUser and it looked pretty easy, so maybe your way was way more elegant than this, but it was a fun exercise. :)

packages/cli/src/commands/generate/service/service.js

packages/core/config/babel-preset.js

packages/core/config/webpack.common.js

packages/core/src/configs/browser/jest.setup.js

packages/testing/src/global.ts

peterp · 2021-01-06T06:35:16Z

packages/core/src/storybook/StorybookProvider.tsx

@@ -35,5 +35,8 @@ export const StorybookProvider: React.FunctionComponent<{
    return null
  }

+  // default to a non-existent user at the beginning of each story
+  mockCurrentUser(null)


I usually like to be explicit for things like this.

Suggested change

mockCurrentUser(null)

window.mockCurrentUser(null)

Just so that people reading the code know that this is a global.

Hmm, problem. Putting window. or global. in front of that function throws an error at build time:

@redwoodjs/core: src/storybook/StorybookProvider.tsx(39,10): error TS2339: Property 'mockCurrentUser' does not exist on type 'Window & typeof globalThis'.

I'm assuming you're offline for the day and we wanna get 0.22 out, so I'll keep this as just mockCurrentUser(null) for now and we can revisit!

packages/core/src/configs/node/jest.setup.js

peterp

I left a few comments about making the code easier to debug for the user, but it looks good to me!

This reverts commit 424ca60.

…h-tokens * 'main' of github.com:redwoodjs/redwood: Move whatwg-fetch from devDep to dep Adds mockCurrentUser() to api-side jest v0.22.1 v0.22.0 Revert "Configure ApolloProvider to always use fresh tokens (redwoodjs#1609)" (redwoodjs#1611) Configure ApolloProvider to always use fresh tokens (redwoodjs#1609) Ignore *.scenarios.* files. (redwoodjs#1607) upgrade prisma v2.12.1 (redwoodjs#1604) Test Scenarios (redwoodjs#1465) Use relative path to config stories location (redwoodjs#1509)

cannikin added 3 commits November 10, 2020 12:36

Import global

0a0a766

Adds testWithFixtures global function

8fe4579

Adds new fixture functions to type defs

ab10139

cannikin mentioned this pull request Nov 10, 2020

improve generated Service test template #1224

Closed

Update to allow named fixtures and scenarios, pass name of scenario a…

57eb180

…s second parameter to test function

dthyresson added the topic/testing label Nov 16, 2020

cannikin added 2 commits November 16, 2020 13:50

Refactor into scenario() function, setup/teardown looking good

9da4fd2

Add eslint hint

72c44a1

Merge branch 'main' into rc-text-fixtures

c3b764a

cannikin marked this pull request as ready for review November 16, 2020 21:53

cannikin requested a review from peterp November 16, 2020 21:54

cannikin marked this pull request as draft November 17, 2020 21:50

cannikin changed the title ~~Test Fixtures~~ Test Scenarios Nov 17, 2020

cannikin added 6 commits November 17, 2020 21:10

Rename fixture -> scenario

b1e9ad2

Merge branch 'rc-text-fixtures' of github.com:redwoodjs/redwood into …

27df756

…rc-text-fixtures

WIP Infinite loop when comment tries to create post which tries to cr…

3fd55be

…eate comment

Comments

130b61a

Avoid infinite loop by ignore relations that aren't required (one-to-…

bab3a5a

…many)

Updating existing tests for new scenario file count

85ca0fc

Adds hasRole() to AuthContext

8460e77

cannikin added 2 commits December 17, 2020 12:03

Get mocked currentUser working in Storybook!

ed80a6d

Have to use var in declaration since we're setting it, but es-lint …

a5bdde1

…doesn't like it

Use data object for currentUser instead of global

92d411c

peterp reviewed Jan 5, 2021

View reviewed changes

packages/testing/src/global.ts Outdated Show resolved Hide resolved

cannikin added 7 commits January 5, 2021 13:20

Convert asyncForEach to for/of loop

d220784

Remove unnecessary mockedCurrentUser

6912a33

Adds defineScenario() function

8e8d658

Remove global for scenario

3d2915d

Merge branch 'main' into rc-text-fixtures

667907d

Remove debugging message

4ffb383

Add defineScenario() wrapper around generated scenarios

156b88b

peterp reviewed Jan 6, 2021

View reviewed changes

packages/core/src/configs/node/jest.setup.js Outdated Show resolved Hide resolved

peterp reviewed Jan 6, 2021

View reviewed changes

packages/core/src/configs/node/jest.setup.js Outdated Show resolved Hide resolved

peterp approved these changes Jan 6, 2021

View reviewed changes

cannikin and others added 5 commits January 6, 2021 10:14

Explicitly call window on default mockCurrentUser()

424ca60

Throw error if scenario() is called with the wrong number of arugments

0bf1b3d

Only swallow error when scenario file not found

8f85f08

Revert "Explicitly call window on default mockCurrentUser()"

75ef59f

This reverts commit 424ca60.

Merge branch 'main' into rc-text-fixtures

d018ffb

cannikin merged commit 6823af3 into main Jan 6, 2021

cannikin deleted the rc-text-fixtures branch January 6, 2021 20:59

thedavidprice added this to the Next Release milestone Jan 6, 2021

thedavidprice mentioned this pull request Jan 7, 2021

"Test Scenarios" results in error after scaffold generator is run #1606

Closed

thedavidprice mentioned this pull request Jan 13, 2021

Web Tests error in v0.22.1 #1633

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Test Scenarios #1465

Test Scenarios #1465

cannikin commented Nov 10, 2020 •

edited

Loading

cannikin commented Nov 14, 2020 •

edited

Loading

peterp commented Nov 16, 2020 •

edited

Loading

cannikin commented Nov 16, 2020

cannikin commented Nov 16, 2020

cannikin commented Nov 16, 2020

github-actions bot commented Nov 16, 2020 •

edited

Loading

Tobbe commented Nov 16, 2020

cannikin commented Nov 17, 2020

cannikin commented Nov 17, 2020

peterp commented Nov 18, 2020

cannikin commented Nov 18, 2020

cannikin commented Dec 11, 2020

cannikin commented Dec 17, 2020

peterp Jan 6, 2021

peterp Jan 6, 2021

cannikin Jan 6, 2021

cannikin Jan 6, 2021

peterp left a comment

Test Scenarios #1465

Test Scenarios #1465

Conversation

cannikin commented Nov 10, 2020 • edited Loading

Definitions

The Problem

The Solution

Todo

cannikin commented Nov 14, 2020 • edited Loading

peterp commented Nov 16, 2020 • edited Loading

cannikin commented Nov 16, 2020

cannikin commented Nov 16, 2020

cannikin commented Nov 16, 2020

github-actions bot commented Nov 16, 2020 • edited Loading

Tobbe commented Nov 16, 2020

cannikin commented Nov 17, 2020

cannikin commented Nov 17, 2020

peterp commented Nov 18, 2020

cannikin commented Nov 18, 2020

cannikin commented Dec 11, 2020

cannikin commented Dec 17, 2020

peterp Jan 6, 2021

Choose a reason for hiding this comment

peterp Jan 6, 2021

Choose a reason for hiding this comment

cannikin Jan 6, 2021

Choose a reason for hiding this comment

cannikin Jan 6, 2021

Choose a reason for hiding this comment

peterp left a comment

Choose a reason for hiding this comment

cannikin commented Nov 10, 2020 •

edited

Loading

cannikin commented Nov 14, 2020 •

edited

Loading

peterp commented Nov 16, 2020 •

edited

Loading

github-actions bot commented Nov 16, 2020 •

edited

Loading