end-to-end tests

[travi]

we've had a few issues come up with recent changes that have been related to dependencies that are stubbed by the current isolated unit tests. i think the current tests are still valuable, but also having larger scoped tests that incorporate octokit behavior by exercising calls against the github api (probably stubbed using nock) could help us catch breaking changes that we are currently missing.

a few of the recent things that slipped through:

• Fix label methods from Octokit updates #143
• Fixed octokit/rest.j sv15.18 deprecations #140

[gr2m]

It's hard to do proper end-to-end tests with Probot right now, definitely an aspect that needs improvement. I'm curious what you come up with, maybe we can adapt Probot's API so make it easier.

nock mocks HTTP requests but does not send webhook event payloads. I always wanted to look into https://github.com/slackapi/steno for that, but didn't have a chance yet

GitHub

slackapi/steno

📼 Slack app testing companion - Record and Replay your HTTP requests, both incoming and outgoing - slackapi/steno

[travi]

i was only thinking of using nock where settings makes calls out to the github api, not for the initial webhook that triggers the run. am i thinking wrong that probot.receive() is enough for that? i was thinking something along the lines of the example in the probot docs.

i'll admit that i haven't gotten through this process yet with a probot app, but it seems like it should be pretty straight forward. now i am curious what i'll run into that you're referring to.

when i started down this path in the past, the one thing that i remember wishing was possible was to set the http status code that probot would return. beyond that, i dont remember any blockers.

[travi]

im hoping to spend some time on this soon. one thing that i've been debating is what to use as the test framework.

jest is obviously being used already on this project, so there could be a benefit to staying consistent with that. i'm not that familiar with jest, but its similar enough to mocha that i'm not concerned about that part.

however, on my own projects i like to use cucumber for larger scoped tests. the main reasons i like this approach is that i think the Given/When/Then interface fits large scoped tests well, but also because it is different from the tools typically used for more isolated unit-tests. I think there is some benefit to mental separation to remind developers that the testing goals are different as they are working with the various test types.

i recognize that the latter approach is probably more beneficial to longer term contributors than might be the case with an OSS project. i'm not tied to any particular tool, but i'm curious about thoughts on the value of a consistent tool vs. the mental reminder from tools that are targeted a bit differently. i think i'm leaning in the direction of using jest for consistency, but figured i'd ask the question to get more opinions.

@jwsloan @kevo1ution @gr2m thoughts?

[gr2m]

I think using jest will lower the barrier for contributions coming from Probot contributors in other projects.

Yet another approach could be to create a new github organization such as @probot-settings-test and then create new repositories in there for real end-to-end tests without any mocking whatsoever. I'm looking into something similar for semantic-release right now: semantic-release/semantic-release#1372. But that might create its own challenges because it's pretty much uncharted territory.

I think I tend towards using jest for now, too.

[travi]

Yet another approach could be to create a new github organization such as @probot-settings-test and then create new repositories in there for real end-to-end tests without any mocking whatsoever.

this touches on the fuzziness of terms like end-to-end or integration when it comes to testing.

my goal at this point is for the scope of this testing to from an incoming request (using .receive()) to as close to the real api call as possible w/o making a real call (stubbing the actual api request/response with nock) and back to the probot response (this part is a little fuzzy to me since i haven't tried much with this yet).

while i do see value to a larger scope an exercising the real api, i think the ROI is better if we stub the API calls for now. as you mention, there are some additional complexities to setting up a test org and making sure that org either has certain scenarios constantly configured or that its possible to set up those scenarios as part of the test logic. with the number of possible key integrations made by semantic-release, i think the payoff is probably better for the investment than it would be here at this stage. good for us to keep in mind, though.

I think using jest will lower the barrier for contributions

i was thinking about this piece being valuable

contributions coming from Probot contributors in other projects

but hadnt considered this specific angle. i agree that that raises the value further

[stale]

Is this still relevant? If so, what is blocking it? Is there anything you can do to help move it forward?