New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Full Network Layer Stubbing / fetch support #687

Open
brian-mann opened this Issue Sep 23, 2017 · 48 comments

Comments

@brian-mann
Member

brian-mann commented Sep 23, 2017

What users want

  • To be able to stub any kind of HTTP request including window.fetch
  • To blacklist certain domains (google analytics, new relic, intercom)
  • To have more control over dynamically routing / stubbing requests and responses
  • To more easily stub graphQL calls

What we do now

  • We modify the host XMLHttpRequest object which limits us to only stubbing XHR's from the main application frame.

What we need to do

  • Throw away the entire existing system for stubbing
  • Move stubbing into the internal Cypress network proxy layer (outside of the browser)
  • Synchronize the state of the browser tests with the backend (so its aware what needs to be stubbed / routed)
  • Remove cy.server and instead just use cy.route
  • Expand the API's for more advanced configuration
  • Accept blacklistDomains option for cypress.json
  • As each request comes in, if we expose callback functions we'll need to constantly communicate with the driver for guidance on what to do - whether to proceed - halt, etc. This can get pretty complicated and might have significant performance issues. We'll need to experiment and we may not be able to provide callback functions anymore.

Things to consider

  • onRequest and onResponse currently provide the raw level xhr object. This would no longer be possible. We'd have to serialize the request / response into basic objects. No method calling would be available.
  • The onError callback could no longer exist
  • Additionally we hook into the onerror events for XHR's and will automatically fail the test if this is called. This would no longer be possible.
  • Blacklisting https cannot be dynamic. It would have to be set in cypress.json or environment variables. The reason is that for each blacklisted domain that goes over https, we would have to route that traffic to a self signed certificate. Browser freak out if traffic to https gets assigned one certificate, and then within the same session it gets another cert applied. What this means is that before spawning the browser we have to isolate that traffic and know ahead of time not to route it to the actual server. This enables us to send a single cert to the browser.
  • We could enable non https hosts to be dynamically blacklisted, but these API's would be async and likely difficult for users to understand / use. We might be able to bake these into cy.route but I'm not sure there's much of a use case here of only dynamically blacklisting non https traffic.
  • We will no longer be able to tell the difference between an XHR and any other kind of http request. We'd likely have to change the page events to say REQ and REQ STUB instead of XHR and XHR STUB.
  • It would be very difficult to provide accurate onResponse callback functions. It's possible for the internal server to know when an HTTP request has been completed, but it's not possible to know whether the browser has actually finished processing it. This may or may not be a problem.
  • How would we handle stubbing binary / streaming responses?
  • How would we handle stubbing things like images? (This is likely pretty simple)

Bonus Points

  • It would likely be possible to stub Websocket traffic in the same manner
  • The problem with this is that most Websocket libraries (ie socket.io) add their own "encoding" pattern on top of each websocket message frame. The ergonomics around stubbing this would be odd. The problem is this is library (and even version of library) specific. This may be a better use case for cy.stub, or perhaps a 3rd party extension

@brian-mann brian-mann self-assigned this Sep 23, 2017

@brian-mann

This comment has been minimized.

Member

brian-mann commented Sep 24, 2017

@jennifer-shehane jennifer-shehane changed the title from Proposal: Full Network Layer Stubbing / Blacklisting / fetch support to Full Network Layer Stubbing / Blacklisting / fetch support Sep 25, 2017

@ericadamski

This comment has been minimized.

ericadamski commented Oct 10, 2017

@brian-mann I would be very willing to help out with this issue. This is a big blocker for use when testing our app.

@brian-mann

This comment has been minimized.

Member

brian-mann commented Oct 10, 2017

@ericadamski could you go into more detail about this? What exactly is a big blocker?

@ericadamski

This comment has been minimized.

ericadamski commented Oct 10, 2017

@brian-mann making network requests (using a fetch polyfill) and asserting on those requests is important to our integration testing. Currently we are just spying on the window.fetch and doing some simple assertions on the params request. It would be ideal to provide the same functionality as XHRs to completely omit the server and mock responses when needed, understanding that we could stub the fetch as well, but this means stubbing each individual request with a tailored response.

@brian-mann

This comment has been minimized.

Member

brian-mann commented Oct 10, 2017

You can do this right now using stubs. Albeit it's certainly not as nice as what we did for XHR stubbing, but it is possible and we have several examples of making this work. This is even in a new getting started video we're releasing later this week.

https://github.com/cypress-io/cypress-example-recipes/blob/master/cypress/integration/spy_stub_clock_spec.js

@ericadamski

This comment has been minimized.

ericadamski commented Oct 10, 2017

Yes we realize it can be done, albeit we would rather wait (or work toward) a more XHR style stubbing, it is just super nice.

@brian-mann

This comment has been minimized.

Member

brian-mann commented Oct 10, 2017

Yeah it is. Okay just making sure we're on the same page.

The proposal here does a decent job outlining what needs to be done but it will be a significant amount of work.

Likely the next steps just need to be formalizing a plan for the API's themselves. There have been many requests for additional features and we should start there.

@ericadamski

This comment has been minimized.

ericadamski commented Oct 10, 2017

That sounds good.

Is it worth creating separate issues for some of the requested features for I know almost nothing about graphQL queries 😜

@brian-mann

This comment has been minimized.

Member

brian-mann commented Oct 10, 2017

They mostly already are other issues. We need to factor in all of them and come up with a set of API's that unifies those different use cases and desired experiences.

@ericadamski

This comment has been minimized.

ericadamski commented Oct 10, 2017

Right OK, I see what you are saying.

@ericadamski

This comment has been minimized.

ericadamski commented Oct 10, 2017

@brian-mann side note, has it been thought of moving away from coffeescript and into es6?

@paulpruteanu

This comment has been minimized.

paulpruteanu commented Dec 3, 2017

Do you consider this feature in the next release? I reckon it would be highly valuable, for instance when

  • authentication/ other actions are performed at script level (<script src/> instead of XHR)
  • needing to bypass other scripts/ pixel images which takes a lot of time to load
@brian-mann

This comment has been minimized.

Member

brian-mann commented Dec 3, 2017

Yeah blacklisting specific requests is part of this feature roadmap

orlandohohmeier added a commit to dcos/dcos-ui that referenced this issue Dec 29, 2017

fix(test): add flags to identify and stub Mesos requests
Add flags (query parameters) to identify and stub Mesos Operator API
requests to fix the integration tests that were otherwise failing or
flaky due to parsing errors in the `MesosStateStore`. The store failed
to parse the `GET_MASTER` response as we could only register one stub
for both the `GET_MASTER` and `SUBSCRIBE` requests resulting in JSON
parse errors due to the different response formats (RecoredIO<JSON>,
JSON).

The flags are necessary as Cypress can only stub requests based on the
URL and request method and doesn't provide the means to dynamically stub
requests based on the payload as documented in
cypress-io/cypress#521. We will remove the
flags once the Cypress team rewrote the network handling and added
support for dynamic stubs as documented in
cypress-io/cypress#687 or we've  addressed
https://jira.mesosphere.com/browse/DCOS_OSS-2021.

orlandohohmeier added a commit to dcos/dcos-ui that referenced this issue Dec 29, 2017

fix(test): add flags to identify and stub Mesos requests
Add flags (query parameters) to identify and stub Mesos Operator API
requests to fix the integration tests that were otherwise failing or
flaky due to parsing errors in the `MesosStateStore`. The store failed
to parse the `GET_MASTER` response as we could only register one stub
for both the `GET_MASTER` and `SUBSCRIBE` requests resulting in JSON
parse errors due to the different response formats (RecoredIO<JSON>,
JSON).

The flags are necessary as Cypress can only stub requests based on the
URL and request method and doesn't provide the means to dynamically stub
requests based on the payload as documented in
cypress-io/cypress#521. We will remove the
flags once the Cypress team rewrote the network handling and added
support for dynamic stubs as documented in
cypress-io/cypress#687 or we've  addressed
https://jira.mesosphere.com/browse/DCOS_OSS-2021.

orlandohohmeier added a commit to dcos/dcos-ui that referenced this issue Dec 29, 2017

fix(test): add flags to identify and stub Mesos requests
Add flags (query parameters) to identify and stub Mesos Operator API
requests to fix the integration tests that were otherwise failing or
flaky due to parsing errors in the `MesosStateStore`. The store failed
to parse the `GET_MASTER` response as we could only register one stub
for both the `GET_MASTER` and `SUBSCRIBE` requests resulting in JSON
parse errors due to the different response formats (RecoredIO<JSON>,
JSON).

The flags are necessary as Cypress can only stub requests based on the
URL and request method and doesn't provide the means to dynamically stub
requests based on the payload as documented in
cypress-io/cypress#521. We will remove the
flags once the Cypress team rewrote the network handling and added
support for dynamic stubs as documented in
cypress-io/cypress#687 or we've  addressed
https://jira.mesosphere.com/browse/DCOS_OSS-2021.
@bahmutov

This comment has been minimized.

Collaborator

bahmutov commented Jun 12, 2018

work in progress

@akapitoha

This comment has been minimized.

akapitoha commented Jun 20, 2018

Hi,
Sorry for possibly duplicated question, but with this feature will it be possible to work (wait, check response body) with Websocket events? thank you

@tdaniely

This comment has been minimized.

tdaniely commented Jul 12, 2018

Do you have an ETA on this?

@yuki-93

This comment has been minimized.

yuki-93 commented Sep 24, 2018

Hey guys,

is there any information, when this feature is going to be ready?

I have an issue when I use cy.server and cy.request as first operation in before() or it() blocks. There are sometimes test runs, where the request seems to be not to trigger. It runs into the requestTimeout but I can see the request in the developer tools. When I reload the spec, than it mostly runs without any problemens. But this is impossible for the CI server.

For me, it looks like the listeners listentens to late on my requests. Is this gona fixed with this issue too?

@Vages

This comment has been minimized.

Vages commented Oct 24, 2018

I see that this package has not been mentioned here. It has solved a lot of my team's problems: https://github.com/ijpiantanida/talkback

@yuki-93

This comment has been minimized.

yuki-93 commented Oct 24, 2018

@Vages do I understand correctly, that talkback is replacement for my "original" / production server? Or is is possible to forward requests from cypress over talkback to my native apache?

Thanks for your reply

@Vages

This comment has been minimized.

Vages commented Oct 24, 2018

Talkback is a proxy which stores the last response for a given URL. Spin up one or more talkback-servers for the APIs you want to mock via cypress/plugins/. When in the test environment, set your frontend to fetch/get data from these URLs instead of the regular URLs.

@joaquinmoreira

This comment has been minimized.

joaquinmoreira commented Oct 24, 2018

I'm using talkback also, working really good!

@timpur

This comment has been minimized.

timpur commented Oct 30, 2018

I mentioned this here #95 (comment) but i wonder why PollyJS wouldnt be a perfect fit, it can do what cypress wants and more and already offers adaptors and other means to hook in. I use it with cypress and its amazing. Would love to see it become part of cypress :). I think it was meant to be :P

Also my 2 cents, Think you should leave the stubbing in the client as this gives you the best flexibility and control, other wise syncing mock settings will be required and adds more complexity. Additionally stubbing in the client renders the fastest response, which is nice for testing purposes. Having stubs that return dynamic content based of your test suite might be tricky as you have to send the response to the server so it can respond to your app which is already in client ???

I feel your mocks should originate from your suite since whats where its executed from, and since this is in the client it feels clunky to push this config to the server and have it respond from there back to your client.

Just use Polly, has the backing and its already developed :P

@aaronmcadam

This comment has been minimized.

aaronmcadam commented Oct 30, 2018

Could we add a recipe for using https://github.com/ijpiantanida/talkback for HTTP stubbing?

@pvawser01

This comment has been minimized.

pvawser01 commented Nov 15, 2018

@Vages Are you able to provide some detail as to how to add Talkback to cypress. I'm getting a bit stuck trying.

@Vages

This comment has been minimized.

Vages commented Nov 15, 2018

Start a Talkback server in cypress/plugins/index.js. It will run automatically when Cypress starts. An example inspired by the Talkback Readme:

// cypress/plugins/index.js
const talkback = require("talkback");
 
const opts = {
  host: "https://api.myapp.com/foo",
  port: 5544,
  path: "./my-tapes"
};

module.exports = () => {
  const server = talkback(opts);
  server.start(() => console.log("Talkback Started"));
}

How this works: When a request is sent to localhost:5544, Talkback will look for a tape (previous request and response) matching the request, looking at the URL, headers, body, and all. If a matching tape is found, Talkback will respond with the stored response. If not, Talkback will forward the request to https://api.myapp.com/foo, storing the request and response as a tape. If you send a request to localhost:5544/bar, it will be forwarded to https://api.myapp.com/foo/bar and so on.

Common pitfall: I have seen people going "Why isn't Talkback storing any tapes?" because they are sending their requests "production style" during testing, e.g. https://api.myapp.com/foo/baz. Sadly, Talkback isn't magic (like Cypress): When you run your tests, you have to make your app request the localhost URL that the Talkback server is attached to. For example localhost:5544/baz instead of https://api.myapp.com/foo/baz. You've probably got a way of changing API base URLs from development to production already, so just make a test configuration that you use when running with Cypress. But it's likely very project specific, so you have to do that homework yourself.

@pvawser01

This comment has been minimized.

pvawser01 commented Nov 15, 2018

Thanks @Vages. That was a very complete explanation. Thanks for your effort. Clears things up for me.

@psachs21

This comment has been minimized.

Contributor

psachs21 commented Nov 19, 2018

I've been trying to get XMLHttpRequest mocking going using pollyjs and the basics have been working (after replacing the xhr adapter with one that works with cypress). I am seeing issues in some cases though, a maximum call stack issue keeps happening. Trying to track it down. Don't know if anyone else has had any luck with pollyjs.

@timpur

This comment has been minimized.

timpur commented Nov 19, 2018

@psachs21 i created this till i have a better solution https://github.com/timpur/cypress-polly-xhr-adapter (sinonjs/nise#70 (comment))

Edit: Just add the git repo url to package.json and it should just work (install)

@psachs21

This comment has been minimized.

Contributor

psachs21 commented Nov 19, 2018

@timpur This is exactly what I was looking for. Figures someone had already solved it. Much appreciated (even as a temporary solution)

@PinkyJie

This comment has been minimized.

PinkyJie commented Nov 20, 2018

For anyone who is interested in auto stub based on existing cypress, I've created a simple example to demonstrate my solution https://github.com/PinkyJie/cypress-auto-stub-example

@jennifer-shehane

This comment has been minimized.

Member

jennifer-shehane commented Nov 29, 2018

There has been some progress made on this issue - but it does involve many parts, so we do not have an ETA at the moment.

@jemerald

This comment has been minimized.

jemerald commented Dec 6, 2018

With the proxy approach, am I correct to say that it will not work for API with Kerberos authentication?

It would be great if the current browser side stubbing option is kept as an alternative.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment