Interaction with ExVCR

[zamith]

I'm building a web scraper using Hound, and it works fine, but when trying to test it using exvcr to mock the responses the responses are pretty much empty (at least no interesting data). I'm not sure if this is an exvcr or hound issue, or even if it is an issue with the way I have them set up, but hopefully you can help me.

I have opened the same issue on the exvcr repo, with the relevant code.

[HashNuke]

@zamith exvcr is for http requests made by your http client. When you use Hound, the http requests are made by the browser being used to test.

Unless required, I would suggest that you write tests at a much lower level (parsing the html obtained etc).

Closing this issue.

[danhper]

@HashNuke After looking at the code, I think he is trying to mock the webdriver responses.

@zamith Why are you trying to do this? I am not sure it is really the way to go, could you describe your use case a little more precisely please.

[zamith]

I was thinking of writing integration level tests for this, since a call to one of the tasks can visit a few different pages. Basically I wanted to mock the responses but not have to do it by hand, have them be automatically generated. I guess I can grab the HTML from the page and create a custom cassette for that. Is that what you're suggesting?

[HashNuke]

@tuvistavie Thanks for pointing out. Just looked at @zamith's code.

@zamith I would suggest not trying to use exvcr on the webdriver responses. If you change the webdriver server (let's say from selenium to using chrome_driver directly), your cassettes will break :)

Yes, having the html of a page and writing tests for the parser is better. Or if it is definitely required, then just scrape the page for real during the test (assuming the site is a really popular site, I don't think they'll mind if you scrape a few pages a month when running tests).

[zamith]

I was trying to do that because each task can interact with multiple pages, and I couldn't think of a better way to do integration tests.

I guess that writing unit tests for parts of the parser is better than nothing. Thanks.