Fix broken scrapers-ca scrapers

[jpmckinney]

Work through the broken scrapers at http://scrapers.herokuapp.com/

• ca_qc_mercier
• ca_qc_montreal_est

Ignore:

• jurisdictions ending in _municipalities
• ca_on_toronto (website frequently fails)

---

Previous issue content

Reserved for Matthew Leon:

• ca_mb

Priority (already in Represent):

• ca_nb_frederiction
• ca_nb_moncton
• ca_on_richmond_hill
• ca_qc_brossard
• ca_qc_senneville

[matthewleon]

FYI I am unable to duplicate the "list index out of range" errors on a bunch of these scrapers. ca_on_guelph, ca_on_markham, ca_on_richmond_hill, ca_qc_saint_jerome all complete without errors on my machine.

[matthewleon]

ca_qc_mercier page is 403'ing the scraper but loading fine in browser. Maybe some kind of user-agent sniffing going on? Will try to check later.

[matthewleon]

same with ca_qc_montreal_est

[jpmckinney]

Yeah, there are four scrapers that only fail on Heroku, as in the issue description. When you call lxmlize, you can pass a user_agent string. ca_pe_stratford uses a string for IE10.

[jpmckinney]

I fixed all the Heroku-only failures. They were mostly around the use of things like [2] in XPath. For ca_nb, for example, where it was picking the wrong image, on Heroku, //img[2] means "the second img within the same parent." Locally, it's interpreted as "the second img anywhere in the document."

[matthewleon]

Why is there this difference in interpretation? Is there a difference version of lxml running on heroku?

[jpmckinney]

The Python package is the same version; maybe the C code is different? I assume only one of the two interpretations is correct, though.

[matthewleon]

Indeed. This is very strange.

[Menerve]

The local interpretation is the good one according to http://www.w3.org/TR/1999/REC-xpath-19991116/

[jpmckinney]

@matthewleon I added user agent strings for ca_qc_mercier and ca_qc_montreal_est. The scrapers now fail for a different reason (pupa.scrape.base.ScrapeError: no objects returned from people scrape) likely because the selectors don't work anymore.