Add spider for Teresina/PI #53
Conversation
| @@ -0,0 +1,53 @@ | |||
| import dateparser | |||
There was a problem hiding this comment.
This file may not be here. It is not related to your PR that references only Teresina-PI spider.
| scraped_at=datetime.utcnow(), | ||
| ) | ||
|
|
||
| next_page_path = response.css(self.NEXT_PAGE_CSS).extract_first() |
There was a problem hiding this comment.
Use urljoin (https://doc.scrapy.org/en/latest/topics/request-response.html#scrapy.http.Response.urljoin) method from the response to handle relative links:
for next_page_url in response.css(self.NEXT_PAGE_CSS).extract():
yield Request(response.urljoin(next_page_url))
|
|
||
| allowed_domains = ['www.dom.teresina.pi.gov.br'] | ||
| name = 'pi_teresina' | ||
| start_urls = [f'{GAZETTE_URL}/lista_diario.php'] |
There was a problem hiding this comment.
As you are using urljoin to follow the pagination (see my comment below), I don't see the need for GAZETTE_URL variable. Just include the full URL in start_urls list.
There was a problem hiding this comment.
Done @rennerocha! 👍
|
@rennerocha @giovanisleite Done with the refactoring and suggestion 👍 |
|
@alfakini mark the change requests as resolved. @rennerocha, does it look good to you? |
|
This PR can be closed (Cc. @jvanz) |
Hey people,
In this PR we add the spider to collect data from Teresina/PI.