-
-
Notifications
You must be signed in to change notification settings - Fork 388
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add spider for Teresina/PI #53
Conversation
@@ -0,0 +1,53 @@ | |||
import dateparser |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This file may not be here. It is not related to your PR that references only Teresina-PI spider.
scraped_at=datetime.utcnow(), | ||
) | ||
|
||
next_page_path = response.css(self.NEXT_PAGE_CSS).extract_first() |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Use urljoin
(https://doc.scrapy.org/en/latest/topics/request-response.html#scrapy.http.Response.urljoin) method from the response to handle relative links:
for next_page_url in response.css(self.NEXT_PAGE_CSS).extract():
yield Request(response.urljoin(next_page_url))
|
||
allowed_domains = ['www.dom.teresina.pi.gov.br'] | ||
name = 'pi_teresina' | ||
start_urls = [f'{GAZETTE_URL}/lista_diario.php'] |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
As you are using urljoin
to follow the pagination (see my comment below), I don't see the need for GAZETTE_URL
variable. Just include the full URL in start_urls
list.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Done @rennerocha! 👍
@rennerocha @giovanisleite Done with the refactoring and suggestion 👍 |
@alfakini mark the change requests as resolved. @rennerocha, does it look good to you? |
This PR can be closed (Cc. @jvanz) |
Hey people,
In this PR we add the spider to collect data from Teresina/PI.