Skip to content

Commit

Permalink
Update scraper.py
Browse files Browse the repository at this point in the history
  • Loading branch information
vicoliveres committed Mar 7, 2018
1 parent de4d95d commit ef78fbb
Showing 1 changed file with 13 additions and 0 deletions.
13 changes: 13 additions & 0 deletions scraper.py
Expand Up @@ -18,6 +18,19 @@
for byline in bylines:
record = { "byline" : byline.text } # column name and value
scraperwiki.sqlite.save(["byline"], record) # save the records one by one

# Find something on the page using css selectors
root = lxml.html.fromstring(html)
bylines = root.cssselect("a.story-header-title-link")

for title in titles:
print lxml.html.tostring(title)
print title.text.encode('utf-8')

for title in titles:
record = { "title" : title.text } # column name and value
scraperwiki.sqlite.save(["title"], record) # save the records one by one


# # Write out to the sqlite database using scraperwiki library
# scraperwiki.sqlite.save(unique_keys=['name'], data={"name": "susan", "occupation": "software developer"})
Expand Down

0 comments on commit ef78fbb

Please sign in to comment.