Table of contents

• About
• External Setup
• Deployment

About

Blog Post

Video

Utilizes Python Requests and Beautiful Soup libraries to scrape Craigslist postings by keyword search. Listings are written to CSV, and posted to a Slack channel with Slack’s API. A text message is sent using Twilio's REST API if the listing is located in the user's city. Deployed on Heroku’s Cloud service. Note: Per my conversation with Slack's support team, there is a bug where Slackbot posts with unfurled images show as "edited". Their team is currently working on getting this fixed.

External Setup

• Slack workspace
• An App on workspace
• Add chat:write links:write OAuth Token scopes under Add Features and Functionality --> Permissions:
  • 
• Install App to workspace
• Create Channel for bot to post into
• Add bot to channel:
  • 
• Store Slack API in secrets.sh file:
  • export SLACK_API_TOKEN='INSERT-TOKEN-HERE'
• Twilio REST API

Running Craigslist-Ropot locally on your computer

1. Clone repository:

$ git clone https://github.com/sshantel/craigslist-ropot

2. Install dependencies:

$ pip install -r requirements.txt

3. For a scraping of items outside of planters in the SF Bay Area, region and term in line 204 of scraper.py will need to be updated to fit your desired needs. The URL in line 62 can be adjusted according to the Craigslist link you wish to scrape.

4. Source the Slackbot and Twilio API by running the following command:

$ source secrets.sh

5. Final command to run Craigslist-Ropot locally:

$ python3 scraper.py

Listings will post on the desired Slack channel if such listings in your Craigslist region with keywords exist.

Deployment

1. Download and Install Heroku

2. Store the Slack and Twilio API using the following commands

heroku config:set SLACK_API_TOKEN='INSERT-TOKEN-BETWEEN-THESE-SINGLE-QUOTES'

heroku config:set twilio_api='INSERT-TOKEN-BETWEEN-THESE-SINGLE-QUOTES'

heroku config:set twilio_auth='INSERT-TOKEN-BETWEEN-THESE-SINGLE-QUOTES'

3. Store personal phone number

heroku config:set my_phone_number='INSERT-PHONE-NUMBER-BETWEEN-THESE-SINGLE-QUOTES'

4. Adjust Heroku timezone according to your location(otherwise it defaults to UTC)

$ heroku config:add TZ="America/Los_Angeles"

5. Add and commit files

$ git add .

6. Scale worker dyno

$ heroku ps:scale worker=1