We use Python 3.7. Install Python dependencies and run database migrations:
$ pip install pipenv
$ pipenv install --dev
We use MySQL (MariaDB 10). To setup database connections:
- Create a database. Take a note of the database name, hostname, username, and password.
- The database connection string will be
mysql+pymysql://<username>:<password>@<hostname>/<database name>
. We use PyMySQL, so the connection string should starts withmysql+pymysql
in order for SQLAlchemy to pick up the right driver. - Copy
.env.default
to.env
. - Edit the value of
DB_URL
to be the connection string. You can leave all other settings in.env
as is for now. - Run db migrations with
pipenv run alembic upgrade head
Then, update your site table. First, you need an API key from Airtable generated here and the id of your base (see here for info). Add the following variables to .env
:
$ echo AIRTABLE_BASE_ID={id_of_your_airtable_base} >> .env
$ echo AIRTABLE_API_KEY={your_api_key} >> .env
$ echo SITE_TYPES=["{site_type_1}", "{site_type_2}",...] >> .env
Afterwards, do the following to update your site table
$ SCRAPY_PROJECT=sitesAirtable pipenv run scrapy crawl updateSites
- Find new articles in all ACTIVE facebook pages/groups listed in Site table in database. Activity is determined by
is_active
column in airtable.
$ python fb.py discover
Optional Arguments:
--limit-sec: process run time limit in seconds, default = 3000.
--site-limit-sec: max load time in seconds for a site, default = 1800.
- Revisit articles in database based on next_snapshot_at parameter in Article Table on the mysql database. The function will save new html to ArticleSnapshot table and update the snapshot parameters in Article Table.
# update all
$ python fb.py update
Optional Arguments:
--limit-sec: process run time limit in seconds, default = 3000.
--article-limit-sec: max load time in seconds for an article, default = 60.
- Find new articles in a specified facebook page/group.
$ python fb.py site discover {site-id}
Optional Arguments:
--limit-sec: process run time limit in seconds, default = 1800.
- Revisit articles in a specified facebook page/group.
$ python fb.py site update {site-id}
Optional Arguments:
--limit-sec: process run time limit in seconds, default = 3000.
--article-limit-sec: max load time in seconds for an article, default = 60.
- Revisit one article specified by id
$ python fb.py post update {article-id}
Optional Arguments:
--limit-sec: process run time limit in seconds, default = 60.