Bridgy pulls comments and likes from social networks back to your web site. You can also use it to publish your posts to those networks.
Python HTML Other
Clone or download
tantek and snarfed issues should be in-reply-to a repo's issues page
issues should be in-reply-to a repo's issues page, also link to indieweb u-in-reply-to instead of microformats rel page
Latest commit b9e4575 Jul 8, 2018
Permalink
Failed to load latest commit information.
.circleci write facebook's obituary, disable signup altogether Jun 4, 2018
docs fix readthedocs: once more, with feeling: use pypi's gdata! Dec 20, 2017
scripts logs: use module_versions instead of version_ids Mar 29, 2018
static write facebook's obituary, disable signup altogether Jun 4, 2018
templates issues should be in-reply-to a repo's issues page Jul 9, 2018
test attempted workaround to prevent flaky test_handlers failure Jul 8, 2018
.gitignore consolidate .gitignore a bit with wildcards Feb 15, 2018
.gitmodules start to port to virtualenv and pip for dependencies Jun 21, 2015
README.md readme: update datastore => bigquery export instructions Jun 20, 2018
admin.py /admin/sources: minor query bug fix Apr 19, 2018
app.py write facebook's obituary, disable signup altogether Jun 4, 2018
app.yaml facebook: remove signup flow Jul 5, 2018
appengine_config.py fix a bug and lots of unused imports/vars found by pyflakes and vulture Mar 28, 2018
background.yaml include app.common.yaml in background.yaml May 2, 2018
beta_users.txt add jackysee to beta users Nov 29, 2017
blog_webmention.py blog webmentions: bump up mention snippet length from 30 chars to 280 Apr 9, 2018
blogger.py python 3: add from __future__ import unicode_literals everywhere Mar 19, 2018
cloud_storage_lifecycle.json bump up datastore backup expiration in cloud storage from 30d to 90d Jun 5, 2016
cron.py try to fix a flaky test by adding import appengine_config lines Apr 4, 2018
cron.yaml move cron jobs to background service Mar 29, 2018
domain_blacklist.txt un-blacklist blog.mahabali.me Nov 15, 2017
dos.yaml blacklist 91.200.12.58, it's a script kiddie looking for xmlrpc vulns Jul 13, 2016
facebook.py facebook: remove signup flow Jul 5, 2018
facebook_test_live.py python 3: add from __future__ import unicode_literals everywhere Mar 19, 2018
flickr.py python 3: add from __future__ import unicode_literals everywhere Mar 19, 2018
github.py fix a bug and lots of unused imports/vars found by pyflakes and vulture Mar 28, 2018
googleplus.py add new per-source constants RATE_LIMIT_HTTP_CODES and DISABLE_HTTP_C… Mar 22, 2018
handlers.py handlers: allow = in ids (for eg github graphql); skip cache in dev_a… Jul 6, 2018
index.yaml add Newer responses link. for #524 Oct 29, 2015
indieauth_client_id finish indieauth signup for instagram scraping! for #603 Apr 3, 2016
instagram.py instagram: use ignore_rate_limit=True during signup Apr 13, 2018
keys.md5 first pass at medium support! outbound webmentions only. #506 Aug 14, 2016
logs.py logs: use module_versions instead of version_ids Mar 29, 2018
mapreduces.py python 3: add from __future__ import unicode_literals everywhere Mar 19, 2018
medium.py python 3: add from __future__ import unicode_literals everywhere Mar 19, 2018
models.py add 'tag' publish type for github Jul 6, 2018
original_post_discovery.py fix a bug and lots of unused imports/vars found by pyflakes and vulture Mar 28, 2018
publish.py write facebook's obituary, disable signup altogether Jun 4, 2018
queue.yaml discover: handle errors, retry up to 3x Apr 26, 2017
requirements.freeze.txt python 3: use my python-future fork with bug fixes Mar 21, 2018
requirements.txt python 3: use my python-future fork with bug fixes Mar 21, 2018
superfeedr.py unify out util.host_url() for generating links to the right domain Apr 12, 2018
tasks.py don't backfeed retweets of @-mentions. fixes #819 May 1, 2018
tumblr.py fix a bug and lots of unused imports/vars found by pyflakes and vulture Mar 28, 2018
twitter.py fix a bug and lots of unused imports/vars found by pyflakes and vulture Mar 28, 2018
util.py stop exempting facebook in Content-Security-Policy header Jun 15, 2018
webmention.py unify out util.host_url() for generating links to the right domain Apr 12, 2018
wordpress_rest.py fix a bug and lots of unused imports/vars found by pyflakes and vulture Mar 28, 2018

README.md

Bridgy Bridgy Circle CI Coverage Status

Got a web site? Want replies and likes from social networks? Want to tweet, post, and more? Bridgy is for you.

https://brid.gy/

Bridgy pulls comments and likes from social networks back to your web site. You can also use it to publish your posts to those networks. See the user docs for more details, or the developer docs if you want to contribute.

License: This project is placed in the public domain.

Development

You'll need the App Engine Python SDK version 1.9.15 or later (for vendor support) or the Google Cloud SDK (aka gcloud) with the gcloud-appengine-python and gcloud-appengine-python-extras components. Add it to your $PYTHONPATH, e.g. export PYTHONPATH=$PYTHONPATH:/usr/local/google_appengine, and then run:

virtualenv local
source local/bin/activate
pip install -r requirements.freeze.txt

# We install gdata in source mode, and App Engine doesn't follow .egg-link
# files, so add a symlink to it.
ln -s ../../../src/gdata/src/gdata local/lib/python2.7/site-packages/gdata
ln -s ../../../src/gdata/src/atom local/lib/python2.7/site-packages/atom

python -m unittest discover

The last command runs the unit tests. If you send a pull request, please include (or update) a test for the new functionality if possible!

To run the entire app locally, run this in the repo root directory:

dev_appserver.py --log_level debug app.yaml background.yaml

If you hit an error during setup, check out the oauth-dropins Troubleshooting/FAQ section. For searchability, here are a handful of error messages that have solutions there:

bash: ./bin/easy_install: ...bad interpreter: No such file or directory

ImportError: cannot import name certs

ImportError: No module named dev_appserver

ImportError: cannot import name tweepy

File ".../site-packages/tweepy/auth.py", line 68, in _get_request_token
  raise TweepError(e)
TweepError: must be _socket.socket, not socket

error: option --home not recognized

There's a good chance you'll need to make changes to granary, oauth-dropins, or webmention-tools at the same time as bridgy. To do that, clone their repos elsewhere, then install them in "source" mode with:

pip uninstall -y oauth-dropins
pip install -e <path to oauth-dropins>
ln -s <path to oauth-dropins>/oauth_dropins \
  local/lib/python2.7/site-packages/oauth_dropins

pip uninstall -y granary
pip install -e <path to granary>
ln -s <path to granary>/granary \
  local/lib/python2.7/site-packages/granary

pip uninstall -y webmentiontools
# webmention-tools isn't in pypi
ln -s <path to webmention-tools>/webmentiontools \
  local/lib/python2.7/site-packages/webmentiontools

The symlinks are necessary because App Engine's vendor module evidently doesn't follow .egg-link or .pth files. :/

To deploy to App Engine, run scripts/deploy.sh.

remote_api_shell is a useful interactive Python shell that can interact with the production app's datastore, memcache, etc. To use it, create a service account and download its JSON credentials, put it somewhere safe, and put its path in your GOOGLE_APPLICATION_CREDENTIALS environment variable.

Adding a new silo

So you want to add a new silo? Maybe MySpace, or Friendster, or even Tinder? Great! Here are the steps to do it. It looks like a lot, but it's not that bad, honest.

  1. Find the silo's API docs and check that it can do what Bridgy needs. At minimum, it should be able to get a user's posts and their comments, likes, and reposts, depending on which of those the silo supports. If you want publish support, it should also be able to create posts, comments, likes, reposts, and/or RSVPs.
  2. Fork and clone this repo.
  3. Create an app (aka client) in the silo's developer console, grab your app's id (aka key) and secret, put them into new local files in the repo root dir, following this pattern. You'll eventually want to send them to @snarfed and @kylewm too, but no hurry.
  4. Add the silo to oauth-dropins if it's not already there:
    1. Add a new .py file for your silo with an auth model and handler classes. Follow the existing examples.
    2. Add a button image.
    3. Add it to the app front page and the README.
  5. Add the silo to granary:
    1. Add a new .py file for your silo. Follow the existing examples. At minimum, you'll need to implement get_activities_response and convert your silo's API data to ActivityStreams.
    2. Add a new unit test file and write some tests!
    3. Add it to api.py (specifically Handler.get), app.py, app.yaml, index.html, and the README.
  6. Add the silo to Bridgy:
    1. Add a new .py file for your silo with a model class. Follow the existing examples.
    2. Add it to app.py, app.yaml, and handlers.py (just import the module).
    3. Add a 48x48 PNG icon to static/.
    4. Add new SILO_signup.html and SILO_user.html files in templates/ and add the silo to listen_signup.html. Follow the existing examples.
    5. Add the silo to about.html and this README.
    6. If users' profile picture URLs can change, add a cron job that updates them to cron.py and cron.yaml. Also add the model class to the datastore backup job there.
  7. Optionally add publish support:
    1. Implement create and preview_create for the silo in granary.
    2. Add the silo to publish.py: import its module, add it to SOURCES, and update this error message.
    3. Add a publish-signup block to SILO_user.html and add the silo to social_user.html.
    4. Update app.yaml.

Good luck, and happy hacking!

Monitoring

App Engine's built in dashboard and log browser are pretty good for interactive monitoring and debugging.

For alerting, we've set up Google Cloud Monitoring (née Stackdriver). Background in issue 377. It sends alerts by email and SMS when HTTP 4xx responses average >.1qps or 5xx >.05qps, latency averages >15s, or instance count averages >5 over the last 15m window.

Stats

I occasionally generate stats and graphs of usage and growth from the BigQuery dataset (#715). Here's how.

  1. Back up the full datastore to Google Cloud Storage. Include all entities except *Auth and other internal details. Check to see if any new kinds have been added since the last time this command was run.

    gcloud datastore export --async gs://brid-gy.appspot.com/stats/ --kinds Blogger,BlogPost,BlogWebmention,FacebookPage,Flickr,GitHub,GooglePlusPage,Instagram,Medium,Publish,PublishedPage,Response,SyndicatedPost,Tumblr,Twitter,WordPress
    

    Note that --kinds is required. From the export docs, Data exported without specifying an entity filter cannot be loaded into BigQuery.

  2. Import it into BigQuery:

    for kind in BlogPost BlogWebmention Publish Response SyndicatedPost; do
      bq load --replace --nosync --source_format=DATASTORE_BACKUP datastore.$kind gs://brid-gy.appspot.com/stats/all_namespaces/kind_$kind/all_namespaces_kind_$kind.export_metadata
    done
    
    for kind in Blogger FacebookPage Flickr GitHub GooglePlusPage Instagram Medium Tumblr Twitter WordPress; do
      bq load --replace --nosync --source_format=DATASTORE_BACKUP sources.$kind gs://brid-gy.appspot.com/stats/all_namespaces/kind_$kind/all_namespaces_kind_$kind.export_metadata
    done
    
  3. Run the full stats BigQuery query. Download the results as CSV.

  4. Open the stats spreadsheet. Import the CSV, replacing the data sheet.

  5. Check out the graphs! Save full size images with OS or browser screenshots, thumbnails with the Save Image button. Then post them!

Misc

The datastore is automatically backed up by a cron job that runs Datastore Admin backup and stores the results in Cloud Storage, in the brid-gy.appspot.com bucket. It backs up all entities monthly, and all entities except Response and SyndicatedPost weekly, since they make up 92% of all entities by size and they aren't as critical to keep.

We use this command to set a Cloud Storage lifecycle policy on that bucket that prunes older backups:

gsutil lifecycle set cloud_storage_lifecycle.json gs://brid-gy.appspot.com

Run this to see how much space we're currently using:

gsutil du -hsc gs://brid-gy.appspot.com/\*

Run this to download a single complete backup:

gsutil -m cp -r gs://brid-gy.appspot.com/weekly/datastore_backup_full_YYYY_MM_DD_\* .

Also see the BigQuery dataset (#715).