Bridgy pulls comments and likes from social networks back to your web site. You can also use it to publish your posts to those networks.
Python HTML Other
Permalink
Failed to load latest commit information.
scripts prevent PPD from fetching the same h-feed multiple times in a single … Apr 25, 2016
static medium: sharpen icon Aug 17, 2016
templates document why outgoing medium webmentions look ugly on their target pages Aug 26, 2016
test user page: handle unicode chars in publish source URLs Aug 18, 2016
.gitignore first pass at medium support! outbound webmentions only. #506 Aug 14, 2016
.gitmodules start to port to virtualenv and pip for dependencies Jun 21, 2015
README.md readme: gsutil -m tweak to command for downloading datastore backup Jun 6, 2016
admin.py narrow /admin/responses to just 10 Responses May 25, 2016
app.py user page: handle unicode chars in publish source URLs Aug 18, 2016
app.yaml first pass at medium support! outbound webmentions only. #506 Aug 14, 2016
appengine_config.py suppress urllib3 and BeautifulSoup4 warnings May 5, 2016
backends.yaml disable twitter-streaming backend. switching to scraping html. :( Feb 16, 2014
beta_users.txt add @dissolve to beta users Apr 28, 2016
blog_webmention.py tweak a few datastore keys in log messages so they get linkified by /log Jul 13, 2016
blogger.py blogger: expand error handling for creating comment Aug 18, 2016
circle.yml circle: upgrade app engine sdk to 1.9.37 May 12, 2016
cloud_storage_lifecycle.json bump up datastore backup expiration in cloud storage from 30d to 90d Jun 5, 2016
cron.py update twitter pictures cron job: handle 404 for single deleted account Jul 1, 2016
cron.yaml add back cron job that updates twitter profile picture URLs May 4, 2016
domain_blacklist.txt blacklist pet.t-com.ne.jp Jul 5, 2016
dos.yaml blacklist 91.200.12.58, it's a script kiddie looking for xmlrpc vulns Jul 13, 2016
facebook.py temporary workaround for @barnabywalters for #689 Aug 8, 2016
facebook_test_live.py facebook_test_live: update link to test user datastore entity Jul 7, 2016
flickr.py add $s to ends of UrlCanonicalizer approve regexps. fixes #686 Jul 19, 2016
googleplus.py google+: switch plus.login oauth scope to plus.me to avoid scary cons… Aug 13, 2016
handlers.py mf2 handlers: handle DisableSource Jul 26, 2016
index.yaml add Newer responses link. for #524 Oct 29, 2015
indieauth_client_id finish indieauth signup for instagram scraping! for #603 Apr 3, 2016
instagram.py add $s to ends of UrlCanonicalizer approve regexps. fixes #686 Jul 19, 2016
keys.md5 first pass at medium support! outbound webmentions only. #506 Aug 14, 2016
logs.py logs: extend sanitize regexp to cover more syntax and code= Jun 23, 2016
mapreduce.yaml add mapreduce to prune the Response.activity_json field. fixes #68 Mar 29, 2014
mapreduces.py noop: remove unused imports, other minor tweaks Dec 4, 2014
medium.py medium: support publications. #506 Aug 17, 2016
models.py drop fast crawl tier back down from 2h to 6h Aug 17, 2016
original_post_discovery.py prevent PPD from fetching the same h-feed multiple times in a single … Apr 25, 2016
publish.py fix crash when previewing page that's already been published. fixes #684 Jul 14, 2016
queue.yaml tentatively drop poll max_concurrent_requests down to 2 Mar 21, 2016
requirements.freeze.txt upgrade requests-toolbelt to 0.6.2 to get the timeout fix in requests… May 10, 2016
requirements.txt upgrade requests-toolbelt to 0.6.2 to get the timeout fix in requests… May 10, 2016
superfeedr.py blog outgoing webmentions: clean source links Aug 17, 2016
tasks.py add missing medium import in tasks.py Aug 18, 2016
tumblr.py fix tumblr bug when account is disabled Apr 3, 2016
twitter.py add $s to ends of UrlCanonicalizer approve regexps. fixes #686 Jul 19, 2016
util.py instagram signup: handle when indieauth gives us back an unexpected s… Aug 13, 2016
webmention.py wordpress: tweak duplicate comment error matching string Mar 29, 2016
wordpress_rest.py set OAuth scopes programmatically Oct 25, 2015

README.md

Bridgy Bridgy Circle CI Coverage Status

Got a web site? Want social network replies and likes on your site? Want to post and tweet from your site? Bridgy is for you.

https://brid.gy/

Bridgy pulls comments and likes from social networks back to your web site. You can also use it to publish your posts to those networks. See the docs for more details.

License: This project is placed in the public domain.

Development

You'll need the App Engine Python SDK version 1.9.15 or later (for vendor support). Add it to your $PYTHONPATH, e.g. export PYTHONPATH=$PYTHONPATH:/usr/local/google_appengine, and then run:

virtualenv local
source local/bin/activate
pip install -r requirements.freeze.txt

# We install gdata in source mode, and App Engine doesn't follow .egg-link
# files, so add a symlink to it.
ln -s ../../../src/gdata/src/gdata local/lib/python2.7/site-packages/gdata
ln -s ../../../src/gdata/src/atom local/lib/python2.7/site-packages/atom

python -m unittest discover

The last command runs the unit tests. If you send a pull request, please include (or update) a test for the new functionality if possible!

If you hit an error during setup, check out the oauth-dropins Troubleshooting/FAQ section. For searchability, here are a handful of error messages that have solutions there:

bash: ./bin/easy_install: ...bad interpreter: No such file or directory

ImportError: cannot import name certs

ImportError: No module named dev_appserver

ImportError: cannot import name tweepy

File ".../site-packages/tweepy/auth.py", line 68, in _get_request_token
  raise TweepError(e)
TweepError: must be _socket.socket, not socket

error: option --home not recognized

There's a good chance you'll need to make changes to granary, oauth-dropins, or webmention-tools at the same time as bridgy. To do that, clone their repos elsewhere, then install them in "source" mode with:

pip uninstall -y oauth-dropins
pip install -e <path to oauth-dropins>
ln -s <path to oauth-dropins>/oauth_dropins \
  local/lib/python2.7/site-packages/oauth_dropins

pip uninstall -y granary
pip install -e <path to granary>
ln -s <path to granary>/granary \
  local/lib/python2.7/site-packages/granary

pip uninstall -y webmentiontools
# webmention-tools isn't in pypi
ln -s <path to webmention-tools>/webmentiontools \
  local/lib/python2.7/site-packages/webmentiontools

The symlinks are necessary because App Engine's vendor module evidently doesn't follow .egg-link or .pth files. :/

To deploy to App Engine, run scripts/deploy.sh.

remote_api_shell is a useful interactive Python shell that can interact with the production app's datastore, memcache, etc. To use it, create a service account and download its JSON credentials, put it somewhere safe, and put its path in your GOOGLE_APPLICATION_CREDENTIALS environment variable.

Adding a new silo

So you want to add a new silo? Maybe MySpace, or Friendster, or even Tinder? Great! Here are the steps to do it. It looks like a lot, but it's not that bad, honest.

  1. Find the silo's API docs and check that it can do what Bridgy needs. At minimum, it should be able to get a user's posts and their comments, likes, and reposts, depending on which of those the silo supports. If you want publish support, it should also be able to create posts, comments, likes, reposts, and/or RSVPs.
  2. Fork and clone this repo.
  3. Create an app (aka client) in the silo's developer console, grab your app's id (aka key) and secret, put them into new local files in the repo root dir, following this pattern. You'll eventually want to send them to @snarfed and @kylewm too, but no hurry.
  4. Add the silo to oauth-dropins if it's not already there:
    1. Add a new .py file for your silo with an auth model and handler classes. Follow the existing examples.
    2. Add a button image.
    3. Add it to the app front page and the README.
  5. Add the silo to granary:
    1. Add a new .py file for your silo. Follow the existing examples. At minimum, you'll need to implement get_activities_response and convert your silo's API data to ActivityStreams.
    2. Add a new unit test file and write some tests!
    3. Add it to activitystreams.py (specifically Handler.get), app.py, app.yaml, index.html, and the README.
  6. Add the silo to Bridgy:
    1. Add a new .py file for your silo with a model class. Follow the existing examples.
    2. Add it to app.py, app.yaml, and handlers.py, (just import the module).
    3. Add a 24x24 PNG icon to static/.
    4. Add new SILO_signup.html and SILO_user.html files in templates/. and add the silo to listen_signup.html. Follow the existing examples.
    5. Add the silo to about.html and this README.
    6. If users' profile picture URLs can change, add a cron job that updates them to cron.py and cron.yaml. Also add the model class to the datastore backup job there.
  7. Optionally add publish support:
    1. Implement create and preview_create for the silo in granary.
    2. Add the silo to publish.py: import its module, add it to SOURCES, and update this error message.
    3. Add a publish-signup block to SILO_user.html and add the silo to social_user.html here.
    4. Update app.yaml.

Good luck, and happy hacking!

Monitoring

App Engine's built in dashboard and log browser are pretty good for interactive monitoring and debugging.

For alerting, we've set up Google Cloud Monitoring (née Stackdriver). Background in #377. It sends alerts by email and SMS when HTTP 4xx responses average >.1qps or 5xx >.05qps, latency averages >15s, or instance count averages >5 over the last 15m window.

Misc

The datastore is automatically backed up by a cron job that runs Datastore Admin backup and stores the results in Cloud Storage, in the brid-gy.appspot.com bucket. It backs up all entities monthly, and all entities except Response and SyndicatedPost weekly, since they make up 92% of all entities by size and they aren't as critical to keep.

We use this command to set a Cloud Storage lifecycle policy on that bucket that prunes older backups:

gsutil lifecycle set cloud_storage_lifecycle.json gs://brid-gy.appspot.com

Run this to see how much space we're currently using:

gsutil du -hsc gs://brid-gy.appspot.com/\*

Run this to download a single complete backup, for e.g. generating usage metrics with to_tsv.py:

gsutil -m cp -r gs://brid-gy.appspot.com/weekly/datastore_backup_full_YYYY_MM_DD_\* .