/twitter-url-regexen

The regular expression(s) Twitter uses to match URLs.
  1. Python 100.0%
Switch branches/tags
Nothing to show
Find file
Clone or download

Clone with HTTPS

Use Git or checkout with SVN using the web URL.

Open in Desktop Download ZIP

Launching GitHub Desktop...

If nothing happens, download GitHub Desktop and try again.

Launching GitHub Desktop...

If nothing happens, download GitHub Desktop and try again.

Launching Xcode...

If nothing happens, download Xcode and try again.

Launching Visual Studio...

If nothing happens, download the GitHub extension for Visual Studio and try again.

Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
alt_impls
tests
.gitmodules
README.md
test_requirements.txt
twitter_regex.py

README.md

Twitter URL Regexen

This is an attempt to extract the URL-matching regular expression(s) from the twitter-text-rb project and port them to Python.

What? Why?

This is important because any service that will be creating tweets on behalf of its users needs to be able to provide those users an accurate character count. Should be easy, right? Nope! Now that Twitter is wrapping URLs in t.co short links, the service in question will need to know exactly what parts of a user's tweet will be replaced by shortened t.co URLs.

This is mind-bendingly stupid.

To alleviate some of the pain, Twitter has kindly provided reference implementations for this behavior for Ruby, JavaScript, and Java, along with a suite of conformance tests for third party implementers.

Unfortunately, they do not provide and maintain a reference implementation for Python, and those that exist are incomplete (this particular little lib very much included).

Okay, so what is this then?

This project, twitter-url-regexen, is just an attempt to extract the URL-related regular expressions from the twitter-text-rb source code (specifically, lib/regex.rb) and port them to Python. Nothing more, nothing less.

How good is this port?

Well, it passes 62 of the 70 (at time of writing) URL-extraction conformance tests. Hopefully, that's good enough for government work (as they say).

What about those 8 failing tests?

See, here's where things get even dumber. Twitter is not just extracting these URLs based on a regular expression. There is also a fair amount of extra work done to, e.g., specially handle t.co URLs, protocol-less ccTLD URLs, non-ASCII URLs, etc.

This library does not attempt to duplicate that logic, so some of the failures stem from that basic incompatibility.

Fuck if I know about the other ones.

Running the tests

First, get the conformance tests:

git submodule update --init

Make sure you have all of the requirements (py.test and pyyaml):

pip install -r test_requirements.txt

Run the tests:

py.test

If you'd like to test some alternate Python implentations (twitter-text-python and twitter-text-py) of the same idea, you'll have to run those tests a little more manually:

py.test tests/alt*

See Also

Twitter's own docs about this inane misfeature: https://dev.twitter.com/docs/tco-url-wrapper/how-twitter-wrap-urls