Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Replaced Blekko API #2

Merged
merged 1 commit into from
Jan 23, 2014
Merged

Replaced Blekko API #2

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
16 changes: 7 additions & 9 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,18 +1,18 @@
# autolink.vim

This vim plugin automatically finds and inserts URLs for links in Markdown and
ReST documents. You can use a search engine (Blekko) or the current tab in
ReST documents. You can use a search engine (DuckDuckGo) or the current tab in
Safari or Chrome.

## Search for URLs

autolink.vim uses the [Blekko][] search engine API to find URLs matching the
autolink.vim uses the [DuckDuckGo][] search engine API to find URLs matching the
link IDs in [Markdown][] and [reStructuredText][] reference-style links. It
automatically inserts the first link found.

[Markdown]: http://daringfireball.net/projects/markdown/
[reStructuredText]: http://docutils.sourceforge.net/rst.html
[Blekko]: http://blekko.com/
[DuckDuckGo]: http://duckduckgo.com/

For example, say you have this document:

Expand All @@ -25,7 +25,7 @@ With your cursor on the last line, you can type ``<leader>ac`` (for

[Markdown]: http://daringfireball.net/projects/markdown/

Behind the scenes, the plugin searches on Blekko for the word "Markdown" and
Behind the scenes, the plugin searches on DuckDuckGo for the word "Markdown" and
inserts the first result's URL, a reasonable guess for a relevant link on the
subject, in the appropriate place. This also works in ReST documents on
hyperlink target lines like `.. _Markdown: link goes here`.
Expand Down Expand Up @@ -68,8 +68,8 @@ To do this with your browser's foremost tab instead, type ``<leader>ab`` (the

# Installing

The plugin requires vim to be built with Python bindings (to communicate with
the Blekko API). If you're using [Pathogen][], just clone this repository into
The plugin requires vim to be built with Python bindings.
If you're using [Pathogen][], just clone this repository into
your bundles directory (and run `:Helptags`). Otherwise, place files in
`plugin`, `autoload`, and `doc` into the corresponding directories in `~/.vim`.

Expand Down Expand Up @@ -97,9 +97,7 @@ place I saw this kind of functionality.
[luckylink]: http://brettterpstra.com/automated-search-and-link-text-service/
[Brett Terpstra]: http://brettterpstra.com/

The code is available under the [MIT license][]. The plugin contains an inlined
copy of [python-blekko][], which is under the same license. My thanks to the
kind folks at [Blekko][] for providing free access to their search API.
The code is available under the [MIT license][].

[MIT license]: http://www.opensource.org/licenses/MIT
[python-blekko]: https://github.com/sampsyo/python-blekko
160 changes: 12 additions & 148 deletions autoload/autolink.vim
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,24 +3,22 @@

" Python support functionality.
python << ENDPYTHON
import re
import sys
import urllib2
import vim

_api = None
def get_link(terms):
# Lazily initialize API singleton.
global _api
if not _api:
_api = Blekko(source='410a531a')
# Perform query.
try:
res = _api.query(terms + ' /ps=1')
except BlekkoError as exc:
return None
if len(res):
return res[0].url
query = 'https://duckduckgo.com/html/?q={}'.format(terms.strip().replace(' ','+'))
html = urllib2.urlopen(query).read()
for link in re.findall('div.*?web-result.*?href="(.*?)"', html, re.DOTALL):
if "duckduckgo.com" not in link:
return link
ENDPYTHON


" Get a Blekko result URL.
" Get a DuckDuckGo result URL.
function! s:link_for_terms(terms)
python << ENDPYTHON
terms = vim.eval("a:terms")
Expand Down Expand Up @@ -163,14 +161,14 @@ endfunction
" Main entry functions and default bindings.

function! autolink#DefComplete()
if &filetype == "markdown"
if (&filetype == "markdown" || &filetype == "mkd")
call s:markdown_complete()
elseif &filetype == "rst"
call s:rest_complete()
endif
endfunction
function! autolink#DefCreate()
if &filetype == "markdown"
if (&filetype == "markdown" || &filetype == "mkd")
call s:markdown_create()
elseif &filetype == "rst"
call s:rest_create()
Expand All @@ -194,137 +192,3 @@ function! autolink#CombinedBrowser()
call autolink#AppendBrowserURL()
execute "normal! `q"
endfunction


" The python-blekko module, inlined.

python << ENDPYTHON
"""Bindings for the Blekko search API."""
import urllib
import time
import threading
import json

BASE_URL = 'http://blekko.com'
RATE_LIMIT = 1.0 # Seconds.

class _rate_limit(object):
"""A decorator that limits the rate at which the function may be
called. Minimum interval is given by RATE_LIMIT. Thread-safe using
locks.
"""
def __init__(self, fun):
self.fun = fun
self.last_call = 0.0
self.lock = threading.Lock()

def __call__(self, *args, **kwargs):
with self.lock:
# Wait until RATE_LIMIT time has passed since last_call,
# then update last_call.
since_last_call = time.time() - self.last_call
if since_last_call < RATE_LIMIT:
time.sleep(RATE_LIMIT - since_last_call)
self.last_call = time.time()

# Call the original function.
return self.fun(*args, **kwargs)

class BlekkoError(Exception):
"""Base class for exceptions raised by this module."""

class ServerError(BlekkoError):
"""Raised when the server denies a request for some reason."""

@_rate_limit
def _http_request(url):
"""Make a (rate-limited) request to the Blekko server and return the
resulting data.
"""
f = urllib.urlopen(url)
code = f.getcode()
if code == 503:
raise ServerError('server overloaded (503)')
elif code != 200:
raise ServerError('HTTP error {}'.format(code))
return f.read()

class ResponseObject(object):
"""An object wrapper for a dictionary providing item access to
values in the underlying dictionary.
"""
def __init__(self, data):
self.data = data

def __getattr__(self, key):
if key in self.data:
return self.data[key]
raise KeyError('no such field {}'.format(repr(key)))

def __repr__(self):
return '{}({})'.format(type(self).__name__, self.data)

class Result(ResponseObject):
"""A single search result. Available fields include url, url_title,
snippet, rss, short_host, short_host_url, and display_url.
"""

class ResultSet(ResponseObject):
"""A set of search results. Behaves as an immutable sequence
containing Result objects (accessible via iteration or
subscripting). Additional available fields include q, noslash_q,
total_num, num_elem_start, num_elem,end, nav_page_range_start,
nav_page_range_end, tag_switches, sug_slash, and
universal_total_results.
"""
def __iter__(self):
for result in self.data['RESULT']:
yield Result(result)

def __getitem__(self, index):
return Result(self.data['RESULT'][index])

def __len__(self):
return len(self.data['RESULT'])

class Blekko(object):
def __init__(self, auth=None, source=None):
"""Create an API object. Either `auth` or `source` must be
provided to identify the application (use whichever was assigned
to you by Blekko).
"""
if not auth and not source:
raise BlekkoError('API key not provided')
self.auth = auth
self.source = source

def _request(self, path, params):
"""Make a (rate-limited) request to the Blekko server and return
the result data.
"""
params = dict(params) # Make a copy.
if self.auth:
params['auth'] = self.auth
else:
params['source'] = self.source
query = urllib.urlencode(params)
url = "{}{}?{}".format(BASE_URL, path, query)
return _http_request(url)

def query(self, terms, page=0):
"""Perform a search and return a ResultSet object."""
data = self._request('/ws/', {
'q': terms + ' /json',
'p': str(page),
})
return ResultSet(json.loads(data))

def pagestats(self, url):
"""Get page statistics for a URL and return a dictionary of
available information.
"""
data = self._request('/api/pagestats', {
'url': url,
})
return json.loads(data)
ENDPYTHON