Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with
or
.
Download ZIP

Loading…

[MRG+2] DOC add links to github sourcecode in API reference #2777

Merged
merged 1 commit into from

10 participants

@jnothman
Owner

I enjoy using the API reference, but think a link to the source code on github would often help.

Because sphinx.ext.linkcode does not provide the full path or line number of the class/function, I'm getting this stuff using grep, which relies on a unix shell and certain path structures, which is a bit not-nice. Since github needs a branch/commit reference, I'm using git rev-parse --abbrev-ref HEAD which can return a branch/tag name, but a more static commit reference (i.e. rev-parse --short) might be more sensible.

WDYT? ping @jaquesgrobler

@jaquesgrobler

I'm a fan of this.. There was some discussion on this idea a while back on #1680 but no consensus was reached..
I scanned through your code and looks fine .. Don't have time for an in detail look this second but will try later tonight or tomorrow.. that asside i'm +1 on this as a few people have requested this

:+1:

@jnothman
Owner
@jnothman
Owner

Advantages of using inspect over grep

  • not shell-specific (although calling system git rev-parse, or employing GitPython, is still needed)
  • can easily produce source links for methods as well as module-level functions
  • doesn't rely on relative path structures, instead using Python's path Disadvantages:
  • copy installed on path may not be in sync (but should be for the rest of doc compilation to work)
  • may be a litle slower than grep: not a single pass through files
  • may fail or get incorrect source for wrapped objects such as abstractmethods
  • requires an alternative solution (open and grep) for pyx objects (currently just sklearn.svm.libsvm.* of http://scikit-learn.org/stable/modules/classes.html#low-level-methods) that may require using relative path knowledge
@GaelVaroquaux
@jnothman
Owner

I've updated with links for classes, functions and methods in Python (but not Cython), using inspect.

I've also changed from using rev-parse --abbrev-ref to rev-parse --short to give a fixed commit reference, rather than a name. Note that I still use the shell for this, but could change it to a file operation (find closest dir containing .git, read .git/HEAD, and dereference the ref from the refs dir), but I think it's safer to rely on the official tools.

@jnothman
Owner

I've updated with links for classes, functions and methods in Python (but not Cython), using inspect.

I've also changed from using rev-parse --abbrev-ref to rev-parse --short to give a fixed commit reference, rather than a name. Note that I still use the shell for this, but could change it to a file operation (find closest dir containing .git, read .git/HEAD, and dereference the ref from the refs dir), but I think it's safer to rely on the official tools.

@coveralls

Coverage Status

Coverage remained the same when pulling cf91053 on jnothman:doc_linkcode into 8c3efd4 on scikit-learn:master.

@jnothman
Owner

I've now noticed scipy has similar, at https://github.com/scipy/scipy/blob/6a4460f68315f0669604054be91ceeacd606f0b6/doc/source/conf.py#L314. I need to look closer to see what cases they handle that we don't, etc. Since they explicitly version their dev builds with git revisions, they are able to use different commit references in the URL when for dev or release (has or tag name respectively).

@GaelVaroquaux

Hey Joel, I think I'll let you have z look at scipy before I review this PR.

@jnothman
Owner

Diffs:

  • scipy uses master or version tags. I'm using commit hashes
  • scipy handled __init__.py where I failed. Now fixed.
@jnothman
Owner

@GaelVaroquaux, I think we should pull this in, and see if there are any problems with the current commit refs.

@GaelVaroquaux

I'll have a look.

I however have a hard time feeling excited about this feature. For me it is additional code that will at some point fail and require debugging or evolution (all code does) for little feature.

I satisfy my need to look at source code by using "symbol??" in IPython.

Sorry, I should be thankful for the feature and it's not much code actually. It's just that I have the feeling that we are quite heavily loaded in terms of maintenance and I personally never have time to push features that I really need. So feel like focusing on the high-value code, where of course high value varies from one person to another. :)

doc/conf.py
((18 lines not shown))
+
+
+GITHUB_FMT = ('https://github.com/scikit-learn/scikit-learn/'
+ 'blob/{revision}/{pre_path}{path}#L{lineno}')
+
+
+def linkcode_resolve(domain, info):
+ """Determine a link to GitHub source for the given class/method/function"""
+
+ if domain not in ('py', 'pyx'):
+ return
+ if not info.get('module') or not info.get('fullname'):
+ return
+
+ try:
+ revision = linkcode_resolve.revision
@GaelVaroquaux Owner

I am a bit dense, but where is this set?

@jnothman Owner

In a previous version of the code, apparently :s

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
@jnothman
Owner
doc/conf.py
((9 lines not shown))
+
+
+def _get_git_revision(CMD='git rev-parse --short HEAD'):
+ try:
+ revision = subprocess.check_output(CMD.split()).strip()
+ except subprocess.CalledProcessError:
+ print('Failed to execute git to get revision')
+ return None
+ return revision
+
+
+GITHUB_FMT = ('https://github.com/scikit-learn/scikit-learn/'
+ 'blob/{revision}/{pre_path}{path}#L{lineno}')
+
+
+def linkcode_resolve(domain, info):
@GaelVaroquaux Owner

I think that you should indicate (eg in a comment, or in the docstring) that this is used by the sphinx linkcode extension.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
@GaelVaroquaux

Lightweight and elegant code.

I made a couple of comment. My only additional and non trivial comments would be:

  • Could you please move the corresponding code in another file in the sphinx_ext. It would make the conf.py easier to read, and all you need is to import the right symbol (linkcode_resolve).

  • Can you check whether it is possible to have nosetest run tests in the sphinx_ext folder (it might require a bit of work on the setup.cfg). I am getting more and more nervous about or sphinx_ext folder and the fact that it is untested.

Thanks a lot for your work, as usual.

@coveralls

Coverage Status

Coverage remained the same when pulling 4b7973c on jnothman:doc_linkcode into b895586 on scikit-learn:master.

@jnothman
Owner
@coveralls

Coverage Status

Coverage remained the same when pulling 8b87b78 on jnothman:doc_linkcode into b895586 on scikit-learn:master.

@jnothman
Owner

The refactored version can now be more easily tested, e.g. using standard library or sphinx modules for independence:

>>> from github_link import _linkcode_resolve
>>> _linkcode_resolve('py', {'module': 'distutils', 'fullname': 'config.PyPIRCCommand.initialize_options'},
...     package='distutils',
...     url_fmt='http://hg.python.org/cpython/file/{revision}/Lib/{package}/{path}#L{lineno}',
...     revision='xxxx')
'http://hg.python.org/cpython/file/xxxx/Lib/distutils/config.py#L105'
@kastnerkyle
Owner

Getting an error building the docs with make html after rebasing this branch with master:

Exception occurred:
  File "/volatile/accounts/kkastner/src/scikit-learn/doc/sphinxext/github_link.py", line 31, in _linkcode_resolve
    class_name = info['fullname'].encode('utf-8').split('.')[0]
TypeError: Type str doesn't support the buffer API

I am also getting an error with nosetests, in the doctest part

ERROR: Failure: ImportError (No module named 'github_link')
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/volatile/accounts/kkastner/anaconda3/lib/python3.4/site-packages/nose/failure.py", line 39, in runTest
    raise self.exc_val.with_traceback(self.tb)
  File "/volatile/accounts/kkastner/anaconda3/lib/python3.4/site-packages/nose/loader.py", line 414, in loadTestsFromName
    addr.filename, addr.module)
  File "/volatile/accounts/kkastner/anaconda3/lib/python3.4/site-packages/nose/importer.py", line 47, in importFromPath
    return self.importFromDir(dir_path, fqname)
  File "/volatile/accounts/kkastner/anaconda3/lib/python3.4/site-packages/nose/importer.py", line 94, in importFromDir
    mod = load_module(part_fqname, fh, filename, desc)
  File "/volatile/accounts/kkastner/anaconda3/lib/python3.4/imp.py", line 235, in load_module
    return load_source(name, filename, file)
  File "/volatile/accounts/kkastner/anaconda3/lib/python3.4/imp.py", line 171, in load_source
    module = methods.load()
  File "<frozen importlib._bootstrap>", line 1220, in load
  File "<frozen importlib._bootstrap>", line 1200, in _load_unlocked
  File "<frozen importlib._bootstrap>", line 1129, in _exec
  File "<frozen importlib._bootstrap>", line 1471, in exec_module
  File "<frozen importlib._bootstrap>", line 321, in _call_with_frames_removed
  File "/volatile/accounts/kkastner/src/scikit-learn/doc/conf.py", line 26, in <module>
    from github_link import make_linkcode_resolve
ImportError: No module named 'github_link'

I am using Python 3.4, other packages are from Anaconda 2.0 - looks like the first might be a Py3 strings issue, specifically buffer vs. memoryview? As for the second, @ogrisel mentioned a possible issue with local imports in python 3?

@kastnerkyle
Owner

In Python 2.7 I only get the github_link error - looks like the first is specific to Python 3.

Additionally, the docs that are built work fine locally (well, as good as master, which still has a few issues it seems). But, if you build from a git branch and click the link, the corresponding github text does not work. IE I get https://github.com/scikit-learn/scikit-learn/blob/03d48dd/sklearn/cluster/dbscan_.py#L178 . Changing 03d48dd to master, so the link becomes https://github.com/scikit-learn/scikit-learn/blob/master/sklearn/cluster/dbscan_.py#L178 , everything works as expected.

I don't know if this is critical or not - a local doc with git already has the source directly, and on the website it should work as expected (I think). Thoughts?

After talking with @ogrisel, it seems this is happening because of the rebase with master - since rebase changed the commit hash the link doesn't show up correctly. It should work fine once merged, beyond the github_link thing, which is not happening on Travis apparently.

@jnothman
Owner

@fabianp I'll revert it and see what happens.

@jnothman
Owner

I can't understand the AppVeyor interface - it seems to be missing the consoles that I usually go to for explanation of the failure.

@ogrisel
Owner

I will disable appveyor in the mean time.

@ogrisel
Owner

@fabianp I'll revert it and see what happens.

@jnothman what do you want to revert? Is this PR ready for merge or not?

@jnothman
Owner

I've done with revert, forcing a Travis check by rebasing.

@jnothman
Owner

Failing with

======================================================================
ERROR: Failure: ImportError (No module named github_link)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/travis/virtualenv/python2.7_with_system_site_packages/local/lib/python2.7/site-packages/nose/loader.py", line 411, in loadTestsFromName
    addr.filename, addr.module)
  File "/home/travis/virtualenv/python2.7_with_system_site_packages/local/lib/python2.7/site-packages/nose/importer.py", line 47, in importFromPath
    return self.importFromDir(dir_path, fqname)
  File "/home/travis/virtualenv/python2.7_with_system_site_packages/local/lib/python2.7/site-packages/nose/importer.py", line 94, in importFromDir
    mod = load_module(part_fqname, fh, filename, desc)
  File "/home/travis/build/scikit-learn/scikit-learn/doc/conf.py", line 26, in <module>
    from github_link import make_linkcode_resolve
ImportError: No module named github_link
@jnothman
Owner

Well, Travis seems to not have a problem with the path modification. It's using:
['/home/travis/build/scikit-learn/scikit-learn/sphinxext', '/home/travis/build/scikit-learn/scikit-learn/doc', '/home/travis/build/scikit-learn/scikit-learn', '/home/travis/anaconda/envs/testenv/bin', '/home/travis/anaconda/envs/testenv/lib/python26.zip', '/home/travis/anaconda/envs/testenv/lib/python2.6', '/home/travis/anaconda/envs/testenv/lib/python2.6/plat-linux2', '/home/travis/anaconda/envs/testenv/lib/python2.6/lib-tk', '/home/travis/anaconda/envs/testenv/lib/python2.6/lib-old', '/home/travis/anaconda/envs/testenv/lib/python2.6/lib-dynload', '/home/travis/anaconda/envs/testenv/lib/python2.6/site-packages/setuptools-3.6-py2.6.egg', '/home/travis/anaconda/envs/testenv/lib/python2.6/site-packages', '/home/travis/build/scikit-learn/scikit-learn/doc']
Why is github_link failing to import from that first path entry?

@jnothman
Owner

Ah. I had thought the test failure was in make test-sphinxext, it's in make test-doc, which I did not realise Travis ran. The reason it is failing is because test-doc is run from the repository root, not from within the doc directory.

The only reason that this test is not failing at master is that there is:

try:
...
except:
   pass

around the other relevant import!

@coveralls

Coverage Status

Coverage remained the same when pulling 5224900 on jnothman:doc_linkcode into 3f5e691 on scikit-learn:master.

@coveralls

Coverage Status

Coverage increased (+0.09%) when pulling abad4ae on jnothman:doc_linkcode into 3f5e691 on scikit-learn:master.

@jnothman
Owner

Yay! Travis likes me.

@fabianp
Owner

Works fine for me. +1 for merge

@jnothman jnothman changed the title from [MRG] DOC add links to github sourcecode in API reference to [MRG+1.5] DOC add links to github sourcecode in API reference
@jnothman jnothman changed the title from [MRG+1.5] DOC add links to github sourcecode in API reference to [MRG+1] DOC add links to github sourcecode in API reference
@kastnerkyle
Owner

I am having trouble on Pyton 3.4 still...

Exception occurred:
  File "<frozen importlib._bootstrap>", line 2281, in _handle_fromlist
TypeError: hasattr(): attribute name must be string
The full traceback has been saved in /tmp/sphinx-err-uabh6l0b.log, if you want to report the issue to the developers.
Please also report this if it was a user error, so that a better error message can be provided next time.
A bug report can be filed in the tracker at <https://bitbucket.org/birkenfeld/sphinx/issues/>. Thanks!
make: *** [html] Error 1

Look sphinx related - any comment? If so, I can try to fix my box and confirm a 3.4 build (unless that is what @fabianp alread did :) )

@jnothman
Owner

Can you provide more of that traceback?

@kastnerkyle
Owner
plot_tree_regression_multioutput
reading sources... [ 29%] data_transforms
reading sources... [ 29%] datasets/covtype
reading sources... [ 29%] datasets/index
reading sources... [ 29%] datasets/labeled_faces
reading sources... [ 30%] datasets/mldata
reading sources... [ 30%] datasets/olivetti_faces
reading sources... [ 30%] datasets/twenty_newsgroups
reading sources... [ 30%] developers/debugging
reading sources... [ 30%] developers/index
reading sources... [ 31%] developers/maintainer
reading sources... [ 31%] developers/performance
reading sources... [ 31%] developers/utilities
reading sources... [ 31%] documentation
reading sources... [ 31%] faq
reading sources... [ 31%] index
reading sources... [ 32%] install
reading sources... [ 32%] model_selection
reading sources... [ 32%] modules/biclustering
reading sources... [ 32%] modules/classes
reading sources... [ 32%] modules/clustering
reading sources... [ 32%] modules/computational_performance
reading sources... [ 33%] modules/covariance
reading sources... [ 33%] modules/cross_decomposition
reading sources... [ 33%] modules/cross_validation
reading sources... [ 33%] modules/decomposition
reading sources... [ 33%] modules/density
reading sources... [ 33%] modules/dp-derivation
reading sources... [ 34%] modules/ensemble
reading sources... [ 34%] modules/feature_extraction
reading sources... [ 34%] modules/feature_selection
reading sources... [ 34%] modules/gaussian_process
reading sources... [ 34%] modules/generated/sklearn.base.BaseEstimator
reading sources... [ 34%] modules/generated/sklearn.base.ClassifierMixin
reading sources... [ 35%] modules/generated/sklearn.base.ClusterMixin
reading sources... [ 35%] modules/generated/sklearn.base.RegressorMixin
reading sources... [ 35%] modules/generated/sklearn.base.TransformerMixin
reading sources... [ 35%] modules/generated/sklearn.base.clone
reading sources... [ 35%] modules/generated/sklearn.cluster.AffinityPropagation

SNIP (A ton of ConvergenceWarnings)

/home/kkastner/src/scikit-learn/sklearn/linear_model/least_angle.py:270: ConvergenceWarning: Regressors in active set degenerate. Dropping a regressor, after 210 iterations, i.e. alpha=6.186e-16, with an active set of 20 regressors, and the smallest cholesky pivot element being 2.980e-08
  ConvergenceWarning)
<string>:69: RuntimeWarning: divide by zero encountered in true_divide
/home/kkastner/src/scikit-learn/sklearn/manifold/locally_linear.py:318: DeprecationWarning: using a non-integer number instead of an integer will result in an error in the future
  Yi = np.empty((n_neighbors, 1 + n_components + dp), dtype=np.float)
/home/kkastner/src/scikit-learn/sklearn/manifold/locally_linear.py:318: DeprecationWarning: using a non-integer number instead of an integer will result in an error in the future
  Yi = np.empty((n_neighbors, 1 + n_components + dp), dtype=np.float)
/home/kkastner/src/scikit-learn/sklearn/manifold/locally_linear.py:318: DeprecationWarning: using a non-integer number instead of an integer will result in an error in the future
  Yi = np.empty((n_neighbors, 1 + n_components + dp), dtype=np.float)
/home/kkastner/src/scikit-learn/sklearn/metrics/metrics.py:1771: UndefinedMetricWarning: Precision and F-score are ill-defined and being set to 0.0 in labels with no predicted samples.
  'precision', 'predicted', average, warn_for)
/home/kkastner/src/scikit-learn/sklearn/feature_selection/univariate_selection.py:106: RuntimeWarning: invalid value encountered in true_divide
  f = msb / msw

Exception occurred:
  File "<frozen importlib._bootstrap>", line 2281, in _handle_fromlist
TypeError: hasattr(): attribute name must be string
The full traceback has been saved in /tmp/sphinx-err-uabh6l0b.log, if you want to report the issue to the developers.
Please also report this if it was a user error, so that a better error message can be provided next time.
A bug report can be filed in the tracker at <https://bitbucket.org/birkenfeld/sphinx/issues/>. Thanks!
make: *** [html] Error 1
@fabianp
Owner

I only checked this under python2 because under python3 I still get the failures detailed #3387

@kastnerkyle
Owner

Ah ok. I guess we will have to dig deeper...

@fabianp
Owner

@kastnerkyle : you are running OSX ?

@kastnerkyle
Owner
doc/sphinxext/github_link.py
@@ -0,0 +1,79 @@
+from operator import attrgetter
+import inspect
+import subprocess
+import os
+import sys
+from functools import partial
+
+
+def _get_git_revision(CMD='git rev-parse --short HEAD'):
@vene Owner
vene added a note

I'm not crazy about this function definition. I can't think of any context where you'd want to call this function with an argument, and argument names shouldn't be all caps.

I'd make CMD=... a variable defined within the function.

@amueller Owner

goes to current head, so everything is fine.

@vene Owner
vene added a note

I was just commenting about the style.

@jnothman Owner

Sometimes default args are used as a not-quite-global way to store constants (which was more relevant when github_link wasn't its own module). However, I may have also done this because one may choose to use --abrev-ref (which shows the branch name) instead of --short (which shows the SHA). I'll change it to a global constant.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
@vene vene commented on the diff
doc/sphinxext/github_link.py
((20 lines not shown))
+
+ This is called by sphinx.ext.linkcode
+
+ An example with a long-untouched module that everyone has
+ >>> _linkcode_resolve('py', {'module': 'tty',
+ ... 'fullname': 'setraw'},
+ ... package='tty',
+ ... url_fmt='http://hg.python.org/cpython/file/'
+ ... '{revision}/Lib/{package}/{path}#L{lineno}',
+ ... revision='xxxx')
+ 'http://hg.python.org/cpython/file/xxxx/Lib/tty/tty.py#L18'
+ """
+
+ if revision is None:
+ return
+ if domain not in ('py', 'pyx'):
@vene Owner
vene added a note

Why is this called "domain"?

@jnothman Owner

ask sphinx.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
@jnothman
Owner

Ubuntu 14.04 LTS, Python3.4, NumPy 1.8.1: I get the same error, @kastnerkyle , on this branch, but not on master

@jnothman
Owner

Found and fixed, @kastnerkyle.

@coveralls

Coverage Status

Coverage remained the same when pulling 79afc76 on jnothman:doc_linkcode into 0d57c23 on scikit-learn:master.

@kastnerkyle
Owner

It builds fine for me now - the only thing I still see is

Warning: Embedding documentation hyperlinks requires internet access

because I am offline right now on my docbuild machine.

This seems fine to me but just wanted to report it. Other than that, this is :+1: for me

@jnothman
Owner
@kastnerkyle
Owner

OK great. I am usually online all the time - my connection bumped sometime during the doc build I guess. This seems like a really nice feature for usability and browsing!

@jnothman jnothman changed the title from [MRG+1] DOC add links to github sourcecode in API reference to [MRG+2] DOC add links to github sourcecode in API reference
@ogrisel
Owner

I am building the doc from this PR on my box as well. Will let you know when it's done.

@ogrisel
Owner

Under Python 3.4 I get URL such as:

https://github.com/scikit-learn/scikit-learn/blob/b%2779afc76%27/sklearn/ensemble/forest.py#L1116

The git hash should be decode with the .decode('ascii') method prior to building the URL.

@kastnerkyle
Owner
@ogrisel
Owner

If I fix the URL manually to remobe the leading b' and the trailing ' in my browser, it I get the right page / line in my browser.

@ogrisel ogrisel commented on the diff
doc/sphinxext/github_link.py
((67 lines not shown))
+ return url_fmt.format(revision=revision, package=package,
+ path=fn, lineno=lineno)
+
+
+def make_linkcode_resolve(package, url_fmt):
+ """Returns a linkcode_resolve function for the given URL format
+
+ revision is a git commit reference (hash or name)
+
+ package is the name of the root module of the package
+
+ url_fmt is along the lines of ('https://github.com/USER/PROJECT/'
+ 'blob/{revision}/{package}/'
+ '{path}#L{lineno}')
+ """
+ revision = _get_git_revision()
@ogrisel Owner
ogrisel added a note

revision = _get_git_revision().decode('ascii') for instance.

@ogrisel Owner
ogrisel added a note

Actually it's probably better to fold the decode call into the _get_git_revision function itself.

@jnothman Owner

Thanks @ogrisel

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
@jnothman
Owner

I've squashed the commit history.

@ogrisel
Owner

Before your pushed 060c248, I launched a full make clean html under Python 3 with the following change:

diff --git a/doc/sphinxext/github_link.py b/doc/sphinxext/github_link.py
index 7e606a6..4f28477 100644
--- a/doc/sphinxext/github_link.py
+++ b/doc/sphinxext/github_link.py
@@ -14,7 +14,7 @@ def _get_git_revision():
     except subprocess.CalledProcessError:
         print('Failed to execute git to get revision')
         return None
-    return revision
+    return revision.decode('ascii')

and I confirm that it works as expected. I have not checked with Python 2 though.

@coveralls

Coverage Status

Coverage remained the same when pulling 060c248 on jnothman:doc_linkcode into e79d2bb on scikit-learn:master.

@jnothman
Owner

I'm now generating the URL as a str in both versions; it works in both. If you'd rather an explicit unicode string in Py2, I'll try that now.

@jnothman
Owner

Changed to produce a unicode string.

@ogrisel
Owner
  • bytes is the type for byte strings in Python 3
  • str is the type for byte strings in Python 2 and the type for unicode strings in Python 3
  • unicode is the type for unicode string in Python 2

So I think we should always decode the bytes to generate URL that is a unicode string, that is str in Python 3 and unicode in Python 2.

@ogrisel
Owner

Thanks for the fix. merging!

@ogrisel ogrisel merged commit 7a7fca8 into scikit-learn:master
@jnothman
Owner

Great! I look forward to using this when introducing people to particular scikit-learn components.

@jnothman jnothman deleted the jnothman:doc_linkcode branch
@arjoly
Owner

Does it point on master or a particular branch?

@jnothman
Owner
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Commits on Jul 21, 2014
  1. @jnothman
This page is out of date. Refresh to see the latest.
Showing with 97 additions and 2 deletions.
  1. +1 −1  Makefile
  2. +12 −1 doc/conf.py
  3. +84 −0 doc/sphinxext/github_link.py
View
2  Makefile
@@ -28,7 +28,7 @@ test-code: in
test-sphinxext:
$(NOSETESTS) -s -v doc/sphinxext/
test-doc:
- $(NOSETESTS) -s -v doc/ doc/modules/ doc/datasets/ \
+ $(NOSETESTS) -s -v doc/*.rst doc/modules/ doc/datasets/ \
doc/developers doc/tutorial/basic doc/tutorial/statistical_inference \
doc/tutorial/text_analytics
View
13 doc/conf.py
@@ -12,6 +12,7 @@
# All configuration values have a default; values that are commented out
# serve to show the default.
+from __future__ import print_function
import sys
import os
from sklearn.externals.six import u
@@ -22,6 +23,8 @@
# absolute, like shown here.
sys.path.insert(0, os.path.abspath('sphinxext'))
+from github_link import make_linkcode_resolve
+
# -- General configuration ---------------------------------------------------
# Try to override the matplotlib configuration as early as possible
@@ -34,7 +37,8 @@
# extensions coming with Sphinx (named 'sphinx.ext.*') or your custom ones.
extensions = ['gen_rst',
'sphinx.ext.autodoc', 'sphinx.ext.autosummary',
- 'sphinx.ext.pngmath', 'numpy_ext.numpydoc'
+ 'sphinx.ext.pngmath', 'numpy_ext.numpydoc',
+ 'sphinx.ext.linkcode',
]
autosummary_generate = True
@@ -231,3 +235,10 @@ def setup(app):
app.add_javascript('js/copybutton.js')
# to format example galleries:
app.add_javascript('js/examples.js')
+
+
+# The following is used by sphinx.ext.linkcode to provide links to github
+linkcode_resolve = make_linkcode_resolve('sklearn',
+ u'https://github.com/scikit-learn/'
+ 'scikit-learn/blob/{revision}/'
+ '{package}/{path}#L{lineno}')
View
84 doc/sphinxext/github_link.py
@@ -0,0 +1,84 @@
+from operator import attrgetter
+import inspect
+import subprocess
+import os
+import sys
+from functools import partial
+
+REVISION_CMD = 'git rev-parse --short HEAD'
+
+
+def _get_git_revision():
+ try:
+ revision = subprocess.check_output(REVISION_CMD.split()).strip()
+ except subprocess.CalledProcessError:
+ print('Failed to execute git to get revision')
+ return None
+ return revision.decode('utf-8')
+
+
+def _linkcode_resolve(domain, info, package, url_fmt, revision):
+ """Determine a link to online source for a class/method/function
+
+ This is called by sphinx.ext.linkcode
+
+ An example with a long-untouched module that everyone has
+ >>> _linkcode_resolve('py', {'module': 'tty',
+ ... 'fullname': 'setraw'},
+ ... package='tty',
+ ... url_fmt='http://hg.python.org/cpython/file/'
+ ... '{revision}/Lib/{package}/{path}#L{lineno}',
+ ... revision='xxxx')
+ 'http://hg.python.org/cpython/file/xxxx/Lib/tty/tty.py#L18'
+ """
+
+ if revision is None:
+ return
+ if domain not in ('py', 'pyx'):
@vene Owner
vene added a note

Why is this called "domain"?

@jnothman Owner

ask sphinx.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
+ return
+ if not info.get('module') or not info.get('fullname'):
+ return
+
+ class_name = info['fullname'].split('.')[0]
+ if type(class_name) != str:
+ # Python 2 only
+ class_name = class_name.encode('utf-8')
+ module = __import__(info['module'], fromlist=[class_name])
+ obj = attrgetter(info['fullname'])(module)
+
+ try:
+ fn = inspect.getsourcefile(obj)
+ except Exception:
+ fn = None
+ if not fn:
+ try:
+ fn = inspect.getsourcefile(sys.modules[obj.__module__])
+ except Exception:
+ fn = None
+ if not fn:
+ return
+
+ fn = os.path.relpath(fn,
+ start=os.path.dirname(__import__(package).__file__))
+ try:
+ lineno = inspect.getsourcelines(obj)[1]
+ except Exception:
+ lineno = ''
+ return url_fmt.format(revision=revision, package=package,
+ path=fn, lineno=lineno)
+
+
+def make_linkcode_resolve(package, url_fmt):
+ """Returns a linkcode_resolve function for the given URL format
+
+ revision is a git commit reference (hash or name)
+
+ package is the name of the root module of the package
+
+ url_fmt is along the lines of ('https://github.com/USER/PROJECT/'
+ 'blob/{revision}/{package}/'
+ '{path}#L{lineno}')
+ """
+ revision = _get_git_revision()
@ogrisel Owner
ogrisel added a note

revision = _get_git_revision().decode('ascii') for instance.

@ogrisel Owner
ogrisel added a note

Actually it's probably better to fold the decode call into the _get_git_revision function itself.

@jnothman Owner

Thanks @ogrisel

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
+ return partial(_linkcode_resolve, revision=revision, package=package,
+ url_fmt=url_fmt)
Something went wrong with that request. Please try again.