Skip to content
This repository
Newer
Older
100644 328 lines (249 sloc) 15.88 kb
2f49d709 » rgommers
2012-03-28 DOC: add document on how to contribute to SciPy.
1 Contributing to SciPy
2 =====================
3
4 This document aims to give an overview of how to contribute to SciPy. It
5 tries to answer commonly asked questions, and provide some insight into how the
6 community process works in practice. Readers who are familiar with the SciPy
7 community and are experienced Python coders may want to jump straight to the
8 `git workflow`_ documentation.
9
10
11 Contributing new code
12 ---------------------
13
14 If you have been working with the scientific Python toolstack for a while, you
15 probably have some code lying around of which you think "this could be useful
16 for others too". Perhaps it's a good idea then to contribute it to SciPy or
17 another open source project. The first question to ask is then, where does
18 this code belong? That question is hard to answer here, so we start with a
19 more specific one: *what code is suitable for putting into SciPy?*
20 Almost all of the new code added to scipy has in common that it's potentially
21 useful in multiple scientific domains and it fits in the scope of existing
22 scipy submodules. In principle new submodules can be added too, but this is
23 far less common. For code that is specific to a single application, there may
24 be an existing project that can use the code. Some scikits (`scikit-learn`_,
25 `scikits-image`_, `statsmodels`_, etc.) are good examples here; they have a
26 narrower focus and because of that more domain-specific code than SciPy.
27
28 Now if you have code that you would like to see included in SciPy, how do you
8c9a0464 » rgommers
2012-04-15 DOC: add section on licensing issues.
29 go about it? After checking that your code can be distributed in SciPy under a
30 compatible license (see FAQ for details), the first step is to discuss on the
31 scipy-dev mailing list. All new features, as well as changes to existing code,
32 are discussed and decided on there. You can, and probably should, already
33 start this discussion before your code is finished.
2f49d709 » rgommers
2012-03-28 DOC: add document on how to contribute to SciPy.
34
35 Assuming the outcome of the discussion on the mailing list is positive and you
36 have a function or piece of code that does what you need it to do, what next?
37 Before code is added to SciPy, it at least has to have good documentation, unit
38 tests and correct code style.
39
40 1. Unit tests
41 In principle you should aim to create unit tests that exercise all the code
8a7327b3 » rgommers
2012-04-10 DOC: fix some typos in contribute howto to address @jakevdp's review …
42 that you are adding. This gives some degree of confidence that your code
2f49d709 » rgommers
2012-03-28 DOC: add document on how to contribute to SciPy.
43 runs correctly, also on Python versions and hardware or OSes that you don't
44 have available yourself. An extensive description of how to write unit
45 tests is given in the NumPy `testing guidelines`_.
46
47 2. Documentation
48 Clear and complete documentation is essential in order for users to be able
49 to find and understand the code. Documentation for individual functions
50 and classes -- which includes at least a basic description, type and
84036581 » rgommers
2012-04-17 DOC: add link to iPYMOTW doctest description. Also fix a typo.
51 meaning of all parameters and returns values, and usage examples in
52 `doctest`_ format -- is put in docstrings. Those docstrings can be read
53 within the interpreter, and are compiled into a reference guide in html and
54 pdf format. Higher-level documentation for key (areas of) functionality is
55 provided in tutorial format and/or in module docstrings. A guide on how to
56 write documentation is given in `how to document`_.
2f49d709 » rgommers
2012-03-28 DOC: add document on how to contribute to SciPy.
57
58 3. Code style
59 Uniformity of style in which code is written is important to others trying
60 to understand the code. SciPy follows the standard Python guidelines for
61 code style, `PEP8`_. In order to check that your code conforms to PEP8,
62 you can use the `pep8 package`_ style checker. Most IDEs and text editors
63 have settings that can help you follow PEP8, for example by translating
64 tabs by four spaces. Using `pyflakes`_ to check your code is also a good
65 idea.
66
67 At the end of this document a checklist is given that may help to check if your
68 code fulfills all requirements for inclusion in SciPy.
69
70 Another question you may have is: *where exactly do I put my code*? To answer
dce5c1c2 » rgommers
2012-05-27 DOC: add info on setting up scipy for developing to contributing HOWTO.
71 this, it is useful to understand how the SciPy public API (application
72 programming interface) is defined. For most modules the API is two levels
73 deep, which means your new function should appear as
74 ``scipy.submodule.my_new_func``. ``my_new_func`` can be put in an existing or
75 new file under ``/scipy/<submodule>/``, its name is added to the ``__all__``
befe60d4 » smcquay
2012-11-04 Fixed minor typo.
76 list in that file (which lists all public functions in the file), and those
2f49d709 » rgommers
2012-03-28 DOC: add document on how to contribute to SciPy.
77 public functions are then imported in ``/scipy/<submodule>/__init__.py``. Any
78 private functions/classes should have a leading underscore (``_``) in their
79 name. A more detailed description of what the public API of SciPy is, is given
80 in `SciPy API`_.
81
82 Once you think your code is ready for inclusion in SciPy, you can send a pull
83 request (PR) on Github. We won't go into the details of how to work with git
84 here, this is described well in the `git workflow`_ section of the NumPy
85 documentation and in the Github help pages. When you send the PR for a new
86 feature, be sure to also mention this on the scipy-dev mailing list. This can
87 prompt interested people to help review your PR. Assuming that you already got
88 positive feedback before on the general idea of your code/feature, the purpose
89 of the code review is to ensure that the code is correct, efficient and meets
90 the requirements outlined above. In many cases the code review happens
91 relatively quickly, but it's possible that it stalls. If you have addressed
92 all feedback already given, it's perfectly fine to ask on the mailing list
93 again for review (after a reasonable amount of time, say a couple of weeks, has
94 passed). Once the review is completed, the PR is merged into the "master"
95 branch of SciPy.
96
8a7327b3 » rgommers
2012-04-10 DOC: fix some typos in contribute howto to address @jakevdp's review …
97 The above describes the requirements and process for adding code to SciPy. It
98 doesn't yet answer the question though how decisions are made exactly. The
99 basic answer is: decisions are made by consensus, by everyone who chooses to
100 participate in the discussion on the mailing list. This includes developers,
101 other users and yourself. Aiming for consensus in the discussion is important
102 -- SciPy is a project by and for the scientific Python community. In those
103 rare cases that agreement cannot be reached, the `maintainers`_ of the module
104 in question can decide the issue.
2f49d709 » rgommers
2012-03-28 DOC: add document on how to contribute to SciPy.
105
106
107 Contributing by helping maintain existing code
108 ----------------------------------------------
109
110 The previous section talked specifically about adding new functionality to
111 SciPy. A large part of that discussion also applies to maintenance of existing
112 code. Maintenance means fixing bugs, improving code quality or style,
ee502782 » rgommers
2012-04-14 DOC: fix two more typos, and add "add missing unit tests" as separate
113 documenting existing functionality better, adding missing unit tests, keeping
114 build scripts up-to-date, etc. The SciPy `Trac`_ bug tracker contains all
115 reported bugs, build/documentation issues, etc. Fixing issues described in
116 Trac tickets helps improve the overall quality of SciPy, and is also a good way
117 of getting familiar with the project. You may also want to fix a bug because
118 you ran into it and need the function in question to work correctly.
119
120 The discussion on code style and unit testing above applies equally to bug
121 fixes. It is usually best to start by writing a unit test that shows the
2f49d709 » rgommers
2012-03-28 DOC: add document on how to contribute to SciPy.
122 problem, i.e. it should pass but doesn't. Once you have that, you can fix the
123 code so that the test does pass. That should be enough to send a PR for this
124 issue. Unlike when adding new code, discussing this on the mailing list may
125 not be necessary - if the old behavior of the code is clearly incorrect, no one
126 will object to having it fixed. It may be necessary to add some warning or
127 deprecation message for the changed behavior. This should be part of the
128 review process.
129
130
131 Other ways to contribute
132 ------------------------
133
134 There are many ways to contribute other than contributing code. Participating
135 in discussions on the scipy-user and scipy-dev *mailing lists* is a contribution
136 in itself. The `scipy.org`_ *website* contains a lot of information on the
137 SciPy community and can always use a new pair of hands. A redesign of this
138 website is ongoing, see `scipy.github.com`_. The redesigned website is a
139 static site based on Sphinx, the sources for it are
140 also on Github at `scipy.org-new`_.
141
dce5c1c2 » rgommers
2012-05-27 DOC: add info on setting up scipy for developing to contributing HOWTO.
142 The SciPy *documentation* is constantly being improved by many developers and
143 users. You can contribute by sending a PR on Github that improves the
144 documentation, but there's also a `documentation wiki`_ that is very convenient
145 for making edits to docstrings (and doesn't require git knowledge). Anyone can
146 register a username on that wiki, ask on the scipy-dev mailing list for edit
147 rights and make edits. The documentation there is updated every day with the
148 latest changes in the SciPy master branch, and wiki edits are regularly
149 reviewed and merged into master. Another advantage of the documentation wiki
150 is that you can immediately see how the reStructuredText (reST) of docstrings
151 and other docs is rendered as html, so you can easily catch formatting errors.
2f49d709 » rgommers
2012-03-28 DOC: add document on how to contribute to SciPy.
152
153 Code that doesn't belong in SciPy itself or in another package but helps users
154 accomplish a certain task is valuable. `SciPy Central`_ is the place to share
155 this type of code (snippets, examples, plotting code, etc.).
156
157
158 Useful links, FAQ, checklist
159 ----------------------------
160
161 Checklist before submitting a PR
162 ````````````````````````````````
163
164 - Are there unit tests with good code coverage?
165 - Do all public function have docstrings including examples?
166 - Is the code style correct (PEP8, pyflakes)
cd7184e7 » rgommers
2012-04-14 DOC: add explanation about version number to use with versionadded di…
167 - Is the new functionality tagged with ``.. versionadded:: X.Y.Z`` (with
168 X.Y.Z the version number of the next release - can be found in setup.py)?
169 - Is the new functionality mentioned in the release notes of the next
170 release?
2f49d709 » rgommers
2012-03-28 DOC: add document on how to contribute to SciPy.
171 - Is the new functionality added to the reference guide?
172 - In case of larger additions, is there a tutorial or more extensive
173 module-level description?
174 - In case compiled code is added, is it integrated correctly via setup.py
175 (and preferably also Bento/Numscons configuration files)?
176 - If you are a first-time contributor, did you add yourself to THANKS.txt?
177 Please note that this is perfectly normal and desirable - the aim is to
178 give every single contributor credit, and if you don't add yourself it's
8a7327b3 » rgommers
2012-04-10 DOC: fix some typos in contribute howto to address @jakevdp's review …
179 simply extra work for the reviewer (or worse, the reviewer may forget).
8c9a0464 » rgommers
2012-04-15 DOC: add section on licensing issues.
180 - Did you check that the code can be distributed under a BSD license?
2f49d709 » rgommers
2012-03-28 DOC: add document on how to contribute to SciPy.
181
182
183 Useful SciPy documents
184 ``````````````````````
185
186 - The `how to document`_ guidelines
187 - NumPy/SciPy `testing guidelines`_
188 - `SciPy API`_
189 - SciPy `maintainers`_
190 - NumPy/SciPy `git workflow`_
191
192
193 FAQ
194 ```
195
8c9a0464 » rgommers
2012-04-15 DOC: add section on licensing issues.
196 *I based my code on existing Matlab/R/... code I found online, is this OK?*
197
198 It depends. SciPy is distributed under a BSD license, so if the code that you
199 based your code on is also BSD licensed or has a BSD-compatible license (MIT,
200 Apache, ...) then it's OK. Code which is GPL-licensed, has no clear license,
201 requires citation or is free for academic use only can't be included in SciPy.
84036581 » rgommers
2012-04-17 DOC: add link to iPYMOTW doctest description. Also fix a typo.
202 Therefore if you copied existing code with such a license or made a direct
8c9a0464 » rgommers
2012-04-15 DOC: add section on licensing issues.
203 translation to Python of it, your code can't be included. See also `license
204 compatibility`_.
205
206
dce5c1c2 » rgommers
2012-05-27 DOC: add info on setting up scipy for developing to contributing HOWTO.
207 *How do I set up SciPy so I can edit files, run the tests and make commits?*
208
209 The simplest method is setting up an in-place build. To create your local git
210 repo and do the in-place build::
211
212 $ git clone https://github.com/scipy/scipy.git scipy
213 $ cd scipy
214 $ python setup.py build_ext -i
215
216 Then you need to either set up a symlink in your site-packages or add this
217 directory to your PYTHONPATH environment variable, so Python can find it. Some
218 IDEs (Spyder for example) have utilities to manage PYTHONPATH. On Linux and OS
219 X, you can for example edit your .bash_login file to automatically add this dir
220 on startup of your terminal. Add the line::
221
222 export PYTHONPATH="$HOME/scipy:${PYTHONPATH}"
223
224 Alternatively, to set up the symlink, use (prefix only necessary if you want to
225 use your local instead of global site-packages dir)::
226
227 $ python setupegg.py develop --prefix=${HOME}
228
229 To test that everything works, start the interpreter (not inside the scipy/
230 source dir) and run the tests::
231
232 $ python
233 >>> import scipy as sp
234 >>> sp.test()
235
236 Now editing a Python source file in SciPy allows you to immediately test and
237 use your changes, by simply restarting the interpreter.
238
239 Note that while the above procedure is the most straightforward way to get
240 started, you may want to look into using Bento or numscons for faster and more
241 flexible building, or virtualenv to maintain development environments for
242 multiple Python versions.
243
244
245 *How do I set up a development version of SciPy in parallel to a released
246 version that I use to do my job/research?*
247
248 One simple way to achieve this is to install the released version in
249 site-packages, by using a binary installer or pip for example, and set up the
250 development version with an in-place build in a virtualenv. First install
251 `virtualenv`_ and `virtualenvwrapper`_, then create your virtualenv (named
252 scipy-dev here) with::
253
254 $ mkvirtualenv scipy-dev
255
256 Now, whenever you want to switch to the virtual environment, you can use the
257 command ``workon scipy-dev``, while the command ``deactivate`` exits from the
258 virtual environment and brings back your previous shell. With scipy-dev
259 activated, follow the in-place build with the symlink install above to actually
260 install your development version of SciPy.
261
262
2f49d709 » rgommers
2012-03-28 DOC: add document on how to contribute to SciPy.
263 *Can I use a programming language other than Python to speed up my code?*
264
265 Yes. The languages used in SciPy are Python, Cython, C, C++ and Fortran. All
266 of these have their pros and cons. If Python really doesn't offer enough
267 performance, one of those languages can be used. Important concerns when
268 using compiled languages are maintainability and portability. For
269 maintainability, Cython is clearly preferred over C/C++/Fortran. Cython and C
270 are more portable than C++/Fortran. A lot of the existing C and Fortran code
271 in SciPy is older, battle-tested code that was only wrapped in (but not
272 specifically written for) Python/SciPy. Therefore the basic advice is: use
273 Cython. If there's specific reasons why C/C++/Fortran should be preferred,
274 please discuss those reasons first.
275
276
277 *There's overlap between Trac and Github, which do I use for what?*
278
279 Trac_ is the bug tracker, Github_ the code repository. Before the SciPy code
280 repository moved to Github, the preferred way to contribute code was to create
281 a patch and attach it to a Trac ticket. The overhead of this approach is much
282 larger than sending a PR on Github, so please don't do this anymore. Use Trac
283 for bug reports, Github for patches.
284
285
286 .. _scikit-learn: http://scikit-learn.org
287
288 .. _scikits-image: http://scikits-image.org/
289
290 .. _statsmodels: http://statsmodels.sourceforge.net/
291
292 .. _testing guidelines: https://github.com/numpy/numpy/blob/master/doc/TESTS.rst.txt
293
294 .. _how to document: https://github.com/numpy/numpy/blob/master/doc/HOWTO_DOCUMENT.rst.txt
295
296 .. _PEP8: http://www.python.org/dev/peps/pep-0008/
297
298 .. _pep8 package: http://pypi.python.org/pypi/pep8
299
300 .. _pyflakes: http://pypi.python.org/pypi/pyflakes
301
302 .. _SciPy API: http://docs.scipy.org/doc/scipy/reference/api.html
303
304 .. _git workflow: http://docs.scipy.org/doc/numpy/dev/gitwash/index.html
305
33135864 » dlax
2012-06-04 DOC: fix URL of MAINTAINERS.rst.txt in HACKING.rst.txt
306 .. _maintainers: https://github.com/scipy/scipy/blob/master/doc/MAINTAINERS.rst.txt
2f49d709 » rgommers
2012-03-28 DOC: add document on how to contribute to SciPy.
307
308 .. _Trac: http://projects.scipy.org/scipy/timeline
309
310 .. _Github: https://github.com/scipy/scipy
311
312 .. _scipy.org: http://scipy.org/
313
314 .. _scipy.github.com: http://scipy.github.com/
315
316 .. _scipy.org-new: https://github.com/scipy/scipy.org-new
317
318 .. _documentation wiki: http://docs.scipy.org/scipy/Front%20Page/
319
320 .. _SciPy Central: http://scipy-central.org/
8c9a0464 » rgommers
2012-04-15 DOC: add section on licensing issues.
321
322 .. _license compatibility: http://www.scipy.org/License_Compatibility
84036581 » rgommers
2012-04-17 DOC: add link to iPYMOTW doctest description. Also fix a typo.
323
324 .. _doctest: http://www.doughellmann.com/PyMOTW/doctest/
dce5c1c2 » rgommers
2012-05-27 DOC: add info on setting up scipy for developing to contributing HOWTO.
325
326 .. _virtualenv: http://www.virtualenv.org/
327
328 .. _virtualenvwrapper: http://www.doughellmann.com/projects/virtualenvwrapper/
329
Something went wrong with that request. Please try again.