Development workflow
Talk is cheap. Show me the code.
—Linus Torvalds [1]
Table of Contents
- Introduction
- How to send a patch
- Coding conventions in Sympy
- Workflow process
- Quick patch variant
- Reviewing patches
- References
In SymPy we encourage collaborative work.
Everyone is welcome to join and to implement new feature, fix some bug, give general advice, etc. Also, we try to discuss everything and to review each other's work so that many eyes can see more thus raising the quality.
General discussion takes place on sympy@googlegroups.com mailing list and in the issues, and the code is discussed in sympy-patches@googlegroups.com mailing list. Some discussion also takes place on IRC (our channel is #sympy at freenode).
As some of you already know, software development is not just coding. Many non-coding tasks have to be done in order to produce good code. For example: setting up infrastructure, designing, testing, documenting, assisting new developers (we are doing it here), and of course programming.
But even programming is not all about writing the code, it is about writing the code and preparing it so that the code can be included in the project.
Both producing the code and bringing it to the project are important parts of the game -- without the code there is nothing to bring in, and having the code outside is a no-win for anyone.
As already said above, we review changes. This idea was borrowed from successful projects like Linux, Python, SAGE and many more. In short, each change is first reviewed by other developers and only when it is approved is the code pushed in.
Like it takes effort to write good and clear code, reviewing other's work needs effort too. There are good practices how to do this so that reviewing is fun for both the author and the reviewer. We try to follow these good practices, and we'll try to show you how to follow them too.
When reviewing other's patches you learn a lot, so why not participate as a reviewer too? Anyone regardless of technical skill can help review code, and it's an excellent way for newcomers to learn about Sympy's development process and community.
License: New BSD License (see the LICENSE file for details) covers all files in the SymPy repository unless stated otherwise.
There are a few ways to create and send a patch.
The best way is to send a GitHub pull request against the sympy/sympy repository. We'll review it and push it in. The GitHub pull request is the preferred method, because it makes it easy for us to review and push the code in.
More quickly, but not convenient for reviewing and merging, is to create a patch-file using git alone. This way can be used if the patch has a high-priority or is significant, only one or two files are involved, or you don't have enough time to use the preferred method.
Although we are grateful for any improvements of Sympy, we strongly recommend you submit your patches as pull requests: this will greatly speed up the processing of the patch and ensure that it doesn't get forgotten due to inactivity.
The basic work-flow for both variants is a follows:
- Create your environment, if it was not created earlier.
- Create a new branch.
- Modify code and/or create tests of it.
- Be sure that all tests of SymPy pass.
- Only then commit changes.
- Create patch file, or pull request from GitHub.
All those are described in the details below Workflow process, but before you read that, it would be useful to acquaint yourself with Coding conventions in Sympy.
If you have any questions you can ask them on the mailinglist.
Follow the standard Style Guide for Python Code when writing code for SymPy, as explained at the following URLs:
In particular,
-
Use 4 spaces for indentation levels.
-
Use all lowercase function names with words separated by underscores. For example, you are encouraged to write Python functions using the naming convention
def set_some_value()
instead of the CamelCase convention.
-
Use CamelCase for class names and major functions that create objects, e.g.
class PolynomialRing(object)
Note, however, that some functions do have uppercase letters where it makes sense. For example, for matrices they are LUdecomposition or T (transposition) methods.
prime's docstring is an example of a well formatted docstring:
"""Return the nth prime. Primes are indexed as prime(1) = 2, prime(2) = 3, etc.... The nth prime is approximately n*log(n) and can never be larger than 2**n. See Also ======== sympy.ntheory.primetest.isprime, primerange, primepi References ========== .. [1] http://primes.utm.edu/glossary/xpage/BertrandsPostulate.html Examples ======== >>> from sympy import prime >>> prime(10) 29 >>> prime(1) 2 """
For more information see Writing documentation article on wiki.
Creating of environment is once-only.
To install git in Linux-like systems you can do it via your native package management system:
$ yum install git
or:
$ sudo apt-get install git
In Windows systems, first of all, install Python from:
http://python.org/download/
by downloading the "Python 2.7 Windows installer" (or Python 2.6 or 2.5) and running it. Then do not forget to add Python to the $PATH environment. And install msysgit:
http://code.google.com/p/msysgit/
In both Mac 10.4 and 10.5, you can install Git via MacPorts
$ sudo port install git
Git tracks who makes each commit by checking the user’s name and email. In addition, we use this info to associate your commits with your GitHub account.
To set these, enter the code below, replacing the name and email with your own (--global is optional).:
$ git config --global user.name "Firstname Lastname" $ git config --global user.email "your_email@youremail.com"
The name should be your actual name, not your GitHub username.
These global options (i.e. applying to all repositories) are placed in ~/.gitconfig. You can edit this file to add setup colors and some handy shortcuts:
[user]
name = Firstname Lastname
email = your_email@youremail.com
[color]
diff = auto
status= auto
branch= auto
interactive = true
[alias]
ci = commit
di = diff --color-words
st = status
co = checkout
log1 = log --pretty=oneline --abbrev-commit
logs = log --stat
It can be convenient in future to tune the bash prompt to display the current git branch.
The easiest way to do it, is to add the snippet below to your .bashrc or .bash_profile:
PS1="[\u@\h \W\$(git branch 2> /dev/null | grep -e '\* ' | sed 's/^..\(.*\)/{\1}/')]\$ "
But better is to use git-completion from the git source. This also has the advantage of adding tab completion to just about every git command. It also includes many other useful features, for example, promptings. To use git-completion, first download the git source code (about 27 MiB), then copy the file to your profile directory:
$ git clone git://git.kernel.org/pub/scm/git/git.git $ cp git/contrib/completion/git-completion.bash ~/.git-completion.sh
Read instructions in '~/.git-completion.sh'
Note that if you install git from the package manager in many Linux distros, this file is already installed for you. You can check if it is installed by seeing if tab completion works on git commands (try, e.g., git commi<TAB>, or git log --st<TAB>). You can also check if the PS1 commands work by doing something like:
$ PS1='\W$(__git_ps1 "%s")\$'
And your command prompt should change to something like:
sympy master$
Note, it is important to define your PS1 using single quotes ('), not double quotes ("), or else bash will not update the branch name.
As you are going to use GitHub you should have a GitHub account. If you have not one yet then sign up at:
Then create your own fork of the SymPy project (if you have not yet). Go to the SymPy GitHub repository:
and click the “Fork” button.
Now you have your own repository for the SymPy project. If your username in GitHub is mynick then the address of the forked project will look something like:
Some tools connect to GitHub without SSH. To use these tools properly you need to find and configure your API Token.
On GitHub, click “Account Settings” then “Account Admin.”
Enter the code below, replacing the mynick and 012-api-token with your own:
$ git config --global github.user mynick $ git config --global github.token 012-api-token
Note: if you ever change your GitHub password, a new token will be created and will need to be updated.
On your machine browse to where you would like to store SymPy, and clone (download) the latest code from SymPy's original repository (about 20 MiB):
$ git clone git://github.com/sympy/sympy.git $ cd sympy
Then assign your read-and-write repo to a remote called "github":
$ git remote add github git@github.com:mynick/sympy.git
For more information about GitHub forking and tuning see: [8], [9] and [11].
To establish a secure connection between your computer and GitHub see detailed instructions in [11].
If you have any problems with SSH access to GitHub, read the troubleshooting instructions at [12], or ask us in mail-list.
And now, do not forget to go to the Create separated branch instructions before modifying the code.
Typically, you will create a new branch to begin work on a new issue. Also pull request related with them.
A branch name should briefly describe the topic of the patch or pull request. If you know the issue number, then the branch name could be, for example, 1234_sequences. To create and checkout (that is, make it the working branch) a new branch
$ git branch 1234_sequences $ git checkout 1234_sequences
or in one command using
$ git checkout -b 1234_sequences
To view all branches, with your current branch highlighted, type:
$ git branch
And remember, never type the following commands in master: git merge, git commit, git rebase.
...
...
...
Do not forget that all new functionality should be tested, and all new methods, functions, and classes should have doctests showing how to use them.
Be sure that all tests of SymPy pass
To ensure everything stays in shape, let’s see if all tests pass:
$ ./bin/test $ ./bin/doctest
If they don't all pass, then the warning DO NOT COMMIT will appear.
Code quality (unwanted spaces and indents) are checked by ./bin/test utilities too. But you can separately run this test with the help of this command:
$ ./bin/test quality
If you have trailing whitespace it will show errors. This one will fix unwanted spaces.
$ ./bin/strip_whitespace <file>
If you want to test only one set of tests try:
$ ./bin/test sympy/concrete/tests/test_products.py
But remember that all tests should pass before committing.
You can check what files are changed:
$ git status
Add new files to the index if necessary:
$ git add new_file.py
Check total changes:
$ git diff
You are ready to commit changes locally. A commit also contains a commit message which describes it.
It is best if your commit messages are wrapped so that no line is longer than 80 characters. Also, the first line should be separated from the rest of the message by a blank line. This makes the message work the best with the various things in git and GitHub that show the message.
If your commit message is one line then you can use this command:
$ git commit -m "1234: sequences base implementation."
or if you plan to enter a fuller description:
$ git commit
An editor window will appear automatically in this case. In Linux, this is vim by default. You can change what editor pops up by changing the $EDITOR shell variable.
Also with the help of option -a you can tell the command commit to automatically stage files that have been modified and deleted, but new files you have not told git about will not be affected, e.g.,:
$ git commit -a -m "1234: sequences base implementation."
If you want to stage only part of your changes, you can use the interactive commit feature. Just type:
$ git commit --interactive
and choose the changes you want in the resulting interface.
Be sure that you are in your own branch, and run:
$ git push github 1234_sequences
This will send your local changes to your fork of the SymPy repository. Then navigate to your repository with the changes you want someone else to pull:
https://github.com/mynick/sympy
Select branch, and press the Pull Request button.
After pressing the Pull Request button, you are presented with a preview page where you can enter a title and optional description, see exactly what commits will be included when the pull request is sent, and also see who the pull request will be sent to:
If you’re sending from a topic branch, the title is pre-filled based on the name of the branch. Markdown is supported in the description, so you can embed images or use preformatted text blocks.
You can switch to the Commits tab to ensure that the correct set of changes is being sent. And review the diff of all changes by switching to the Files Changed.
Once you’ve entered the title and description, made any necessary customizations to the commit range, and reviewed the commits and file changes to be sent, press the Send pull request button.
The pull request is sent immediately. You’re taken to the main pull request discussion and review page. Additionally, all repository collaborators and followers will see an event in their dashboard.
That's all.
See also Updating your pull request
If after a time you need to make changes in pull request then the best way is to add a new commit in you local repository and simply repeat push command:
$ git commit -m "1234: some additional corrections." $ git push github 1234_sequences
Note that if you do any rebasing or in any way edit your commit history, you will have to add the -f (force) option to the push command for it to work:
$ git push -f github
Note, that those operations must be carried out with accuracy and with understanding what you do.
Your code changes and the history of them can be lost!
This is a frequent situation, when the branch is out of date with sympy/sympy repository.
Note, that after cloning a repository, it has a default remote called origin that points to the sympy/sympy repository. And your fork remote named as github. You can observe the remotes names with the help of this command:
$ git remote -v github git@github.com:mynick/sympy.git (fetch) github git@github.com:mynick/sympy.git (push) origin git://github.com/sympy/sympy.git (fetch) origin git://github.com/sympy/sympy.git (push)
As an example, consider that we have these commits in the master branch of local git repository:
A---B---C master
Then we have divergent branch 1234_sequences:
A---B---C master
\
a---b 1234_sequences
In the meantime the remote sympy/sympy master repository was updated too:
A---B---C---D origin/master
A---B---C master
\
a---b 1234_sequences
There are basically two ways to get up to date with a changed master: rebasing and merging. Rebasing is recommended for smaller branches, and is also the way to edit/squash commits (so you might as well do it if you are doing those too). For people who are new to git, we recommend rebasing for this reason.
For larger branches (for example, 20 commits or more), there are some advantages to merging instead of rebasing. Rebasing reapplies each commit iteratively over master, and if the state of the files changed by that commit is different from when it was originally made, the commit will change. This means what you can end up getting commits that are broken, or commits that do not do what they say they do (because the changes have been "rebased out"). This can lead to confusion if someone in the future tries to test something by checking out commits from the history.
Merging keeps everything intact. The commits you make are exactly the same, down to the SHA1 hash, which means that if you checkout a commit from a merged branch, it is exactly the same as checking it out from a non-merged branch. What it does instead is create a single commit, the merge commit, that makes it so that the history is both master and your branch. This commit contains all merge conflict resolution information, which is another advantage over rebasing (all merge conflict resolutions when rebasing are "sifted" into the commits that caused them, making them invisible).
Since this guide is aimed at new git users, you should be learning how to rebase. Also, git rebase has a very friendly interface (compared to git merge) that guides you through each action (editing commits, fixing merge conflicts, etc.). And, while the merge conflict resolutions will be hidden in the final result, they are easier to apply with git rebase because you only have to apply the resolution for one commit at a time, instead of the whole branch at once.
The final aim, that we want to obtain is:
A---B---C---D master
\
a---b 1234_sequences
The way to do it is first of all to merge local repository with the remote sympy/sympy:
$ git checkout master $ git pull
So we obtain:
A---B---C---D master
\
a---b 1234_sequences
Then:
$ git checkout 1234_sequences $ git rebase master
Note that this last one will require you to fix some merge conflicts if there are changes to the same file in master and 1234_sequences. Open the file that it tells you is wrong, fix the code with >>> and <<< around it to what it should be.
Then be sure that all tests pass:
$ ./bin/test $ ./bin/doctest
Then do:
$ git add sympy/matrices/your_conflict_file $ git rebase --continue
(git rebase will also guide you in this).
If the task consist just in to edit commit messages then it might be more simple rebasing usage. Consider these commit messages:
$ git log --oneline 7bbbc06 bugs fixing 4d6137b some additional corrections. 925d88fx sequences base implementation.
Then run rebase command in interactive mode:
$ git rebase --interactive 925d88fx
Or you can use other ways to point to commits, e.g. `git rebase --interactive HEAD^^` or `git rebase --interactive HEAD~2`.
A new editor window will appear (note that order is reversed with respect to the git log command):
pick 4d6137b some additional corrections. pick 7bbbc06 bugs fixing # Rebase 925d88f..7bbbc06 onto 925d88f # # Commands: # p, pick = use commit # r, reword = use commit, but edit the commit message # e, edit = use commit, but stop for amending # s, squash = use commit, but meld into previous commit # f, fixup = like "squash", but discard this commit's log message
Change pick to reword (or on old versions of git, to edit) for those that you want to edit and save that file.
After that, git will drop you back into your editor for every commit you want to reword, and into the shell for every commit you wanted to edit:
$ (Change the commit in any way you like.) $ git commit --amend "your new message" $ git rebase --continue
Most of this sequence will be explained to you by the output of the various commands of git. Continue until it says:
Successfully rebased and updated refs/heads/master.
Note, that for people who are new to git, we recommend rebasing for the above reasons.
Nevertheless, first merge your local repository with the remote:
$ git checkout master $ git pull
This results in:
A---B---C---D master
\
a---b 1234_sequences
Then merge your 1234_sequences branch from 1234_sequences:
$ git checkout 1234_sequences $ git merge master
Probably the last command will tell you that conflicts must be solved for a few indicated files.
If that's the case then the marks >>> and <<< will appear at those files. Fix the code with >>> and <<< around it to what it should be. You must manually remove useless pieces, and leave only new changes from your branch.
Then be sure that all tests pass:
$ ./bin/test $ ./bin/doctest
and commit:
$ git commit -m "Merging with master"
So the result will be like that (automatic merging c):
A---B---C-------D master
\ \
a---b---c 1234_sequences
Note that the GitHub variant is highly preferred, and will increase your chances of the patch not being lost or forgotten about.
First clone the read-only SymPy repository:
$ git clone git://github.com/sympy/sympy.git $ cd sympy
then you can create a new branch for your code modifications.
Also take into account all which was written for the GitHub procedure about git tuning, code conversions, the testing and the commit operations.
Then create a patch-file:
$ git format-patch master --stdout > 1234_sequences_base_implementation.patch
And send us it.
Coding's only half the battle in software development: our code also has to be thoroughly reviewed before release. Reviewers thus are an integral part of the development process. Note that you do not have to have any special pull or other privileges to review patches: anyone with Python on his/her computer can review.
Pull requests (the preferred avenue for patches) for sympy are located here. Feel free to view any open pull request. Each contains a Discussion section for comments, Commits section for viewing the author's commit files and documentation, and Diff section for viewing all the changes in code. To browse the raw code files for a commit, select a commit in the Commits section and click on the "View file" link to view a file.
Based on your level of expertise, there are two ways to participate in the review process: manually running tests and using sympy-bot. Whichever option you choose, you should also make sure that the committed code complies with the Writing documentation guidelines.
If you prefer to test code manually, you will first have to set up your environment as described in the Workflow process section. Then, you need to obtain the patched files. If you're reviewing a pull request, you should get the requested branch into your sympy folder. Go into your folder and execute (<username> being the username of the pull requester and <branchname> being the git branch of the pull request):
$ git remote add <username> git://github.com/<username>/sympy.git $ git fetch <username> $ git checkout -b <branchname> <username>/<branchname>
After obtaining the pull request or patch, go to your sympy root directory and execute:
$ ./bin/test $ ./bin/doctest
If there are any problems, notify the author in the pull request by commenting.
A good option for both new and veteran code reviewers is the sympy-bot program, which automatically tests code in pull requests and posts the results in the appropriate pull requests. To run sympy-bot, download the archive and decompress it. Go into the folder and execute:
$ ./sympy-bot list
This will list all open pull requests on GitHub. To review a pull request and run all tests, execute:
$ ./sympy-bot review <pullrequest_number>
Alternatively, to review all open pull requests, execute:
$ ./sympy-bot review all
The review command will post the results of all tested pull requests in the appropriate Github page. For more information on sympy-bot, visit the readme.
A pull request or patch must meet the following requirements during review before being considered as ready for release.
-
- All tests must pass.
-
- Rationale: We need to make sure we're not releasing buggy code.
- If new features are being implemented and/or bug fixes are added, tests should be added for them as well.
-
- The reviews (at least 1) must all be positive.
-
- Rationale: We'd like everyone to agree on the merits of the patch.
- If there are conflicting opinions, the reviewers should reach a consensus.
-
- The patch must have been posted for at least 24 hours.
-
- Rationale: This gives a chance for everyone to look at the patch.
This page is based upon present SymPy pages [2-6], GitHub help [8-9], [11-12] and inspired by Sage guide [10]:
| [1] | http://lkml.org/lkml/2000/8/25/132 |
| [2] | http://docs.sympy.org/dev/sympy-patches-tutorial.html#quick-start |
| [3] | http://sympy.org/development.html |
| [4] | https://github.com/sympy/sympy/wiki |
| [5] | https://github.com/sympy/sympy/wiki/Pushing-patches |
| [6] | https://github.com/sympy/sympy/wiki/Getting-the-bleeding-edge |
| [7] | https://github.com/sympy/sympy/wiki/Git-hg-rosetta-stone |
| [8] | http://help.github.com/pull-requests/ |
| [9] | http://help.github.com/fork-a-repo/ |
| [10] | http://sagemath.org/doc/developer/ |
| [11] | (1, 2) http://help.github.com/linux-set-up-git/ |
| [12] | http://help.github.com/troubleshooting-ssh/ |