Skip to content

99. Developer

Jump to bottom
github-actions[bot] edited this page Mar 14, 2024 · 2 revisions

tl;dr

After developing a change locally, ensure that both lints and tests pass. Commits should be squashed into logical units, and all commits must be signed (e.g. with the -s git flag). csp requires Developer Certificate of Origin for all contributions.

If your work is still in-progress, open a draft pull request. Otherwise, open a normal pull request. It might take a few days for a maintainer to review and provide feedback, so please be patient. If a maintainer asks for changes, please make said changes and squash your commits if necessary. If everything looks good to go, a maintainer will approve and merge your changes for inclusion in the next release.

Please note that non substantive changes, large changes without prior discussion, etc, are not accepted and pull requests may be closed.

Setting up a development environment

To work on csp, you are going to need to build it from source. See https://github.com/Point72/csp/wiki/98.-Building-From-Source for detailed build instructions.

Once you've built csp from a git clone, you will also need to configure git and your GitHub account for csp development.

Configuring Git and GitHub for Development

Create your fork

The first step is to create a personal fork of csp. To do so, click the "fork" button at https://github.com/Point72/csp, or just navigate here in your browser. Set the owner of the repository to your personal GitHub account if it is not already set that way and click "Create fork".

Configure remotes

Next, you should set some names for the git remotes corresponding to main Point72 repository and your fork. If you started with a clone of the main Point72 repository, you could do something like:

cd csp
git remote rename origin upstream

# for SSH authentication
git remote add origin git@github.com:<username>/csp.git

# for HTTP authentication
git remote add origin https://github.com/<username>/csp.git

Authenticating with GitHub

If you have not already configured ssh access to GitHub, you can find instructions to do so here, including instructions to create an SSH key if you have not done so. Authenticating with SSH is usually the easiest route. If you are working in an environment that does not allow SSH connections to GitHub, you can look into configuring a hardware passkey or adding a personal access token to avoid the need to type in your password every time you push to your fork.

Configure commit signing

Additionally, you will need to configure your local git setup and GitHub account to use commit signing. All commits to the csp repository must be signed to increase the difficulty of a supply-chain attack against the csp codebase. The easiest way to do this is to configure git to sign commits with your SSH key. You can also use a GPG key to sign commits.

In either case, you must also add your public key to your github account as a signing key. Note that if you have already added an SSH key as an authentication key, you will need to add it again as a signing key.

Github for maintainers

Triaging Issues

The bug tracker is both a venue for users to communicate to the developers about defects and a database of known defects. It's up to the maintainers to ensure issues are high quality.

We have a number of labels that can be applied to sort issues into categories. If you notice newly created issues that are poorly labeled, consider adding or removing some labels that do not apply to the issue.

The issue template encourages users to write bug reports that clearly describe the problem they are having and include steps to reproduce the issue. However, users sometimes ignore the template or are not used to GitHub and make mistakes in formatting or communication.

If you are able to infer what they meant and are able to understand the issue, feel free to edit their issue description to fix formatting or correct issues with a script demonstrating the issue.

If there is still not enough information or if the issue is unclear, request more information from the submitter. If they do not respond or do not clarify sufficiently, close the issue. Try to be polite and have empathy for inexperienced issue authors.

How to check out a PR locally

This workflow is described in the GitHub docs.

  1. Identify the pull request ID. This is the number of the pull request in the GitHub UI, which shows up in the URL for the pull request. For example, https://github.com/Point72/csp/pull/98 has PR ID 98.

  2. Fetch the pull request ref and assign it to a local branch name.

    git fetch upstream pull/<ID>/HEAD/:LOCAL_BRANCH_NAME

    where <ID> is the PR ID number and LOCAL_BRANCH_NAME is a name chosen for the PR branch in your local checkout of csp.

  3. Switch to the PR branch

    git switch LOCAL_BRANCH_NAME

  4. Rebuild csp

Pushing Fixups to Pull Requests

Sometimes pull requests don't quite make it across the finish line. In cases where only a small fixup is required to make a PR mergeable and the author of the pull request is unresponsive to requests, the best course of action is often to push to the pull request directly to resolve the issues.

To do this, check out the pull request locally using the above instructions. Then make the changes needed for the pull request and push the local branch back to GitHub:

git push upstream LOCAL_BRANCH_NAME

Where LOCAL_BRANCH_NAME is the name you gave to the PR branch when you fetched it from GitHub.

Note that if the user who created the pull request selected the option to forbid pushes to their pull request, you will instead need to recreate the pull request by pushing the PR branch to your fork and making a pull request like normal.

Release Manual

Doing a "normal" release

Making regular releases allows users to test code as it gets added and allows developers to quickly get feedback on changes and new features. A release announcement is the primary way projects communicate with the majority of users, so you should think of the release and accompanying notes as a way to communicate expectations for users.

Choosing a version number

The clearest signal you can send to users about what to expect from a release is the version number you choose. Most users expect version numbers to follow semantic versioning or semver, where the version number indicates the sorts of changes to expect.

With semver, There are three kinds of releases, each of which have a different potential impact on users.

  • Patch release

    This is the most common kind of release. A patch release should only include fixes for bugs or other changes that cannot impact code a user writes with the csp package. A user should be able to safely upgrade csp from the previous version to a new patch release with no changes to the output of their code and no new errors being raised, except for fixed bugs. Whether or not a bug fix is sufficiently impactful to break backward compatibility is a judgement call. It is best to err on the side of safety and do a major release if you are unsure.

  • Minor releases

    This indicates a new feature has been added to the library in a backward incompatible. Minor releases can include bugfixes as well but should not include backward incompatible changes. For example, adding a new keyword argument to a Python function is a backward compatible change but removing an argument or changing its name is backwards incompatible.

  • Major releases

    A major release indicates that there are changes in the release that are not backward compatible, and users may need to update their code to accommodate the change. When possible, efforts should be made to communicate to users how to migrate their code.

Note that the primary concern is user impact. Sometimes a bug fix is so disruptive to users that fixing it qualifies as a major change. Sometimes a new feature is so big that it fundamentally changes the nature of the library and it deserves a major release to indicate that, even if there are no breaking changes. Sometimes a change is breaking, but only in a very unlikely scenario that can be safely ignored in practice. It is best to use your good judgement when choosing a new version number and not slavishly follow rules. Be empathetic to your users.

Preparing and tagging a release

Follow these steps when it's time to tag a new release. Before doing this, you will need to ensure bump2version is installed into your development environment.

  1. Ensure your local clone of csp is synced up with GitHub, including any tags that have been pushed since you last synced:

    git pull upstream main --tags

  2. Make a branch and update version numbers in your local clone using the bump2version integration in the Makefile.

    First, make a branch that will be pushed to the main csp repository. Using a name like release/v0.3.4 should avoid any conflicts and make it clear what the branch is for.

    git checkout -b release/v0.x.x

    For example, for a bugfix release, bump2version will automatically update the codebase to use the next bugfix version number if you do:

    make patch

    Similarly, make minor and make major will update the version numbers for minor and major releases, respectively. Double-check that the version numbers have been updated correctly with git diff, and then git commit the change.

  3. Push your branch to GitHub, and trigger a "full" test run on the branch.

    Navigate to the GitHub Actions "build status" workflow. Click the white "Run workflow" button, make sure the "Run full CI" radio button is selected, and click the green "Run workflow" button to launch the test run.

  4. Propose a pull request from the branch containing the version number updated. Add a link to the successful full test run for reviewers.

  5. Review and merge the pull request. Make sure you delete the branch afterwards.

  6. Tag the release

    Use the version number bump2version generated and make sure the tag name begins with a v.

    git tag v0.2.0

  7. Push the tag to GitHub

    git push upstream main --follow-tags

    You will need access in the repository settings to be able to push tags to the repository. Access to merge pull requests is not sufficient to overcome the tag protection settings in the repository.

Pushing a tag name that begins with v should automatically trigger a full test run that will generate a GitHub release. You can follow along with this run in the GitHub actions interface. There should be two actions running, one for the push to main and one for the new tag. You want to inspect the action running for the new tag. Once the run finishes, there should be a new release on the "Releases" page.

Releasing to PyPI

A developer's first release

If this is your first release, you will need an account on pypi.org and your account will need to be added as a maintainer to the csp project on pypi. You will also need to have two factor authentication enabled on your PyPI account.

Once that is set up, navigate to the API token page in your PyPI settings and generate an API token scoped to the csp project. Do not navigate away from the page displaying the API token before the next step.

Optionally, you can create an account on test.pypi.org if you would like to do a dry run of the release. You will also need to set up an API token on test.pypi.org.

Create a .pypirc file in your home directory, and add the following content:

[distutils]
  index-servers =
    testpypi
    csp
[testpypi]
  username = __token__
  password = <your API key, including pypi- prefix>
[csp]
  repository = https://upload.pypi.org/legacy/
  username = __token__
  password = <your API key, including pypi- prefix>

Doing the release

Download release artifacts from github actions

Make sure you are in the root of the csp repository and execute the following commands.

rm -r dist
mkdir dist
cd dist
curl -sL https://api.github.com/repos/Point72/csp/releases/latest | grep browser_download_url | cut -d '"' -f 4 | xargs -I '{}' curl -LO {}
cd ..

You should end up with a copy of the wheel files and source distribution associated with the GitHub release. You should verify that all files were successfully downloaded. Currently there are 6 MacOS wheels and 4 linux wheels. There should only be one source distribution.

Optionally, you can lint the release artifacts with

twine check --strict dist/*

This happens as part of the CI so this should only be a double-check.

Optionally upload to testpypi to test "pip install"
twine upload --repository testpypi dist/*

You can check that the wheel installs correctly with pip from testpypi like so:

pip install --index-url https://test.pypi.org/simple --extra-index-url https://pypi.org/simple csp

Note that extra-index-url is necessary to ensure downloading dependencies succeeds.

Upload to pypi

If you are sure the release is ready, you can upload to pypi like so:

twine upload --repository csp dist/*`

Note that this assumes you have a .pypirc set up as explained above.

Assuming the upload completes successfully, you should see a message like

View at:
https://pypi.org/project/csp/<version number>/

Dealing with release mistakes

Sometimes releases go wrong. Here is what to do when that happens. This blog post from Brett Cannon covers how to deal with various kinds of release mistakes on PyPI. Some things to remember after reading that post:

  • Completely broken releases should be yanked.
    • A yanked release is not permanently deleted
  • Problems with metadata (e.g. the readme or metadata in pyproject.toml) can be dealt with by creating a post release.
  • A problem with a wheel binary can be fixed by simply replacing the wheel file with a new wheel that has an incremented build number with twine upload.
  • If private data is published accidentally, a release can be permanently deleted. This should only be used as a last resort option.

Clone this wiki locally