Skip to content

about:workflow

Jump to bottom
Jonathan Perkin edited this page Apr 29, 2024 · 19 revisions

Contents

  1. Introduction
  2. Repositories
    1. pkgsrc
    2. pkgsrc-joyent
    3. pkgsrc-wip
  3. Repository Flows
  4. Branches
    1. pkgsrc-YYYYQQ
    2. backports
    3. ctf
    4. miscfix
    5. multiarch
    6. pbulk
    7. pbulkmulti
    8. release
  5. Creating A New Release Branch
  6. Handling Forced Updates
  7. FAQ
    1. Find which repository a package came from

Introduction

Our pkgsrc releases are made up from a number of different repositories and branches, which can be confusing for newcomers. This document aims to explain the relationships clearly and make it easy for users to know where issues and pull requests should go.

Repositories

We have three main repositories, though two of them are themselves forks of upstream repositories:

pkgsrc

This is the primary repository, located at https://github.com/TritonDataCenter/pkgsrc/, and contains the majority of packages. It is a downstream fork of the official https://github.com/netbsd/pkgsrc/ repository.

pkgsrc-joyent

This is a secondary repository, located at https://github.com/TritonDataCenter/pkgsrc-joyent/. This repository contains a number of packages that are not (yet) suitable for committing either to pkgsrc or pkgsrc-wip, mostly because they have specific requirements.

pkgsrc-wip

This is another secondary repository, located at https://github.com/TritonDataCenter/pkgsrc-wip/, and based on the pkgsrc-wip project. This repository contains packages that are considered as not yet ready for inclusion into the main pkgsrc repository, but we find enough merit in some of them to include them in our package repositories.

Repository Flows

The following diagram shows how changes flow through the different repositories.

┌──────────────────────────────────────┬───────────────────────────────────────┐
│                                      │                                       │
│         The NetBSD Foundation        │                 Joyent                │
│         =====================        │                 ======                │
│                                      │                                       │
│       Commit directly to pkgsrc      │  Commit directly to pkgsrc-joyent and │
│                                      │    local branches of pkgsrc and wip   │
│                                      │                                       │
│  Access to pkgsrc is limited to TNF  │  Access limited to Joyent employees   │
│   members (via sponsorship process)  │                                       │
│                                      │                                       │
│  ┌────────────────────────────────┐  │                                       │
│  │ cvs.netbsd.org:/cvsroot/pkgsrc │  │                                       │
│  └───────────────┬────────────────┘  │                                       │
│                 [1]                  │                                       │
│     ┌────────────v─────────────┐     │     ┌──────────────────────────┐      │
│     │ github.com/NetBSD/pkgsrc ├────[2]───>│ github.com/joyent/pkgsrc │      │
│     └──────────────────────────┘     │     └──────────────────────────┘      │
│                                      │                                       │
├──────────────────────────────────────┤                                       │
│                                      │                                       │
│       The pkgsrc-wip project         │                                       │
│       ======================         │  ┌─────────────────────────────────┐  │
│                                      │  │ github.com/joyent/pkgsrc-joyent │  │
│    Commit directly to pkgsrc-wip     │  └─────────────────────────────────┘  │
│                                      │                                       │
│    Access open to anyone who asks    │                                       │
│                                      │                                       │
│    ┌────────────────────────────┐    │    ┌──────────────────────────────┐   │
│    │ wip.pkgsrc.org:/pkgsrc-wip ├───[3]──>│ github.com/joyent/pkgsrc-wip │   │
│    └────────────────────────────┘    │    └──────────────────────────────┘   │
│                                      │                                       │
└──────────────────────────────────────┴───────────────────────────────────────┘
  1. Regular fossil-based full conversion from CVS to Git exported to GitHub
  2. Forked repository regularly updated from upstream
  3. Loose fork, updates from wip are merged into TritonDataCenter/pkgsrc-wip manually

Branches

We have developed a number of patch sets that are not yet suitable for pushing upstream to pkgsrc. In order to keep track of them they are separated into feature branches and then merged together to form our release branch.

As of our 2020Q4 release it looks something like this:

┌──────────────────────────────────────┬───────────────────────────────────────┐
│                                      │                                       │
│         The NetBSD Foundation        │                 Joyent                │
│         =====================        │                 ======                │
│                                      │                                       │
│     ┌──────────────────────────┐     │     ┌──────────────────────────┐      │
│     │ github.com/NetBSD/pkgsrc │     │     │ github.com/joyent/pkgsrc │      │
│     └────────────┬─────────────┘     │     └──────────────────────────┘      │
│                  v                   │                                       │
│     ┌──────────────────────────┐     │     ┌──────────────────────────┐      │
│     │       pkgsrc-2020Q4      ├──────────>│      pkgsrc-2020Q4       │      │
│     └──────────────────────────┘     │     └────────────┬─────────────┘      │
│                                      │                  v                    │
│                                      │ ┌──────────────────────────────────┐  │
│                                      │ │ joyent/feature/backports/2020Q4  │  │
│                                      │ │ joyent/feature/ctf/2020Q4        │  │
│                                      │ │ joyent/feature/miscfix/2020Q4    │  │
│                                      │ │ joyent/feature/pbulk/2020Q4      │  │
│                                      │ │ joyent/feature/pbulkmulti/2020Q4 │  │
│                                      │ └────────────────┬─────────────────┘  │
│                                      │                  v                    │
│                                      │     ┌──────────────────────────┐      │
│                                      │     │   joyent/release/2020Q4  │      │
│                                      │     └──────────────────────────┘      │
└──────────────────────────────────────┴───────────────────────────────────────┘

The branches are as follows:

pkgsrc-YYYYQQ

The pristine branch from upstream. This is managed by the releng-pkgsrc@pkgsrc.org team who backport changes from pkgsrc trunk as requested by developers. This process is documented at https://www.netbsd.org/developers/releng/pullups.html

We pull this branch as-is into our own pkgsrc-YYYYQQ branch, and that is then used as the basis for our feature branches documented below.

backports

A branch where we perform our own pullups of changes from pkgsrc trunk that might not be suitable for general pkgsrc use, or are done after releng-pkgsrc have stopped maintaining their branch. These branches are usually only created for LTS branches.

ctf

Our CTF work that enables CTF debugging support in supported packages.

miscfix

A general dumping ground for changes that aren't big enough for their own branch.

pbulk

A set of changes to enhance pbulk, the software that we use to bulk build our package sets.

pbulkmulti

Changes to enable more multi-package builds, for example building p5-DBD-MySQL against every version of MySQL, MariaDB, and Percona that are supported in pkgsrc instead of just the default.

release

This is a combined merge of all of the feature branches, plus the addition of our pkgsrc-joyent and pkgsrc-wip submodules. This is the branch that is used to build the release packages.

Creating A New Release Branch

First, ensure your pkgsrc git clone has the following remotes configured:

$ cd pkgsrc; git remote -v
netbsd	git@github.com:NetBSD/pkgsrc.git (fetch)
netbsd	git@github.com:NetBSD/pkgsrc.git (push)
origin	git@github.com:TritonDataCenter/pkgsrc.git (fetch)
origin	git@github.com:TritonDataCenter/pkgsrc.git (push)

And that the joyent and wip sub-directories are available with the following remotes configured:

$ cd pkgsrc/joyent; git remote -v
origin	https://github.com/TritonDataCenter/pkgsrc-joyent.git (fetch)
origin	git@github.com:TritonDataCenter/pkgsrc-joyent.git (push)
$ cd pkgsrc/wip; git remote -v
netbsd	git@github.com:NetBSD/pkgsrc-wip.git (fetch)
netbsd	git@github.com:NetBSD/pkgsrc-wip.git (push)
origin	https://github.com/TritonDataCenter/pkgsrc-wip.git (fetch)
origin	git@github.com:TritonDataCenter/pkgsrc-wip.git (push)

Create our copy of the upstream tracking branch:

$ BRANCH=2020Q4
$ cd pkgsrc
$ git fetch netbsd
$ git checkout -b pkgsrc-${BRANCH} netbsd/pkgsrc-${BRANCH}
$ git push -u origin pkgsrc-${BRANCH}

Calculate the common ancestor commit between the branch and trunk, this will be our branch point:

$ sha1=$(git rev-list --boundary netbsd/pkgsrc-${BRANCH}..netbsd/trunk | awk '/^-/ {sub(/^-/, ""); print}')
$ git show ${sha1}

Update trunk to the branch point. This ensures that when we create the quarterly branches from their trunk counterparts that we do not have any conflicts to manage:

$ git checkout trunk
$ git pull -r netbsd $sha1
$ git push -v

For each of our current trunk feature trees, update them to the branch point, and rebase to tidy up any fixup commits and ensure our changes are the most recent. As we're rebasing, a force commit is required, so you may want to be careful to verify what you're about to push.

$ git branch -r | grep origin.feature.*trunk

$ for branch in each trunk feature branch that is still active; do
    git checkout feature/${branch}/trunk
    git pull -r netbsd $sha1
    git push -fv
  done

Create the quarterly branches, based on the trunk branches.

$ for branch in each feature branch; do
    git checkout -b feature/${branch}/${BRANCH} netbsd/pkgsrc-${BRANCH}
    git cherry-pick $(git rev-list --reverse $sha1..feature/${branch}/trunk)
    git push -u origin feature/${branch}/${BRANCH}
done

If this is a quarterly release, create the backports branch:

$ git checkout -b feature/backports/${BRANCH} netbsd/pkgsrc-${BRANCH}
$ git push -u origin feature/backports/${BRANCH}

Now we're ready to create the release branch:

$ git checkout -b release/${BRANCH} netbsd/pkgsrc-${BRANCH}

Merge in the appropriate feature trees. It's worth doing miscfix first as it's the largest. There may be release-specific branches, e.g. openssl111.

$ git merge feature/miscfix/${BRANCH}
$ git merge feature/pbulk/${BRANCH}
$ git merge feature/pbulkmulti/${BRANCH}
$ git merge feature/vars/${BRANCH}

Update submodules:

$ (cd extra; git pull)
$ (cd joyent; git pull)

Update wip submodule and create branch:

$ (
    cd wip
    git fetch --all
    git checkout master
    git pull -r netbsd master
    git push -fv
    git checkout -b ${BRANCH} master
    git push -u origin ${BRANCH}
)

Add the submodules:

$ git submodule add https://github.com/TritonDataCenter/pkgsrc-extra.git extra
$ git submodule add https://github.com/TritonDataCenter/pkgsrc-joyent.git joyent
$ git submodule add https://github.com/TritonDataCenter/pkgsrc-wip.git wip
$ git config -f .gitmodules submodule.wip.branch ${BRANCH}
$ git add .gitmodules
$ git commit -m "Add submodules."

When you're ready to push (it's usually best to configure Jenkins etc first), then do so:

$ git push -u origin release/${BRANCH}

Handling Forced Updates

Due to imperfections in the CVS to Git conversion, occasionally the Git repository is forcibly updated due to changes in the history which causes the SHA checksums to no longer match. This is usually due to $NetBSD$ tag expansion or a manual edit to a commit message.

We try hard to avoid these but sometimes things still go wrong, and when they do the trunk branch needs to be recreated. We don't perform this procedure for quarterly branches, instead it's simpler just to cherry-pick any updates that are required.

The procedure is mostly identical to above, but when selecting the commits to be applied the commands a slightly different.

for branch in macos miscfix pbulk pbulkmulti performance; do

	# Extract current patches
	git format-patch -p --ignore-submodules --no-merges \
	    -o ~/commits-${branch} --perl-regexp \
        --author='^((?!.*(@pkgsrc.org|srcmastr)).*)$' \
	    feature/${branch}/trunk netbsd/trunk

	# Reset current branch to new trunk
	git checkout feature/${branch}/trunk
	git reset --hard netbsd/trunk

	# Apply patch set
	git am --3way ~/commits-${branch}/*

	# Force push new branch
	git push -fu origin feature/${branch}/trunk

	# Clean up commits directory.
	rm -rf ~/commits-${branch}

done

Then recreate the release branch and force push.

FAQ

Find which repository a package came from

You can use pkg_info to show where a package came from, and thus which repository to log any issues or pull requests against. For example:

$ pkg_info -Q PKGPATH runit rust zoneinit
wip/runit
lang/rust
joyent/zoneinit

If it starts with joyent/ then pkgsrc-joyent, if wip/ then pkgsrc-wip, otherwise pkgsrc

pkgsrc is a registered trademark of The NetBSD Foundation, Inc.

pkgsrc.smartos.org | @pkgsrc | @jperkin | MNX Cloud

pkgsrc wiki

pkgsrc logo

Home

About

Usage Guides

Package Development

External Links

Clone this wiki locally