Maintainer Guide

Help Track Bugfixes to Backport

It helps that maintainers generally participate in some basic amount of oversight to spot important bugs for which users will appreciate patch-releases:

• Potential security issues have their own private-tracking system rather than discuss/tracking them in the public GitHub. See the Security Process for other/related info.

• A summarized list of bug fixes to backport into patch releases is currently tracked here and extended when notable bugs are found: https://github.com/zeek/zeek/issues/724.

• See the Release Cadence for an overview of Zeek's release/branch structure and basic awareness that a bug may warrant fixing in multiple branches: e.g. master, release/4.0, and release/4.1 may all need to receive patches for a given bug.

• The decision-tree/flow-chart of what gets backported to patch releases can be nuanced and can also consider the risks associated with any given changeset, but historically bugs which are security problems and bugs which interfere with something a user was trying to do, have no workaround, and do not require extensive new features/code tend to get fixed in patch releases.

• CI platform updates do get backported, so our supported releases see the same CI coverage, unless covering a platform involves an unreasonably large changeset. (This approach is effective starting with the 4.2 release, and will therefore apply to 5.0. It does not apply to 4.0.x.)

Merging Pull Requests

One of the most common maintainer tasks is reviewing and merging GitHub Pull Requests (PRs). A standard/conventional process for doing that basically follows these steps:

• Assign the PR to yourself to signal to others you are working on it.

• Perform your code review and judge whether it's ready to be merged or needs further revisions. The next steps assume you're ready to start merging into your local branch.

• git checkout master (assuming most PRs are destined for the master branch)

• git pull (make sure your local branch is up-to-date, if it's not, you'll just have to do more work to merge in upstream changes before the final git push will work).

• git merge --log --no-commit --no-ff origin/topic/person/example (if branch is in an offical upstream Zeek GitHub repo under https://github.com/zeek/) or git pull --log --no-commit --no-ff https://github.com/person/zeek their-branch (if branch is in an external Zeek fork). Note, these are equivalent to the git merge-branch or git pull-branch aliases from the Useful Git Aliases section. Main thing to notice is the convention of not using fast-forwarding when merging: a dedicated merge commit always gets created.

• If there's any minor adjustments you want to make (e.g. whitespace cleanup), you can do that now and stage the changes via the typical git add -u. If you're going to make any significant changes, it's usually better to do another round of revisions in the PR/branch first so that the full CI process covers them.

  A common thing to do at this point is to add entries to NEWS for anything significant: interesting new features, changes in behavior, removed/deprecated functionality.

  Another common thing to be aware of is that some PRs end up touching Git submodules. In those cases, you'll want to merge the changes into submodules first. The merging process for most submodules is similar to the steps you're reading now. Also, don't forget to eventually push them in a postorder manner, children before parents. You'll also have to ensure submodule pointers in parent repos are correctly staged into their merge commit. For example you might see in the zeek repo that a git status shows modified: auxil/broker (new commits). If you find that auxil/broker is now actually at the correct commit, you just run git add -u auxil/broker to stage that into the changes.

  And similarly, a PR may have made changes to external test suites that you have to merge. Those are not handled with Git submodules directly, but it's still a similar idea: first merge the changes to the external test suite repo (and remember they'll need to be pushed before zeek is pushed). Then there are files named testing/external/commit-hash.* that you can update with the associated Git commit hash that will inform CI about commit of each external repo it needs to use. (The update-changes scripts in a later step also tries to help catch mismatches in these commit-hash files and may prompt/remind you to check whether it's correct).

• Run the test suite locally and verify it passes. Note this can be important especially for external contributions since CI will not run some private test suites using that (untrusted) code.

• git commit to finalize the merge commit. If there's any notable adjustments you did in a previous step, it can be helpful to note them in the commit message.

• Run the script which updates the VERSION and CHANGES files: zeek-aux/devel-tools/update-changes. You'll be given a chance to make adjustments to the new CHANGES entries: you can generally use some judgement to just get rid of entries that few people would ever care about, like commits that have no semantic changes and are things like "Fixed whitespace in Foo.cc". You can also choose to re-contextualize some entries that aren't easily understood just going by the description in original git commit message or to re-summarize and aggregate commits that are conceptually part of a higher-level feature that was just added (e.g. if there's a bunch of commits related to "Added a test case for Foo", it's usually assumed that if we added a "Foo", it comes with test cases, so don't bother listing all those lower-level details with the inceptual change).

  (If you're merging things into a release/x.y branch, you can also consider just skipping the update-changes and save doing that until just before your ready to do the release).

• git push to send everything upstream. If you've also merged anything into submodules or external test suites, make sure to push those in a postorder fashion: children before parents.

Running the testsuite

The testsuite resides in the testing directory of the distribution. It includes a self-contained collection of btests for Zeek, a code coverage toolchain, and a mechanism for adding external testsuites. The Zeek project has two such suites:

• https://github.com/zeek/zeek-testing: tests that involve public, large traces.
• https://github.com/zeek/zeek-testing-private: similar, but with content that we cannot make public. As a merge master you're able to clone this repo.

There's additional documentation for external testsuites here.

To add external suites, clone them into testing/external. If you're only using the public external testsuite, you can just say make init. Run ./scripts/find-git-repos in that folder to verify that the test setup correctly identifies them.

The external testsuites support release branches, matching those of Zeek itself. If you're running the testsuite for a Zeek release branch (e.g. release/4.0), switch the testsuites over to the corresponding branch.

To run the testsuite, cd into testing and say make.

Special Git Commit Message Keywords

Commit messages may contain special keywords/tags you can choose to use:

• Standard GitHub keywords to link/close issues

  Using GH-N as an alternative to just #N (where N is an issue/PR number) also works just the same (and may more clearly indicate the issue is on GitHub, as opposed to some other bug tracker, of which Zeek has now used several).

• All repos under https://github.com/zeek are under automatic watch of a github-notifier (from https://github.com/rsmmr/git-notifier) to send emails to the commits mailing lists, but if you want to prevent a given commit from generating an email, put [nomail] somewhere in the commit message.

• To skip CI for a commit, put [skip ci] somewhere in the commit message. You'd usually only want to do this to avoid wasting energy/time testing a commit that you're confident won't break anything (e.g. a small README or documentation change).

Useful Git Aliases

Git allows to define custom commands in ~/.gitconfig that then turn into more complex ones. You have to put them into an [alias] section. Here are some that may help:

• Submodule operations:

  • fix-submodules = submodule update --recursive --init

    Recursively checks out all Git submodules to commit indicated by their parent. Typically do this after git pull or git checkout operations.

  • recurse = "!sh -c 'for i in . `git submodule foreach -q --recursive pwd`; do (cd $i && $@); done' -"

    E.g. git recurse git status will recursively run git status (or any arbitrary shell command) in all submodule directories, traversing in a preorder manner.

  • revrecurse = "!sh -c 'for i in `git submodule foreach -q --recursive pwd | tac` .; do (cd $i && $@); done' -"

    This is the "reverse" of the above recurse alias, running commands in all submodules in postorder fashion, which is children-before-parents, and that's typically what you want if you're doing bulk Git operations like generating commits or pushing things.

    (See below for a version of this that may work better on macOS, like if you don't have a tac)

  • revrecursemac = "!sh -c 'for i in `git submodule foreach -q --recursive pwd | tail -r` .; do (cd $i && $@); done' -"

    Same as revrecurse above, but more compatible with macOS.

  • commit-modules = commit -am 'Update submodule(s) [nomail]'

    A typical way to generate a commit that does nothing other than update a submodule. The [nomail] tag is used because few people care when these events happen, or they can already infer they happened from associated commits that did generate an email.

• Aliases to merge stuff The Zeek Way. The --log option will include the first sentence of each topic-branch commit in the commit log. The --no-commit will give you a chance to review staged changes and make adjustments before doing the commit. The --no-ff will prevent all fast-forwards and always generate a merge-commit: useful to avoid losing historical information about the existence of a topic branch and having it record as parent pointers to the branch HEADs that were involved in the merge (and also maintainers tend to make small adjustments in the process of merging and the merge commit serves as a place to collect those).

  • merge-branch = merge --log --no-commit --no-ff

    Conventional way to merge branches located in offical upstream Zeek repos under https://github.com/zeek/. Example: git merge-branch origin/topic/person/example.

  • pull-branch = pull --log --no-commit --no-ff

    Conventional way to merge branches located in external Zeek forks. Example: git pull-branch https://github.com/person/zeek their-branch

    Note: if you use a branch.autosetuprebase setting in your ~/.gitconfig, you may also need to add --rebase=false.

• wdiff = diff --word-diff

  Use git wdiff or git wdiff --staged to do a word-based diff.

• undo = reset HEAD~

  Use git undo to backout the last git commit you did (mostly only for use when you realize a mistake in the last commit before having pushed it).

Other Git Tips

• When cherry-picking commits, using the -x flag ensures that the commit message includes a sentence that explains the commit the cherry-pick came from.