How do I fetch all commits only in the PR branch

[samholmes]

How do I tell actions/checkout to fetch all the commits up to the fork point? I want all the commits up to the fork point from the base branch. I don't see any examples on how to do this in the README. I'd assume this is a common need.

[wmertens]

workaround: assume that forks are never more than 300 commits and fetch 300 commits

[wmertens]

also duplicate of #520

[samholmes]

If you want your script to only consider the commits in the PR, then by over fetching you'll pollute this list with commits not in the PR.

[jbreckmckye]

This is a necessity for most monorepo builds

[kettanaito]

I'd like to run an automated release cron job that'd analyze commits for the past N hours and create a release. Turns out that actions/checkout will only have a single commit—the one that's triggered the corn job.

I know there's a fetch-depth option but it pushes me away with the "all branches and all tags" statement. I really need only all commits for the default branch to be present in Git history. I wonder how actions like semantic-release-action even work with corn releases then. It'd be silly to expect a new release for each commit.

[jbreckmckye]

Without being able to do it via the checkout action, here's a workaround in GHA

- name: Checkout
  uses: actions/checkout@v2
- name: Checkout PR changes
  run: |
    # Un-shallow
    git config remote.origin.fetch "+refs/heads/*:refs/remotes/origin/*"
    # Deepen topic branch; checkout topic branch
    git fetch origin ${{ github.event.pull_request.head.ref }} --depth=$(( ${{ github.event.pull_request.commits }} + 1 ))
    git checkout ${{ github.event.pull_request.head.ref }}
    # Fetch main for common origin
    git fetch origin main:main --depth=50
- name: Get BRANCH_POINT
  id: branch_point
  run: |
    # Find common ancestor between main and topic branch
    BRANCH_POINT=$(git merge-base $(git rev-parse --abbrev-ref HEAD) main)
    [[ -z BRANCH_POINT ]] && echo "No branch point" && exit 1
    echo "::set-output name=REF::$BRANCH_POINT"
- name: List changed files
  run: git diff --name-only ${{ steps.branch_point.outputs.REF }}

(Improvements welcome)

[polarathene]

I want all the commits up to the fork point from the base branch.

You can use the github context to get how many commits belong to the PR, then fetch that depth.

---

Example - Fetch enough commits from PR & base branch (eg: master)

To minimize commits fetched to compare two branches, a git fetch to the other branch (eg: master / main) will identify a common commit between the two branches and retrieve roughly only the commits needed to support that minimal history.

If the local commit history doesn't already have a commit from the 2nd branch being fetched, the full history (or whatever depth is requested) is fetched for that branch instead. This usually requires fetching one additional commit (the one you branched from) for the 1st branch (eg: PR branch).

Example for the PR branch to pull enough commit history to include a commit the other branch also has:

- name: 'PR commits + 1'
  run: echo "PR_FETCH_DEPTH=$(( ${{ github.event.pull_request.commits }} + 1 ))" >> "${GITHUB_ENV}"

- name: 'Checkout PR branch and all PR commits'
  uses: actions/checkout@v3
    with:
      ref: ${{ github.event.pull_request.head.ref }}
      fetch-depth: ${{ env.PR_FETCH_DEPTH }}

- name: 'Fetch the other branch with enough history for a common merge-base commit'
  run: |
    git fetch origin ${{ github.event.pull_request.base.ref }}

More Info

It's important to set the ref to the PR head ref as above, since the default ref of this action is one extra "merge-commit" (the PR into the base branch), which will not only offset your fetch-depth by 1 additional commit needed, but possibly cause other issues (eg: with git fetch on the base branch, and trying to get commit history that can successfully derive a merge-base).

NOTE: Merge-commits (those from the base branch into the PR branch, not the action default test merge-commit ref) contribute to the total commits from github context, and the fetch depth range would not only get N commits from your branch but also N commits associated to the history of merge commits.

You may not need the extra commit - If you have a merge commit from the base branch, then your local history will have a commit prior to the merge that it can probably use instead. Otherwise fetching an extra commit should ensure a common ancestor.

EDIT: Fetching to a common ancestor may fail when you have commits belonging to the base branch in local history which have not been merged into the PR branch. This problem occurs with the default ref (the generated test merge-commit of the PR branch into the base branch) with a fetch-depth of 2 or more.

---

An earlier attempt for fetch that didn't work out

Originally I was suggesting a way to provide a more specific commit for fetch to negotiate with, but I think it usually won't make much of a difference vs default git fetch? I also ran into scenarios where it broke, keeping for reference here as it may be helpful to others and not common to find online.

# Get the oldest commit in the branch which should have no parents,
# `--first-parent` only follows first parent encountered (should be our PR branch commits, not commits associated to merge commits from base branch):
FIRST_COMMIT_IN_BRANCH=$(git rev-list --first-parent --max-parents=0 --max-count=1 HEAD)

# Use that commit hash to lookup the next parent (that we don't have in history):
PARENT_COMMIT_HASH=$(git cat-file -p "${FIRST_COMMIT_IN_BRANCH}" | sed -n 's/^parent //p')

# Use that parent commit as a hint for fetch auto-depth, otherwise default fetch increments by the 
# fibonacci sequence until a commit from the base branch is found that also exists in our local commit history:
git fetch --negotiation-tip "${PARENT_COMMIT_HASH}" origin ${{ github.event.pull_request.base.ref }}

# Note this hint seems to fail if you already have a newer commit in local history from the base branch, 
# Such as when the default merge-commit ref with fetch depth of 2 or more pulls in base branch commits.
# Thus not always reliable to establish a merge-base...
#
# It will also fail if the root commit (first commit in repo) was the oldest commit found for FIRST_COMMIT_IN_BRANCH,
# Due to the PARENT_COMMIT_HASH not being possible to resolve.
# That can happen with too many merge commits of base branch to PR branch, as it bumps up the github context
# for commit count, fetching more history than needed (those merge commits don't seem to count for history depth)

Otherwise I found with a merge-commit, the fetch depth for +1 commits of your PR commit count could result in history being more than the expected number of commits fetched (the depth is technically correct from the commit chain associated to the merge commit).

With that lengthy command, you shouldn't need to request one more commit. But I've since found it unreliable (explained in comments for snippet).

[polarathene]

(Improvements welcome)

@jbreckmckye if you're interested in a file-name diff (specifically files added/changed, ignoring others like renames or deletions), here's a few examples.

This is nice and small, and should be fine AFAIK, see commented version below for more details:

- name: 'Checkout PR branch (with test merge-commit)'
  uses: actions/checkout@v3
    with:
      fetch-depth: 2
- name: 'Get a list of changed files to process'
  run: git diff-tree --no-commit-id --name-only --diff-filter 'AM' -r HEAD^1 HEAD

---

Variants and details

I've collapsed the original content (still useful maybe if you want to understand why a --merge-base is important), as I have curated a better source of this information here.

Click to view more info

- name: 'Checkout PR branch'
  uses: actions/checkout@v3
    with:
      ref: ${{ github.event.pull_request.head.ref }}

- name: 'Get a list of changed files to process'
  run: |
    # Fetch enough history for a common merge-base commit
    git fetch origin ${{ github.event.pull_request.head.ref }} --depth $(( ${{ github.event.pull_request.commits }} + 1 ))
    git fetch origin ${{ github.event.pull_request.base.ref }}

    # Show only files from the PR with content filtered by specific git status (Added or Modified):
    git diff-tree --name-only --diff-filter 'AM' -r \
      --merge-base origin/${{ github.event.pull_request.base.ref }} ${{ github.event.pull_request.head.ref }}

The --merge-base option will find a common ancestor like git merge-base, and use that commit as the "before" reference to diff against, replacing the base branch ref value (thus: --merge-base <common ancestor commit> <PR head commit>)

If your action(s) involved use actions/checkout prior to this point, careful of the default "test merge-commit" ref being in the local history still. That can cause git fetch origin ${{ github.event.pull_request.base.ref }} to not fetch extra history needed for a merge-base commit, assuming that ref had a fetch-depth of 2 or more (which will fetch commits from both base and PR branches at that depth).

If you do need to avoid that failure scenario, you can lookup the date of the commit you branched from the base branch, and request all commits since then:

# This should get the oldest commit in the local fetched history (which may not be the branched base commit):
BRANCHED_FROM_COMMIT=$( git rev-list --first-parent --max-parents=0 --max-count=1 ${{ github.event.pull_request.head.ref }} )
UNIX_TIMESTAMP=$( git log --format=%ct "${BRANCHED_FROM_COMMIT}" )

# Get all commits since that commit for the base branch (eg: master):
git fetch --shallow-since "${UNIX_TIMESTAMP}" origin ${{ github.event.pull_request.base.ref }}

---

If you only need the diff from the PR, you may not need the potentially lengthy commit history from the above approach and can instead use either of these:

- name: 'Checkout PR branch (with test merge-commit)'
  uses: actions/checkout@v3
    with:
      # Merge commit + two commits (1 from each branch, the 2nd depth level)
      fetch-depth: 2

# Show only files from the PR with content filtered by specific git status (Added or Modified)
# HEAD^1 is the base branch commit, HEAD is the merge commit, no merge-base needed
- name: 'Get a list of changed files to process'
  run: git diff-tree --no-commit-id --name-only --diff-filter 'AM' -r HEAD^1 HEAD

Instead of using HEAD, you can use the actual default ref, and set a --merge-base, but since it's effectively on the same branch (base) to compare, there isn't much benefit from doing so, this is just more verbose for the sake of it (but useful if you need/prefer to use refs instead of HEAD or commit hashes):

- name: 'Checkout PR branch (with test merge-commit)'
  uses: actions/checkout@v3
    with:
      # Merge commit + two commits (1 from each branch, the 2nd depth level)
      fetch-depth: 2

- name: 'Get a list of changed files to process'
  env:
    # This is the default ref value, not exactly the same value as `github.ref` context:
    PR_REF: refs/remotes/pull/${{ github.event.pull_request.number }}/merge
  run: |
    # No extra commits need to be fetched, the base branch ref will be set to HEAD^1 commit
    git fetch origin ${{ github.event.pull_request.base.ref }}

    # Show only files from the PR with content filtered by specific git status (Added or Modified)
    git diff-tree --name-only --diff-filter 'AM' -r \
      --merge-base origin/${{ github.event.pull_request.base.ref }} ${PR_REF}

If you tried to use --merge-base against the other commit (from the PR) belonging to the merge commit, then it'd fail to derive until both branches fetch enough commits into local history, as explained earlier with --shallow-since approach.

WARNING: If you removed the --merge-base and changed ${PR_REF} to the PR commit, you'd get a diff between those two commits, which isn't that useful (eg: If the base branch has deleted a file since, but the PR hasn't, then that file is shown in part of the diff as "Added", which is wrong).

For those unfamiliar with why --merge-base is important, that's why, but a non-issue AFAIK when comparing with the "test merge commit" that Github provides and fetches by default.

[jbreckmckye]

@polarathene What I was actually after was that BRANCH_POINT ref. The reason being, I'm using Yarn workspace tools, which take the reference to figure out which things need to be re-built / re-tested. However, the diff itself is still very relevant for things like linting.

I'm finding this is really helping me optimise my PR flow. For example, I'm working on repo that used to have a twenty-five minute PR flow for lint, unit-tests, static analysis, etc. Now I have differential builds I can run a differential lint and a subset of unit tests, and a PR build can now take as little as two minutes.

Your examples are still helpful though - I didn't realise I could parameterise the checkout action with the fetch depth, and I'd never really looked too hard at diff-tree. Seems like exactly the command I needed.

[polarathene]

What I was actually after was that BRANCH_POINT ref.

# Avoid the default ref "test merge commit" generated by Github (`github.sha`),
# Otherwise `git fetch` will fail to retrieve enough commits for a merge-base,
# The head ref (with default `fetch-depth: 1`) references the latest commit from the PR branch:
- name: 'Checkout PR branch'
  uses: actions/checkout@v3
    with:
      ref: ${{ github.event.pull_request.head.ref }}

- name: 'Find the merge-base commit hash'
  env:
    branch_main: ${{ github.event.pull_request.base.ref }}
    branch_topic: ${{ github.event.pull_request.head.ref }}
  run: |
    # Fetch enough history to find a common ancestor commit (aka merge-base):
    git fetch origin ${{ env.branch_topic }} --depth $(( ${{ github.event.pull_request.commits }} + 1 ))
    git fetch origin ${{ env.branch_main }}

    BRANCH_POINT=$( git merge-base origin/${{ env.branch_main }} ${{ env.branch_topic }} )
    echo "::set-output name=REF::$BRANCH_POINT"

Same thing really. Just remember that won't identify the commit that the PR branched from.

If there was a merge commit from base branch into the PR branch, the last commit from the base branch becomes the merge-base target instead (which is probably what you'd want, anyway).

WARNING: While using the branch refs as shown above works most of the time, it's not as deterministic for when you re-run a previous workflow job, as those branches may have new commits since, and especially for the PR branch, with a stale github context in the re-run, you risk an inaccurate --depth to fetch. Prefer referencing the base.sha and head.sha instead, as demonstrated at the end of this response.

---

Example of when merge-base derived is not the original commit you branched the PR from:

Example graph with merge-base when merge-commits are in history

git init /tmp/example && cd /tmp/example
git remote add origin https://github.com/user-name/repo-name

# PR has 6 commits (1 is a merge-commit from `main` branch), so fetch that + 1:
git fetch origin pr-branch --depth=7
git log --oneline --graph --all

* 264e38d (origin/pr-branch) PR-F
*   3a2b30b Merge branch 'main' into pr-branch <PR-E>
|\  
| * 43653c4 MAIN-E
| * 1f7e88e MAIN-D
| * 50feed1 MAIN-C
* | e210a28 PR-D
* | 60f8089 PR-C
* | 6f71a7f PR-B
* | 194ad14 PR-A
|/  
* d9411d4 MAIN-B <branched from here> <fetch-depth from 264e38d down PR commit chain>
* 6a18b1c (grafted) MAIN-A <fetch-depth from 264e38d down MAIN commit chain>

It's a bit harder to read the graph output without the colours CLI uses, but you can see the two lines on the left, the * is a commit in that line/branch, currently it only knows of the pr-branch and that some other commits were merged into the pr-branch.

The pr-branch chain looks like a depth of 8, but it's due to the merge commit chain also ensuring a depth of 7 commits (*).

Next fetch the main branch (base branch):

git fetch origin main
git log --oneline --graph --all

* 60b5fd6 (origin/main) MAIN-H <test merge-commit target>
* d1eeab7 MAIN-G
* b0d5074 MAIN-F
| * 264e38d (origin/pr-branch) PR-F
| *   3a2b30b Merge branch 'main' into pr-branch <PR-E>
| |\  
| |/  
|/|   
* | 43653c4 MAIN-E <merge-base result> <current commit for ${{ github.event.pull_request.base.sha }}>
* | 1f7e88e MAIN-D
* | 50feed1 MAIN-C
| * e210a28 PR-D
| * 60f8089 PR-C
| * 6f71a7f PR-B
| * 194ad14 PR-A
|/  
* d9411d4 MAIN-B <branched from here>
* 6a18b1c (grafted) MAIN-A

The graph output now shows commit 43653c4 (MAIN-E) connects to 3a2b30b (PR-E) commit, but also continues to the newer b0d5074 (MAIN-F) commit as the main branch has had new commits since the last merge-commit to the PR happened. MAIN-E is used as the merge-base, not MAIN-B as that is where main diverges from pr-branch now.

Additional tips:

• ${{ github.event.pull_request.base.sha }} could point to the commit hash of MAIN-D, if at the time the PR was opened on the repo when that was the latest commit on the main branch.
• ${{ github.event.pull_request.base.sha }} value then updates to the commit hash of MAIN-E once the merge commit is made at a later point in time.
• With the default ref for the action however, the generated test merge commit is ${{ github.sha }}, but the parent commit on main (which pr-branch was merged with to create the test commit) was the latest commit on main when commit history of pr-branch last updated. Do not mistake ${{ github.event.pull_request.base.sha }} for that same commit (which is updated to the latest main commit when main is merged into pr-branch, or when rebasing pr-branch onto main).
• When there are multiple merge commits from the base branch into the PR branch, they do count towards the fetch-depth progress when encountered and are part of the commits context count that Github provides. The other commit parent of each merge commit however will branch off and continue to retrieve N commits for the remaining depth as well (which can sometimes result in retrieving as far back as the root commit).

[marc-hb]

I want all the commits up to the fork point from the base branch.

git merge-base is the "obvious" solution and it works in most cases but it does NOT work in all cases. So it cannot be used as the solution in the most common action that absolutely everyone uses.

One alternative is to use https://api.github.com/repos/$gh_project/pulls/$pr_number/commits, see example in
https://github.com/thesofproject/sof/blob/73df64444b752/.github/workflows/codestyle.yml#L38

Two problems with git merge-base:

• In non trivial git histories there is more than one merge-base. For instance look at the "criss-cross" paragraph and diagram in the documentation https://git-scm.com/docs/git-merge-base. Depending on which one you pick the results will change.
• git merge-base is not compatible with shallow clones. Relying on it can make you believe the entire git history has been submitted in the Pull Request. See example and diagram in Shallow cloning can create pull requests with one million commits thesofproject/linux#2556 (learned the really hard way!)