-
Notifications
You must be signed in to change notification settings - Fork 79
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Update the instructions for upgrade of LLVM/Clang. #1146
Changes from 6 commits
70c99a8
8e865ea
d10bc48
d0acbbc
2394cef
f9dbb04
6fa40b7
ce1616d
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,38 @@ | ||
# Notes on LLVM/Clang Upgrades | ||
|
||
This file gives details about the commit hashes up to which the `baseline` and | ||
`master` branches of the `checkedc-clang` repository have been upgraded with | ||
respect to the upstream LLVM/Clang repository. | ||
|
||
|
||
# Upgrade to LLVM/Clang 11.x | ||
|
||
The commit hash on the `main` branch of the upstream LLVM/Clang repository | ||
up to which the `baseline` and the `master` branches of the `checkedc-clang` | ||
repository have been upgraded: | ||
<branch-point-of-11-on-main> : 2e10b7a39b930ef8d9c4362509d8835b221fbc0a | ||
|
||
Cherry-picked commits (on `master` of `checkedc-clang`) from the upstream | ||
LLVM/Clang repository: | ||
Details not available. | ||
|
||
The branch `release_11.x` of the `checkedc-clang` repository is upgraded to | ||
<llvm-11.1.0-commit-hash> : 1fdec59bffc11ae37eb51a1b9869f0696bfd5312 | ||
|
||
|
||
# Upgrade to LLVM/Clang 12.x | ||
|
||
The commit hash on the `main` branch of the upstream LLVM/Clang repository | ||
up to which the `baseline` and the `master` branches of the `checkedc-clang` | ||
repository have been upgraded: | ||
<branch-point-of-12-on-main> : 8e464dd76befbc4a39a1d21968a3d5f543e29312 | ||
|
||
Cherry-picked commits (on `master` of `checkedc-clang`) from the upstream | ||
LLVM/Clang repository: | ||
|
||
- Commit hash 0024efc69ea6cd0b630cd11cef5991b7edb73ffc on the `main` branch | ||
of the upstream LLVM/Clang repo (See LLVm/Clang bug | ||
[#48920](https://bugs.llvm.org/show_bug.cgi?id=48920) for details.) | ||
|
||
The branch `release_12.x` of the `checkedc-clang` repository is upgraded to | ||
<llvm-12.0.1-commit-hash> : fed41342a82f5a3a9201819a82bf7a48313e296b |
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -1,81 +1,251 @@ | ||
# Instructions for updating to the latest LLVM/Clang sources | ||
|
||
We are staying in sync with the LLVM/Clang mainline sources. The baseline branch is a pristine copy of | ||
LLVM/Clang sources. We periodically update the baseline branch and then push the changes to other branches. | ||
|
||
The first step is to create updated baseline branches: | ||
1. Create new branches of your local baseline branches (we suggest creating new | ||
branches so that you can run automated testing. You'll need new branches | ||
that you can push to GitHub). | ||
2. Update those branches to the latest sources. | ||
3. Run testing on those branches to make sure things are stable. | ||
|
||
The second step is to create updated master branches: | ||
1. Create branches of your updated baseline branches. | ||
2. Merge changes from the Checked C master branches into those branches. | ||
3. Fix merge conflicts and run testing. You will likely need to fix some issues | ||
and re-run testing until all issues are fixed. | ||
|
||
The third step is to merge your changes back into your baseline and master branches. | ||
|
||
## Create updated branches of your baseline branches | ||
|
||
First create remote to the mirrored GitHub repo that contains the updated sources | ||
for LLVM/Clang. Go to your LLVM/Clang repo and do: | ||
|
||
git remote add mirror https://github.com/llvm-mirror/llvm-project | ||
|
||
Then branch your baseline branch and merge changes into it: | ||
|
||
We are staying in sync with the upstream LLVM/Clang repository. The `baseline` | ||
branch of the `checkedc-clang` repository is a pristine copy of LLVM/Clang | ||
sources, and the `master` branch contains LLVM/Clang sources plus the support | ||
for Checked C language extensions. We periodically upgrade the `baseline` branch | ||
and the `master` branch to sources of the most recent releases from the upstream | ||
LLVM/Clang repository. | ||
|
||
The branching and merging policies of the `checkedc-clang` repository are | ||
aligned with the versioning and branching of upstream LLVM/Clang repository in | ||
order to closely follow the LLVM/Clang releases. LLVM/Clang releases are | ||
versioned as <major-version>.<minor-version>.<patch-version>. | ||
The LLVM/Clang repository has a branch called `main`, and several release | ||
branches (created off `main`) called `release/<major-version>.x`, each | ||
corresponding to the major version number of a release. New features go on the | ||
`main` branch. At an appropriate commit on the `main` branch, a release branch | ||
is created. Bug fixes go on the release branch and only some of these bug fixes | ||
get merged back into the `main` branch. Typically, LLVM/Clang makes two final | ||
releases for each major version. | ||
|
||
Now we describe our approach for upgrading the `checkedc-clang` repository to | ||
LLVM/Clang sources corresponding to the two releases for each major version, | ||
for example, 12.0.0 and 12.0.1. The upgrade is performed in two phases as | ||
described below. This description is followed by detailed instructions for each | ||
phase. | ||
|
||
**Phase 1**: We upgrade the `master` branch of the `checkedc-clang` repository | ||
up to the commit hash on the `main` branch of the LLVM/Clang repository at which | ||
the branch `release/12.x` is created - we shall refer to this commit hash as | ||
<branch-point-of-12-on-main>. We get this commit hash by executing the | ||
command `git merge-base main release/12.x` in a clone of the LLVM/Clang | ||
repository. Note that <branch-point-of-12-on-main> | ||
needs to be the full commit hash and not just the 7-digit abbreviation. This | ||
phase constitutes the major portion of the developer effort towards the merge. | ||
|
||
**Phase 2**: We create a branch called `release_12.x` off the `master` branch in | ||
the `checkedc-clang` repository. Then there are two sub-phases: | ||
- When we wish to make a Checked C compiler release based on the 12.0.0 | ||
release of LLVM/Clang: | ||
- We first upgrade the `release_12.x` branch (of the `checkedc-clang` | ||
repository) to the sources corresponding to the 12.0.0 release on the | ||
`release/12.x` branch of the LLVM/Clang repository. | ||
- Next, we merge Checked C features and bug fixes from the `master` branch | ||
to the `release_12.x` branch. | ||
- The release of the Checked C compiler happens from the `release_12.x` | ||
branch. | ||
- When we wish to make a Checked C compiler release based on the 12.0.1 | ||
release of LLVM/Clang: | ||
- We upgrade the same `release_12.x` branch (of the `checkedc-clang` | ||
repository) to the sources corresponding to the 12.0.1 release on the | ||
`release/12.x` branch of the LLVM/Clang repository. | ||
- Next, we merge Checked C features and bug fixes from the `master` branch | ||
to the `release_12.x` branch. | ||
- The release of the Checked C compiler happens from the `release_12.x` | ||
branch. | ||
|
||
|
||
We do not expect substantial developer effort to be expended in Phase 2 as it is | ||
highly unlikely that the bug fixes on the release branches of the LLVM/Clang | ||
repository will break Checked C build and tests. | ||
|
||
Note that Phase 1 and the sub-phases of Phase 2 may not be executed one after | ||
another in one continuous stretch. They may be spread over several weeks and may | ||
be interspersed with Checked C development on the `master` branch. | ||
|
||
## Detailed Instructions - Phase 1 | ||
|
||
1. Create a new branch off the `baseline` branch in the `checkedc-clang` | ||
repository. Let us call this new branch `updated_baseline`. | ||
``` | ||
git checkout -b updated_baseline baseline | ||
``` | ||
2. Upgrade the branch `updated_baseline` to <branch-point-of-12-on-main> | ||
commit hash on the branch `main` of the LLVM/Clang repository. Record the commit | ||
hash <branch-point-of-12-on-main> in the document LLVM-Upgrade-Notes.md. | ||
``` | ||
git remote add upstream https://github.com/llvm/llvm-project.git | ||
git pull upstream <branch-point-of-12-on-main> | ||
``` | ||
3. Execute LLVM/Clang tests on the branch `updated_baseline` on your local | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Nit: extra space between "Execute" and "LLVM/Clang" |
||
machine to make sure that the upgrade is stable. | ||
|
||
4. Merge the `updated_baseline` branch back to the `baseline` branch. We expect | ||
branch `updated_baseline` to be a descendant of branch `baseline`. | ||
``` | ||
git checkout baseline | ||
git checkout -b updated-baseline | ||
git pull mirror master | ||
|
||
## Run testing on your branched baseline branches. | ||
|
||
You can run testing locally or push your branched baseline branches to GitHub | ||
and use automated testing. This will show you whether there are any unexpected | ||
failures in your baseline branch. | ||
|
||
If you use automated testing, make sure to clean the build directory first. | ||
Enough changes will likely have accumulated that things may go wrong without doing | ||
that first. | ||
|
||
## Branch your new baseline branches and merge master changes | ||
|
||
You can now branch your baseline branches to create a new master branch: | ||
|
||
git checkout -b updated-master | ||
git merge --ff-only updated_baseline | ||
``` | ||
5. Create a new branch, say, `updated_baseline_master_12` from the `baseline` | ||
branch. | ||
``` | ||
git checkout -b updated_baseline_master_12 baseline | ||
``` | ||
6. Merge the `master` branch of the checkedc-clang repository into the | ||
`updated_baseline_master_12` branch. This merge may cause several merge | ||
conflicts and test case failures. After the merge command, git may suggest to | ||
you to set merge.renamelimit to a specific value. It is a good idea to set it as | ||
suggested and re-run the merge. The last section in this document gives | ||
guidelines on fixing the merge conflicts and test failures. | ||
``` | ||
git checkout updated_baseline_master_12 | ||
git merge master > merge_conflict_list.txt | ||
``` | ||
7. The resolution of test failures may require you to cherry-pick commits from | ||
the upstream LLVM/Clang repository. Record the cherry-picked commits in the | ||
document LLVM-Upgrade-Notes.md. Let <commit-to-cherry-pick> be one | ||
such commit hash on branch <cherry-pick-branch> of the LLVM/Clang | ||
repository. | ||
``` | ||
git fetch upstream <cherry-pick-branch> | ||
git cherry-pick -x <commit-to-cherry-pick> | ||
``` | ||
8. While the merge conflicts and test failures are being fixed, it is possible | ||
that the `master` branch has had several commits. Merge the `master` branch into | ||
`updated_baseline_master_12` branch, and resolve the new conflicts and failures | ||
if necessary. | ||
``` | ||
git merge master >> merge_conflict_list.txt | ||
``` | ||
9. After all the merge conflicts and test failures on the local machine are | ||
fixed, push the `baseline` and `updated_baseline_master_12` branches to the | ||
remote repository and execute the ADO tests on branch | ||
`updated_baseline_master_12`. Commits to the `master` branch should be held off | ||
during the final execution of these tests, i.e. just prior to executing step 10 | ||
below. | ||
``` | ||
git push origin baseline | ||
git push origin updated_baseline_master_12 | ||
``` | ||
10. After the ADO tests are successful, merge the `updated_baseline_master_12` | ||
branch into `master` and push to the remote repository. If all the changes on | ||
the `master` branch have been merged into the `updated_baseline_master_12` | ||
branch in step 8, then it should be possible to do a fast-forward merge of | ||
branch `updated_baseline_master_12` into the `master` branch. | ||
``` | ||
git checkout master | ||
git merge --ff-only updated_baseline_master_12 | ||
git push origin master | ||
``` | ||
|
||
## Detailed Instructions - Phase 2 | ||
We give the instructions for the first sub-phase of Phase 2. Similar steps | ||
have to be followed for the other sub-phase. | ||
|
||
1. Create a new branch, called `release_12.x`, from the `master` branch in the | ||
`checkedc-clang` repository. If the branch already exists, then merge the | ||
`master` branch into it. Resolve merge conflicts and test failures if any. | ||
``` | ||
git checkout -b release_12.x master | ||
OR | ||
git checkout release_12.x | ||
git merge master | ||
|
||
You will very likely have merge conflicts and some test failures. The test | ||
failures usually stem from incorrect merges or Checked C-specific data not being | ||
initialized by new or changed constructor methods. | ||
|
||
You may also need to pick up changes from LLVM/Clang for fixes to any unexpected | ||
baseline failures. | ||
|
||
You can push your updated master branches up to GitHub for automated | ||
testing. If you haven't cleaned the build directory as described earlier, | ||
make sure you do that. | ||
|
||
You'll want to run automated tests on Linux and Windows x86/x64, as well as | ||
``` | ||
2. Upgrade the branch `release_12.x` to the LLVM/Clang sources corresponding to | ||
the 12.0.0 release. The commit hash associated with the 12.0.0 release is given | ||
by the LLVM/Clang tag `llvmorg-12.0.0`. Resolve merge conflicts and test | ||
failures if any. Record the fact that the `release_12.x` branch has been | ||
upgraded upto the commit hash given by the `llvmorg-12.0.0` tag, in the document | ||
LLVM-Upgrade-Notes.md. | ||
``` | ||
git remote add upstream https://github.com/llvm/llvm-project.git | ||
git pull upstream llvmorg-12.0.0 | ||
``` | ||
3. Push the branch `release_12.x` to the remote repository, run ADO tests | ||
on it and fix test failures to make sure that the branch is stable. | ||
``` | ||
git push origin release_12.x | ||
``` | ||
## Practical Tips | ||
|
||
1. If you use automated testing, make sure to clean the build directory first. | ||
Enough changes will likely have accumulated that things may go wrong without | ||
doing that first. | ||
|
||
2. The test failures usually stem from incorrect merges or Checked C-specific data | ||
not being initialized by new or changed constructor methods. | ||
|
||
3. You'll want to run automated tests on Linux and Windows x86/x64, as well as | ||
Linux LNT tests. You may find in some cases that tests need to be updated | ||
based on new warnings or slightly different compiler behavior. | ||
|
||
## Merge your branched baseline and master branches | ||
|
||
Once all testing is passing, you can merge your branches back into | ||
your baseline and master branches. | ||
|
||
|
||
git checkout baseline | ||
git merge updated-baseline | ||
git checkout master | ||
git merge updated-master | ||
|
||
## Push the updated branches to GitHub | ||
|
||
The changes will be extensive enough that you don't want to do a pull request | ||
4. The changes will be extensive enough that you don't want to do a pull request | ||
on GitHub. Just push the branches up to GitHub. | ||
|
||
|
||
## Handling Merge Conflicts and Test Failures | ||
|
||
1. When merge.renamelimit is set to a large value, the output of the `git merge` | ||
command may have several lines like: | ||
- CONFLICT (rename/delete): ... in master renamed to ... in HEAD. Version | ||
HEAD ... left in tree. | ||
- CONFLICT (rename/delete): ... in master renamed to ... in HEAD. Version | ||
master ... left in tree. | ||
|
||
If the version in HEAD is left in tree, it is very likely a valid change - | ||
the file may have been moved. If the version in `master` is left in tree, | ||
the files may be valid files that have been added as part of Checked C | ||
support: | ||
- `llvm/projects/checkedc-wrapper/lit.site.cfg.py` | ||
- `llvm/projects/checkedc-wrapper/lit.cfg.py` | ||
|
||
Others may need more investigation. For example, in the LLVM/Clang 11 to 12 | ||
merge, two files which were reported by git as "Version master ... left in | ||
tree" were old versions lying around that needed to be deleted: | ||
- `clang/include/clang/StaticAnalyzer/Core/PathSensitive/SMTAPI.h` | ||
- `clang/include/clang/AST/TypeNodes.def` | ||
|
||
2. While resolving merge conflicts, it is a good idea to have three reference | ||
code snapshots to perform diffs of individual files to help distinguish between | ||
checkedc-specific changes and llvm-11-to-12 changes: | ||
- The `master` branch of the `checkedc-clang` repository at the commit hash | ||
which was merged into `updated_baseline_master_12`. | ||
- The pristine LLVM/Clang source at <branch-point-of-11-on-main> | ||
- The pristine LLVM/Clang source at <branch-point-of-12-on-main> | ||
|
||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I wholeheartedly agree with the value of being able to see all three versions of a file involved in a merge. However, the description of the three versions here is accurate only for the initial merge from LLVM 11 to 12 in phase 1 step 6. For a merge of additional Checked C feature development in phase 1 step 8, the relevant versions are We are already assuming that
|
||
The description of the three snapshots above holds only for the initial | ||
merge from LLVM 11 to 12 in phase 1 step 6. For a merge of additional | ||
Checked C feature development in phase 1 step 8, here are the relevant | ||
snapshots and their commit hashes, which we get from the in-progress merge: | ||
- A snapshot of branch `updated_baseline_master_12` after phase 1 step 7, or | ||
an earlier iteration of step 8. The commit hash that corresponds to this | ||
snapshot is `HEAD` - use command `git rev-parse HEAD` to dump the commit | ||
hash. | ||
- A snapshot of the `master` branch that participated in the previous merge | ||
that produced the above snapshot. The commit hash that corresponds to this | ||
snapshot is the common ancestor - use command | ||
`git merge-base HEAD MERGE_HEAD` to dump the commit hash. | ||
- A snapshot of the current `master` branch. The commit hash that | ||
corresponds to this is `MERGE_HEAD` - use command `git rev-parse MERGE_HEAD` | ||
to dump the commit hash. | ||
|
||
Here are some alternatives to the above approach of using source snapshots | ||
to manually perform diffs of the relevant files: | ||
- Run `git mergetool` - you may need to have a merge tool that is suitable | ||
for your environment (Linux/Windows) installed and configured | ||
(ex.: `kdiff3`). | ||
- Set `git config merge.conflictstyle diff3` before running `git merge` so | ||
that the conflict markup written to the file shows all three versions rather | ||
than just two. | ||
- Run | ||
`git show :1:clang/path/to/some/File.cpp > clang/path/to/some/File.1.cpp` | ||
(and similarly with 2 or 3 in place of 1) to write out the three versions | ||
for inspection with other tools. | ||
|
||
3. When bitfields are modified (or need modification to accommodate changes | ||
related to Checked C) in files like `clang/include/clang/AST/Stmt.h` (see | ||
commit hash af12895f631), and `clang/include/clang/AST/DeclBase.h` (see the diff | ||
between commits bc556e5 and e08e3f6 - this change was required for a previous | ||
merge), it may need more manual investigation to check the correctness of the | ||
changes. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Nit:
LLVm
=>LLVM