-
Notifications
You must be signed in to change notification settings - Fork 72
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
* Revert "[BoundsWidening] Determine checked scope specifier per statement (#1139)" (#1141) This reverts commit 980321d. * Determine checked scopes per statement (#1142) We introduce a 2-bit field called CheckedScopeSpecifier in the Stmt class. During parsing when a compound statement is created we iterate the elements (statements) of the compound statement and set the checked scope specifier for each element to the checked scope specifier of the compound statement. We can get the checked scope specifier for a statement by calling the getCheckedScopeSpecifier method on the statement. * Update the instructions for upgrade of LLVM/Clang. (#1146) * Updated the instructions for upgrade of LLVM/Clang. Also added a new file LLVM-Upgrade-Notes.md to track important information related to upgrades. * Fixed typos. * Addressed review comments. * Fixed an inadvertent deletion. * Addressed review comments. * Incorporated review comments. * Fixed minor typos. * Fixed typos. * Add new flags for available facts analysis * Add the analysis into the build script and the sema bounds * Add utility functions to check whether a var is used in a Expr and a BoundsExpr * Add AbstractFact as a basic available fact; Add InferredFact and adjust WhereClauseFact to be a subclass of AbstractFact * Add data structures used in the analysis * Add print and dump functions * Add utility functions which are also used by BoundsWideningAnalysis. * Add other utility functions. `IsSwitchCaseBlock`: use `dyn_cast_or_null` to cover the null pointer case. `ConditionOnEdge`: do not test if there is no edge between pred to curr since it will only be called if there is an edge. `GetModifiedVars`: use `TranspareCasts` to bypass some casting. The feature to deal with membership access and the array indexing is still TODO. * Add fact comparision and fact-realted set oerations (contains TODO). * Add testscases (one covers basic features and the other is converted from the previous available facts analysis) * Dataflow analysis: Add statement-based Gen/Kill. * Dataflow analysis: Add block-edge-based Gen set. * Dataflow analysis: Add function to compute In and Out set. * Dataflow analysis: Addworklist algorithm. * Add desctrutors to release the memory * Fix: modify the Gen/Kill rules to match the design doc; It also fixes a bug to visit dead blocks. * Cleanup comments * Fix: use the exisiting functions to find a `VarDecl` in an expr * Change the equal check on fact collections to equal size check * Update the testcases with the updated Gen/Kill * Remove debug flag for available facts. * Use lexco-compare for `EqualityOpFact` and `InferredFact`. * Add a map to store the comparision results of facts. * Change the source location of a fact to its near expr. * Use a dedicated list to collect created facts and clean them finally. * Verify if an expr contains errors before checking invertibility (#1154) The community has introduced a new annotation called "contains-errors" on AST nodes that contain semantic errors. As a result, after the upgrade of Checked C sources to LLVM 12 we need to check if an expr contains errors before operating on the expr. One such place is in InverseUtil::IsInvertible where we need to check if the input modifying expr contains errors. * Added containsErrors checks to InverUtil::Inverse * [BoundsWidening] Handle complex conditionals in bounds widening (#1149) Support bounds widening in presence of complex conditionals like: "if (*p != 0)", "if ((c = *p) == 'a')", etc. * Don't record temporary equality between expressions such as x and x + 1 in TargetSrcEquality (#1162) * Add AllowTempEquality parameter to RecordEqualityWithTarget * Use a ModifiedSameValue variable to determine the return value for UpdateSameValueAfterAssignment * Rename ModifiedSameValue to RemovedAnyExprs and clean up comments * Treat address-of array subscripts the same way as address-of dereferences (#1163) * In CheckAddressOfOperand, add case for address-of array subscripts to C99-specific logic * Move address-of array subscript check after other checks such as taking the address of an lvalue * Adjust expected AST output to account for different types of address-of array subscripts * Restore deleted comment about checking for array subscript expressions * Add comment explaining the placement of the address-of array subscript logic * Put &e1[e2] typing rules under a Checked C flag * Update the available facts analysis. Co-authored-by: Mandeep Singh Grang <magrang@microsoft.com> Co-authored-by: Sulekha Kulkarni <Sulekha.Kulkarni@microsoft.com> Co-authored-by: Katherine Kjeer <6687333+kkjeer@users.noreply.github.com>
- Loading branch information
1 parent
3d783f4
commit a167f43
Showing
29 changed files
with
4,379 additions
and
570 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,38 @@ | ||
# Notes on LLVM/Clang Upgrades | ||
|
||
This file gives details about the commit hashes up to which the `baseline` and | ||
`master` branches of the `checkedc-clang` repository have been upgraded with | ||
respect to the upstream LLVM/Clang repository. | ||
|
||
|
||
# Upgrade to LLVM/Clang 11.x | ||
|
||
The commit hash on the `main` branch of the upstream LLVM/Clang repository | ||
up to which the `baseline` and the `master` branches of the `checkedc-clang` | ||
repository have been upgraded: | ||
<branch-point-of-11-on-main> : 2e10b7a39b930ef8d9c4362509d8835b221fbc0a | ||
|
||
Cherry-picked commits (on `master` of `checkedc-clang`) from the upstream | ||
LLVM/Clang repository: | ||
Details not available. | ||
|
||
The branch `release_11.x` of the `checkedc-clang` repository is upgraded to | ||
<llvm-11.1.0-commit-hash> : 1fdec59bffc11ae37eb51a1b9869f0696bfd5312 | ||
|
||
|
||
# Upgrade to LLVM/Clang 12.x | ||
|
||
The commit hash on the `main` branch of the upstream LLVM/Clang repository | ||
up to which the `baseline` and the `master` branches of the `checkedc-clang` | ||
repository have been upgraded: | ||
<branch-point-of-12-on-main> : 8e464dd76befbc4a39a1d21968a3d5f543e29312 | ||
|
||
Cherry-picked commits (on `master` of `checkedc-clang`) from the upstream | ||
LLVM/Clang repository: | ||
|
||
- Commit hash 0024efc69ea6cd0b630cd11cef5991b7edb73ffc on the `main` branch | ||
of the upstream LLVM/Clang repo (See LLVM/Clang bug | ||
[#48920](https://bugs.llvm.org/show_bug.cgi?id=48920) for details.) | ||
|
||
The branch `release_12.x` of the `checkedc-clang` repository is upgraded to | ||
<llvm-12.0.1-commit-hash> : fed41342a82f5a3a9201819a82bf7a48313e296b |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -1,81 +1,251 @@ | ||
# Instructions for updating to the latest LLVM/Clang sources | ||
|
||
We are staying in sync with the LLVM/Clang mainline sources. The baseline branch is a pristine copy of | ||
LLVM/Clang sources. We periodically update the baseline branch and then push the changes to other branches. | ||
|
||
The first step is to create updated baseline branches: | ||
1. Create new branches of your local baseline branches (we suggest creating new | ||
branches so that you can run automated testing. You'll need new branches | ||
that you can push to GitHub). | ||
2. Update those branches to the latest sources. | ||
3. Run testing on those branches to make sure things are stable. | ||
|
||
The second step is to create updated master branches: | ||
1. Create branches of your updated baseline branches. | ||
2. Merge changes from the Checked C master branches into those branches. | ||
3. Fix merge conflicts and run testing. You will likely need to fix some issues | ||
and re-run testing until all issues are fixed. | ||
|
||
The third step is to merge your changes back into your baseline and master branches. | ||
|
||
## Create updated branches of your baseline branches | ||
|
||
First create remote to the mirrored GitHub repo that contains the updated sources | ||
for LLVM/Clang. Go to your LLVM/Clang repo and do: | ||
|
||
git remote add mirror https://github.com/llvm-mirror/llvm-project | ||
|
||
Then branch your baseline branch and merge changes into it: | ||
|
||
We are staying in sync with the upstream LLVM/Clang repository. The `baseline` | ||
branch of the `checkedc-clang` repository is a pristine copy of LLVM/Clang | ||
sources, and the `master` branch contains LLVM/Clang sources plus the support | ||
for Checked C language extensions. We periodically upgrade the `baseline` branch | ||
and the `master` branch to the sources of the most recent releases from the | ||
upstream LLVM/Clang repository. | ||
|
||
The branching and merging policies of the `checkedc-clang` repository are | ||
aligned with the versioning and branching of upstream LLVM/Clang repository in | ||
order to closely follow the LLVM/Clang releases. LLVM/Clang releases are | ||
versioned as <major-version>.<minor-version>.<patch-version>. | ||
The LLVM/Clang repository has a branch called `main`, and several release | ||
branches (created off `main`) called `release/<major-version>.x`, each | ||
corresponding to the major version number of a release. New features go on the | ||
`main` branch. At an appropriate commit on the `main` branch, a release branch | ||
is created. Bug fixes go on the release branch and only some of these bug fixes | ||
get merged back into the `main` branch. Typically, LLVM/Clang makes two final | ||
releases for each major version. | ||
|
||
Now we describe our approach for upgrading the `checkedc-clang` repository to | ||
LLVM/Clang sources corresponding to the two releases for each major version, | ||
for example, 12.0.0 and 12.0.1. The upgrade is performed in two phases as | ||
described below. This description is followed by detailed instructions for each | ||
phase. | ||
|
||
**Phase 1**: We upgrade the `master` branch of the `checkedc-clang` repository | ||
up to the commit hash on the `main` branch of the LLVM/Clang repository at which | ||
the branch `release/12.x` is created - we shall refer to this commit hash as | ||
<branch-point-of-12-on-main>. We get this commit hash by executing the | ||
command `git merge-base main release/12.x` in a clone of the LLVM/Clang | ||
repository. Note that <branch-point-of-12-on-main> | ||
needs to be the full commit hash and not just the 7-digit abbreviation. This | ||
phase constitutes the major portion of the developer effort towards the merge. | ||
|
||
**Phase 2**: We create a branch called `release_12.x` off the `master` branch in | ||
the `checkedc-clang` repository. Then there are two sub-phases: | ||
- When we wish to make a Checked C compiler release based on the 12.0.0 | ||
release of LLVM/Clang: | ||
- We first upgrade the `release_12.x` branch (of the `checkedc-clang` | ||
repository) to the sources corresponding to the 12.0.0 release on the | ||
`release/12.x` branch of the LLVM/Clang repository. | ||
- Next, we merge Checked C features and bug fixes from the `master` branch | ||
to the `release_12.x` branch. | ||
- The release of the Checked C compiler happens from the `release_12.x` | ||
branch. | ||
- When we wish to make a Checked C compiler release based on the 12.0.1 | ||
release of LLVM/Clang: | ||
- We upgrade the same `release_12.x` branch (of the `checkedc-clang` | ||
repository) to the sources corresponding to the 12.0.1 release on the | ||
`release/12.x` branch of the LLVM/Clang repository. | ||
- Next, we merge Checked C features and bug fixes from the `master` branch | ||
to the `release_12.x` branch. | ||
- The release of the Checked C compiler happens from the `release_12.x` | ||
branch. | ||
|
||
|
||
We do not expect substantial developer effort to be expended in Phase 2 as it is | ||
highly unlikely that the bug fixes on the release branches of the LLVM/Clang | ||
repository will break Checked C build and tests. | ||
|
||
Note that Phase 1 and the sub-phases of Phase 2 may not be executed one after | ||
another in one continuous stretch. They may be spread over several weeks and may | ||
be interspersed with Checked C development on the `master` branch. | ||
|
||
## Detailed Instructions - Phase 1 | ||
|
||
1. Create a new branch off the `baseline` branch in the `checkedc-clang` | ||
repository. Let us call this new branch `updated_baseline`. | ||
``` | ||
git checkout -b updated_baseline baseline | ||
``` | ||
2. Upgrade the branch `updated_baseline` to <branch-point-of-12-on-main> | ||
commit hash on the `main` branch of the LLVM/Clang repository. Record the commit | ||
hash <branch-point-of-12-on-main> in the document LLVM-Upgrade-Notes.md. | ||
``` | ||
git remote add upstream https://github.com/llvm/llvm-project.git | ||
git pull upstream <branch-point-of-12-on-main> | ||
``` | ||
3. Execute LLVM/Clang tests on the branch `updated_baseline` on your local | ||
machine to make sure that the upgrade is stable. | ||
|
||
4. Merge the `updated_baseline` branch back to the `baseline` branch. We expect | ||
branch `updated_baseline` to be a descendant of branch `baseline`. | ||
``` | ||
git checkout baseline | ||
git checkout -b updated-baseline | ||
git pull mirror master | ||
|
||
## Run testing on your branched baseline branches. | ||
|
||
You can run testing locally or push your branched baseline branches to GitHub | ||
and use automated testing. This will show you whether there are any unexpected | ||
failures in your baseline branch. | ||
|
||
If you use automated testing, make sure to clean the build directory first. | ||
Enough changes will likely have accumulated that things may go wrong without doing | ||
that first. | ||
|
||
## Branch your new baseline branches and merge master changes | ||
|
||
You can now branch your baseline branches to create a new master branch: | ||
|
||
git checkout -b updated-master | ||
git merge --ff-only updated_baseline | ||
``` | ||
5. Create a new branch, say, `updated_baseline_master_12` from the `baseline` | ||
branch. | ||
``` | ||
git checkout -b updated_baseline_master_12 baseline | ||
``` | ||
6. Merge the `master` branch of the checkedc-clang repository into the | ||
`updated_baseline_master_12` branch. This merge may cause several merge | ||
conflicts and test case failures. After the merge command, git may suggest to | ||
you to set `merge.renamelimit` to a specific value. It is a good idea to set it | ||
as suggested and re-run the merge. The last section in this document gives | ||
guidelines on fixing the merge conflicts and test failures. | ||
``` | ||
git checkout updated_baseline_master_12 | ||
git merge master > merge_conflict_list.txt | ||
``` | ||
7. The resolution of test failures may require you to cherry-pick commits from | ||
the upstream LLVM/Clang repository. Record the cherry-picked commits in the | ||
document LLVM-Upgrade-Notes.md. Let <commit-to-cherry-pick> be one | ||
such commit hash on branch <cherry-pick-branch> of the LLVM/Clang | ||
repository. | ||
``` | ||
git fetch upstream <cherry-pick-branch> | ||
git cherry-pick -x <commit-to-cherry-pick> | ||
``` | ||
8. While the merge conflicts and test failures are being fixed, it is possible | ||
that the `master` branch has had several commits. Merge the `master` branch into | ||
`updated_baseline_master_12` branch, and resolve the new conflicts and failures | ||
if necessary. | ||
``` | ||
git merge master >> merge_conflict_list.txt | ||
``` | ||
9. After all the merge conflicts and test failures on the local machine are | ||
fixed, push the `baseline` and `updated_baseline_master_12` branches to the | ||
remote repository and execute the ADO tests on branch | ||
`updated_baseline_master_12`. Commits to the `master` branch should be held off | ||
during the final execution of these tests, i.e. just prior to executing step 10 | ||
below. | ||
``` | ||
git push origin baseline | ||
git push origin updated_baseline_master_12 | ||
``` | ||
10. After the ADO tests are successful, merge the `updated_baseline_master_12` | ||
branch into `master` and push to the remote repository. If all the changes on | ||
the `master` branch have been merged into the `updated_baseline_master_12` | ||
branch in step 8, then it should be possible to do a fast-forward merge of | ||
branch `updated_baseline_master_12` into the `master` branch. | ||
``` | ||
git checkout master | ||
git merge --ff-only updated_baseline_master_12 | ||
git push origin master | ||
``` | ||
|
||
## Detailed Instructions - Phase 2 | ||
We give the instructions for the first sub-phase of Phase 2. Similar steps | ||
have to be followed for the other sub-phase. | ||
|
||
1. Create a new branch, called `release_12.x`, from the `master` branch in the | ||
`checkedc-clang` repository. If the branch already exists, then merge the | ||
`master` branch into it. Resolve merge conflicts and test failures if any. | ||
``` | ||
git checkout -b release_12.x master | ||
OR | ||
git checkout release_12.x | ||
git merge master | ||
|
||
You will very likely have merge conflicts and some test failures. The test | ||
failures usually stem from incorrect merges or Checked C-specific data not being | ||
initialized by new or changed constructor methods. | ||
|
||
You may also need to pick up changes from LLVM/Clang for fixes to any unexpected | ||
baseline failures. | ||
|
||
You can push your updated master branches up to GitHub for automated | ||
testing. If you haven't cleaned the build directory as described earlier, | ||
make sure you do that. | ||
|
||
You'll want to run automated tests on Linux and Windows x86/x64, as well as | ||
``` | ||
2. Upgrade the branch `release_12.x` to the LLVM/Clang sources corresponding to | ||
the 12.0.0 release. The commit hash associated with the 12.0.0 release is given | ||
by the LLVM/Clang tag `llvmorg-12.0.0`. Resolve merge conflicts and test | ||
failures if any. Record the fact that the `release_12.x` branch has been | ||
upgraded upto the commit hash given by the `llvmorg-12.0.0` tag, in the document | ||
LLVM-Upgrade-Notes.md. | ||
``` | ||
git remote add upstream https://github.com/llvm/llvm-project.git | ||
git pull upstream llvmorg-12.0.0 | ||
``` | ||
3. Push the branch `release_12.x` to the remote repository, run ADO tests | ||
on it and fix test failures to make sure that the branch is stable. | ||
``` | ||
git push origin release_12.x | ||
``` | ||
## Practical Tips | ||
|
||
1. If you use automated testing, make sure to clean the build directory first. | ||
Enough changes will likely have accumulated that things may go wrong without | ||
doing that first. | ||
|
||
2. The test failures usually stem from incorrect merges or Checked C-specific data | ||
not being initialized by new or changed constructor methods. | ||
|
||
3. You'll want to run automated tests on Linux and Windows x86/x64, as well as | ||
Linux LNT tests. You may find in some cases that tests need to be updated | ||
based on new warnings or slightly different compiler behavior. | ||
|
||
## Merge your branched baseline and master branches | ||
|
||
Once all testing is passing, you can merge your branches back into | ||
your baseline and master branches. | ||
|
||
|
||
git checkout baseline | ||
git merge updated-baseline | ||
git checkout master | ||
git merge updated-master | ||
|
||
## Push the updated branches to GitHub | ||
|
||
The changes will be extensive enough that you don't want to do a pull request | ||
4. The changes will be extensive enough that you don't want to do a pull request | ||
on GitHub. Just push the branches up to GitHub. | ||
|
||
|
||
## Handling Merge Conflicts and Test Failures | ||
|
||
1. When `merge.renamelimit` is set to a large value, the output of the | ||
`git merge` command may have several lines like: | ||
- CONFLICT (rename/delete): ... in master renamed to ... in HEAD. Version | ||
HEAD ... left in tree. | ||
- CONFLICT (rename/delete): ... in master renamed to ... in HEAD. Version | ||
master ... left in tree. | ||
|
||
If the version in HEAD is left in tree, it is very likely a valid change - | ||
the file may have been moved. If the version in `master` is left in tree, | ||
the files may be valid files that have been added as part of Checked C | ||
support: | ||
- `llvm/projects/checkedc-wrapper/lit.site.cfg.py` | ||
- `llvm/projects/checkedc-wrapper/lit.cfg.py` | ||
|
||
Others may need more investigation. For example, in the LLVM/Clang 11 to 12 | ||
merge, two files which were reported by git as "Version master ... left in | ||
tree" were old versions lying around that needed to be deleted: | ||
- `clang/include/clang/StaticAnalyzer/Core/PathSensitive/SMTAPI.h` | ||
- `clang/include/clang/AST/TypeNodes.def` | ||
|
||
2. While resolving merge conflicts, it is a good idea to have three reference | ||
code snapshots to perform diffs of individual files to help distinguish between | ||
checkedc-specific changes and llvm-11-to-12 changes: | ||
- The `master` branch of the `checkedc-clang` repository at the commit hash | ||
which was merged into `updated_baseline_master_12`. | ||
- The pristine LLVM/Clang source at <branch-point-of-11-on-main> | ||
- The pristine LLVM/Clang source at <branch-point-of-12-on-main> | ||
|
||
The description of the three snapshots above holds only for the initial | ||
merge from LLVM 11 to 12 in phase 1 step 6. For a merge of additional | ||
Checked C feature development in phase 1 step 8, here are the relevant | ||
snapshots and their commit hashes, which we get from the in-progress merge: | ||
- A snapshot of branch `updated_baseline_master_12` after phase 1 step 7, or | ||
an earlier iteration of step 8. The commit hash that corresponds to this | ||
snapshot is `HEAD` - use command `git rev-parse HEAD` to dump the commit | ||
hash. | ||
- A snapshot of the `master` branch that participated in the previous merge | ||
that produced the above snapshot. The commit hash that corresponds to this | ||
snapshot is the common ancestor - use command | ||
`git merge-base HEAD MERGE_HEAD` to dump the commit hash. | ||
- A snapshot of the current `master` branch. The commit hash that | ||
corresponds to this is `MERGE_HEAD` - use command `git rev-parse MERGE_HEAD` | ||
to dump the commit hash. | ||
|
||
Here are some alternatives to the above approach of using source snapshots | ||
to manually perform diffs of the relevant files: | ||
- Run `git mergetool` - you may need to have a merge tool that is suitable | ||
for your environment (Linux/Windows) installed and configured | ||
(ex.: `kdiff3`). | ||
- Set `git config merge.conflictstyle diff3` before running `git merge` so | ||
that the conflict markup written to the file shows all three versions rather | ||
than just two. | ||
- Run | ||
`git show :1:clang/path/to/some/File.cpp > clang/path/to/some/File.1.cpp` | ||
(and similarly with 2 or 3 in place of 1) to write out the three versions | ||
for inspection with other tools. | ||
|
||
3. When bitfields are modified (or need modification to accommodate changes | ||
related to Checked C) in files like `clang/include/clang/AST/Stmt.h` (see | ||
commit hash af12895f631), and `clang/include/clang/AST/DeclBase.h` (see the diff | ||
between commits bc556e5 and e08e3f6 - this change was required for a previous | ||
merge), more manual investigation may be needed to check the correctness of the | ||
changes. |
Oops, something went wrong.