Skip to content

Conversation

pfez
Copy link
Contributor

@pfez pfez commented May 27, 2021

This PR is still a draft.

It's known to compile, and pass all the llvm testsuite with orc install --test llvm, with the only pre-requisite that orchestra is based on revng/orchestra#36

revng can be successfully built from this, but there are still some failing tests (see revng/revng#76).

This patchset is still dirty: there are still some explictly reverted out-of-tree patches in the history. They are not required for building llvm, nor for revng, but they might still be needed for successfully building some downstream project (like revng-c, caliban, or cold-revng). However, until all revng tests pass, we are not able to verify this on downstream projects. When we fix the revng tests failing, I'll figure out if these pending reverted patches need to be applied (so I'll remove the revert commit), or can be dropped (so I'll leave them entirely out of the new history).

@aleclearmind when you're done fixing revng tests, I'll figure out this pending reverts issues an update the PR.

@pfez pfez requested a review from aleclearmind May 27, 2021 10:36
@pfez pfez force-pushed the feature/llvm-12 branch 2 times, most recently from 8f79642 to 3b8a5ce Compare May 28, 2021 13:39
@pfez pfez marked this pull request as ready for review May 28, 2021 13:40
@pfez
Copy link
Contributor Author

pfez commented May 28, 2021

UPDATE

  • I managed to compile all our downstream projects up to ui/cold-revng
  • I dropped both the useless reverts and the original patches they were reverting, since it's confirmed they are not needed anymore
  • There are still failing tests in revng and revng-c

@aleclearmind
Copy link
Contributor

Looks great! LLVM 12, here we come.

Just a couple of comments:

  1. "APFixedPointTest.cpp: fix C++20 compilation error": can you briefly describe the problem in the commit message? Dumping the error is not very nice.
  2. "Fix C++20 UTF8 string literals" is a bit overkill IMO. It uses a series of macro tricks to fix a bug that doesn't really concern us in a test. I would have been fine with disabling the test. This kind of effort makes sense if we want to upstream the fixes, but updating LLVM is already quite time consuming as it is now.
  3. "Add support to Dominators for FilteredGraphTraits": do we really, really, really need to rename DomTreeNodeBase to DomTreeNodeOnView and the like? Looking at the patchset with proper default values the patch could shrink in size significantly. A change in a well isolated feature ends up touching files everywhere. It feels like it increases the rebase burden with little return.

@aleclearmind
Copy link
Contributor

We have a failing test due to the fact that we're running the test suite as root. Basically, a command is expected to fail but it doesn't since root bypasses permissions checks.
Gentoo is experiencing the bug too.

Given also the other problematic tests, I suggest we have a series of commits in our LLVM fork that disable tests that we know are going to fail but we don't really care that much. Then, every time we upgrade LLVM we drop those commits and see if any of these issue has been solved.
It's quite a waste of time trying to fix this stuff for us and increases the friction towards upgrading LLVM, which is very important for us.

I know this is suboptimal, but I think it's the most reasonable thing to do.

These are the two tests that are failing due to permission issues:

  LLVM :: tools/llvm-ar/error-opening-permission.test
  LLVM :: tools/llvm-elfabi/fail-file-write.test

Can you please disable them?

@pfez
Copy link
Contributor Author

pfez commented Jun 1, 2021

These are the two tests that are failing due to permission issues:

  LLVM :: tools/llvm-ar/error-opening-permission.test
  LLVM :: tools/llvm-elfabi/fail-file-write.test

Can you please disable them?

Done!

Starting from C++20, the equality comparison should be reversible, i.e.
the expression `A == B` and `B == A` should select the same overload for
`operator==()`. If this does not happen, the language treats the call to
`operator==()` as ambiguous.

The ASSERT_EQ performs an equality comparison under the hood, using
`operator==()`. In the following expression, the LHS has type int64_t,
while the RHS had type uint64_t before this commit.

  ASSERT_EQ(APFixedPoint::getMax(Sema).getIntPart(), Expected);

Because of the operator overloads of `APSInt`, this caused the call to
`operator==()` hidden inside ASSERT_EQ to be ambiguous between:

1) /orchestra/sources/llvm/llvm/include/llvm/ADT/APInt.h:2035:13:
inline bool operator==(uint64_t V1, const APInt &V2) { return V2 == V1; }
            ^
2) /orchestra/sources/llvm/llvm/include/llvm/ADT/APSInt.h:339:13:
inline bool operator==(int64_t V1, const APSInt &V2) { return V2 == V1; }
            ^

The ambiguity is resolved changing the type of `Expected` to `int64_t`.
@aleclearmind
Copy link
Contributor

aleclearmind commented Jun 1, 2021

OK, once the "APFixedPointTest.cpp: fix C++20 compilation error" thing is done, and the CI passes is OKGO.

pfez added 5 commits June 1, 2021 10:39
Since C++20, UTF8 string literals (in the form `u8"blabla"`) have type
`const char8_t[N]`, which does not play nice with llvm::StringRef and
llvm::StringLiteral.

This commit fixes compilation of DJBTest.cpp and in C++20 explicitly
converting those strings literals to regular string literals.

To fix compilation of DJBTest.cpp, we had to define a new UTF8 based
constructor for StringRef and one for StringLiteral.
These cannot be constexpr because const char8_t * cannot be converted at
compile time to const char *.

With these new constructors, the StringRef(data, length) constructor
becomes ambiguous if used with nullptr as first argument, such as
  StringRef(nullptr, 0);
This was done in XCOFFObjectFile.cpp that was fixed to use the default
StringRef constructor.
The exact warning was:
include/clang/AST/DeclarationName.h:210:52: error: arithmetic between different enumeration types ('clang::DeclarationName::StoredNameKind' and 'clang::detail::DeclarationNameExtra::ExtraKind') is deprecated [-Werror,-Wdeprecated-enum-enum-conversion]
Before this patch, LLVM's DominatorTree and related templates
(DominatorTree, DomTreeBuilder) did not support the computation of
dominator and post-dominator trees on graphs with markers, such as
GraphTraits<llvm::Inverse<T>>.
They only supported plain graph traits.

This patch restructures all the necessary templates, adding an
additional template argument (View), which allows to compute DT and PDT
on all the graphs that use GraphTraits, potentially equipped with a
View.
View is a marker, that allows to provide a different view on a graph
that has GraphTraits.
A typical example of this is llvm::Inverse.

This mechanism of View markers will be used in rev.ng to implement
filtered graph traits, to filter the edges of a graph implemented with
GraphTraits.
The new DT and PDT on the marked version of the GraphTraits will be
computed on the same graphs, but treating some edges as if they are seen
through the view.
For instance, the Post-Dominator Tree of a graph marked with
llvm::Inverse view is the Dominator Tree.

The previous default behavior of DT and PDT is implemented by means of
the DTIdentityView, which simply leaves the node type of the GraphTraits
untouched.
This is a transparent view that allows all the code in llvm to continue
working as before this patch.
These tests pass successfully when running as unprivileged user, but
fail if run as `root`, because the superuser has privileges to ignore
file permissions.

This is a known issue, reported e.g. by Gentoo as well:
  https://bugs.gentoo.org/775050

For now, disable these tests, since they fail in docker on our CI.
@pfez pfez force-pushed the feature/llvm-12 branch from c97371c to 62f0749 Compare June 1, 2021 08:42
@pfez
Copy link
Contributor Author

pfez commented Jun 1, 2021

OK, once the "APFixedPointTest.cpp: fix C++20 compilation error" thing is done, and the CI passes is OKGO.

Done! See if you like the commit message better.

@aleclearmind
Copy link
Contributor

Done! See if you like the commit message better.

Looks great.
Now running on the CI.

@aleclearmind
Copy link
Contributor

Merged in c97371c. Thanks!

@pfez pfez mentioned this pull request Jun 16, 2021
tvandera pushed a commit to tvandera/llvm-project that referenced this pull request Oct 5, 2021
Andrei Matei reported a llvm11 core dump for his bpf program
   https://bugs.llvm.org/show_bug.cgi?id=48578
The core dump happens in LiveVariables analysis phase.
  revng#4 0x00007fce54356bb0 __restore_rt
  revng#5 0x00007fce4d51785e llvm::LiveVariables::HandleVirtRegUse(unsigned int,
      llvm::MachineBasicBlock*, llvm::MachineInstr&)
  revng#6 0x00007fce4d519abe llvm::LiveVariables::runOnInstr(llvm::MachineInstr&,
      llvm::SmallVectorImpl<unsigned int>&)
  revng#7 0x00007fce4d519ec6 llvm::LiveVariables::runOnBlock(llvm::MachineBasicBlock*, unsigned int)
  revng#8 0x00007fce4d51a4bf llvm::LiveVariables::runOnMachineFunction(llvm::MachineFunction&)
The bug can be reproduced with llvm12 and latest trunk as well.

Futher analysis shows that there is a bug in BPF peephole
TRUNC elimination optimization, which tries to remove
unnecessary TRUNC operations (a <<= 32; a >>= 32).
Specifically, the compiler did wrong transformation for the
following patterns:
   %1 = LDW ...
   %2 = SLL_ri %1, 32
   %3 = SRL_ri %2, 32
   ... %3 ...
   %4 = SRA_ri %2, 32
   ... %4 ...

The current transformation did not check how many uses of %2
and did transformation like
   %1 = LDW ...
   ... %1 ...
   %4 = SRL_ri %2, 32
   ... %4 ...
and pseudo register %2 is used by not defined and
caused LiveVariables analysis core dump.

To fix the issue, when traversing back from SRL_ri to SLL_ri,
check to ensure SLL_ri has only one use. Otherwise, don't
do transformation.

Differential Revision: https://reviews.llvm.org/D97792

(cherry picked from commit 51cdb78)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants