Initial LLVM3.5 support #141

delcypher · 2014-08-21T00:00:21Z

LLVM 3.5 is going to be released very soon and RC2 is already available so the LLVM3.5 API should be stable. These commits patch KLEE to compile against LLVM3.5.

Some of the big changes are ....

we must compile with C++11 for LLVM3.5 . I've taught the configure script to do the right thing for LLVM3.5 and newer and added a configure flag to force C++11 as well.
The introduction of LLVM's ErrorOr<T> has made the code a lot more complicated.
The archive linking code is a total mess. It compiles but it's very hard to read! This needs fixing

These commits could certainly do with a tidy but I think it's a good start.

For LLVM >= 3.5 explicitly force this to be enabled otherwise we can't compile against LLVM.

``Support/Debug.h``. We need to define our own DEBUG_TYPE The LLVM change responsible for this breakage is r206822

has been removed. The commit responsible for this is r202816. We should check if this header file is actually being used for older LLVM versions because I'm not convinced it is.

removed. The commit responsible for this is r202816 and r202827.

The commit responsible for this is r202816

DEBUG_TYPE. The LLVM change responsible for this breakage is r206822

The commit responsible for this is r202816

assertion entirely?

commit responsible is r199751.

The commit responsible for this breakage is r211033.

This needs rewriting!

moving.

was removed by r210803

The commit that caused this is r202052.

was caused by r199082

MemoryBuffer from the std::unique_ptr when getLazyBitcodeModule() succesfully takes ownership.

field was added to the struct in the array of global constructors. I'm completly ignore the extra field because I don't know what it does and the LLVM LangRef isn't very clear at all about this.

delcypher · 2014-09-05T12:46:29Z

LLVM3.5 has been released and I can confirm that the commits on this pull request still compile correctly.

delcypher · 2014-09-12T12:57:06Z

Correction it doesn't compile correctly. I had the LLVM3.4 headers on my system so those were getting picked up so this doesn't compile correctly.

llvm/Linker.h -> llvm/Linker/Linker.h (r203065 responsible) llvm/Assembly/AssemblyAnnotationWriter.h -> llvm/IR/AssemblyAnnotationWriter.h (r198688 responsible) llvm/DebugInfo.h -> llvm/IR/DebugInfo.h (r203046 responsible)

It was removed by r202839.

moves it to "llvm/IR/InstIterator.h" (r202814 is responsible). It turns out only InstructionInfoTable.cpp was actually using this include so I just removed the include from the other files.

delcypher · 2014-09-12T13:46:08Z

Okay compilation is fixed. These commits need a massive tidy up though.

ddunbar · 2014-09-12T19:24:29Z

I want to start landing some of these patches, so we can sort out the ones
that need work from the rest. Any objections to me cherry picking some of
your work over to master?

On Fri, Sep 12, 2014 at 6:46 AM, Dan Liew notifications@github.com wrote:

Okay compilation is fixed. These commits need a massive tidy up though.

—
Reply to this email directly or view it on GitHub
#141 (comment).

delcypher · 2014-09-12T19:58:23Z

@ddunbar No problem. Just to warn you some of them are gross. The "linking" one is the worst. You're probably aware of this but support for linking archives of bitcode modules was dropped. As a temporary solution I duplicated the linking code from LLVM3.2 and implemented our own linker. I think this is the wrong way to do things. I think all the linking and (most optimisations) should be done outside of KLEE. That way

We use a production quality linker (ld gold) rather than the pile of crap I wrote
Optimizations actually reflect what clang really does (e.g. -O3)

ddunbar · 2014-09-12T20:01:16Z

I am actually the one that ripped out the linking code from LLVM. :)

Yeah, that is gross, but I am ok with it as a temporary way forward. It is
true that in general KLEE should move towards people using proper linker
support for merging bitcode files in order to get its input, but the
current solution has some benefits for now.

Given that we only use the linking code for pulling in our own archive
objects, an alternate way to do this would be to link those archive objects
into a single LLVM bitcode ourselves at build time, and then just link the
two modules (and strip unused functions, ideally).

On Fri, Sep 12, 2014 at 12:58 PM, Dan Liew notifications@github.com wrote:

@ddunbar https://github.com/ddunbar No problem. Just to warn you some
of them are gross. The "linking" one is the worst. You're probably aware of
this but support for linking archives of bitcode modules was dropped. As a
temporary solution I duplicated the linking code from LLVM3.2 and
implemented our own linker. I think this is the wrong way to do things. I
think all the linking and (most optimisations) should be done outside of
KLEE. That way

We use a production quality linker (ld gold) rather than the pile of
crap I wrote

Optimizations actually reflect what clang really does (e.g. -O3)

—
Reply to this email directly or view it on GitHub
#141 (comment).

delcypher · 2014-09-12T20:10:06Z

Given that we only use the linking code for pulling in our own archive
objects, an alternate way to do this would be to link those archive objects
into a single LLVM bitcode ourselves at build time, and then just link the
two modules (and strip unused functions, ideally).

IIRC @MartinNowack tried that and the performance was terrible because klee-uclibc as a single LLVM bitcode module is huge and linking the module you want to run with it takes quite a while.

MartinNowack · 2014-09-12T20:22:23Z

The total run time for test suite is according to @delcyper's tests (#70)

88.16s for current implementation (implemented by @delcyper)
139.05s prelinking all modules as suggested workaround (I suggested the workaround due to the slowness)
508.84s conservative linking (my first implementation)

I’m also more for removing the whole opt stuff out of it and providing KLEE with the final bc file.

On 12 Sep 2014, at 22:10, Dan Liew notifications@github.com wrote:

Given that we only use the linking code for pulling in our own archive
objects, an alternate way to do this would be to link those archive objects
into a single LLVM bitcode ourselves at build time, and then just link the
two modules (and strip unused functions, ideally).

IIRC @MartinNowack tried that and the performance was terrible because klee-uclibc as a single LLVM bitcode module is huge and linking the module you want to run with it takes quite a while.

—
Reply to this email directly or view it on GitHub.

ddunbar · 2014-09-12T20:38:35Z

Ah, yes, I can see how the performance might be an issue. Whether or not we
optimize separately is a separate issue though (do we have a separate bug
tracking whether or not to remove that code? I often found it useful to be
able to quickly turn optimizations on during different test runs, but I can
see the argument for ditching it).

For now, I think we should just take the approach of bringing in the legacy
code from LLVM to keep the old behavior. I'll have to think more about what
the right solution is. I think probably what we (LLVM) want here is for
there to be a method in LLVM that allows lazily linking in a module to
another one, giving the performance benefits of the archive style but
without the complexity overhead of actually having to have Archive reading
support.

On Fri, Sep 12, 2014 at 1:22 PM, MartinNowack notifications@github.com
wrote:

The total run time for test suite is according to @delcyper's tests (
#70)

88.16s for current implementation (implemented by @delcyper)

139.05s prelinking all modules as suggested workaround (I suggested the
workaround due to the slowness)

508.84s conservative linking (my first implementation)

I’m also more for removing the whole opt stuff out of it and providing
KLEE with the final bc file.

On 12 Sep 2014, at 22:10, Dan Liew notifications@github.com wrote:

Given that we only use the linking code for pulling in our own archive
objects, an alternate way to do this would be to link those archive
objects
into a single LLVM bitcode ourselves at build time, and then just link
the
two modules (and strip unused functions, ideally).

IIRC @MartinNowack tried that and the performance was terrible because
klee-uclibc as a single LLVM bitcode module is huge and linking the module
you want to run with it takes quite a while.

—
Reply to this email directly or view it on GitHub.

—
Reply to this email directly or view it on GitHub
#141 (comment).

ddunbar · 2014-09-14T23:43:05Z

I just landed a bunch of the trivial ones on master. I ended up rewriting some of the commits which is going to make it hard to rebase this branch, but I plan on continuing to work through this. The main thing left is the changes to ModuleUtil, and then a pass over everything else to make sure I didn't miss any semantic changes.

delcypher · 2014-09-15T14:42:20Z

@ddunbar We can probably just chuck this branch. It was more of test to see if I could port it to LLVM3.5. If you cherry-pick the good commits from this branch then we can try to tackle the difficult parts in a cleaner way.

I think probably what we (LLVM) want here is for
there to be a method in LLVM that allows lazily linking in a module to
another one, giving the performance benefits of the archive style but
without the complexity overhead of actually having to have Archive reading
support.

Does getLazyBitcodeModule() serve that purpose? I remember trying to use it the linking code I wrote but it didn't work (I couldn't read the symbol table correctly) so I just loaded the modules the normal way.

ddunbar · 2014-09-15T15:22:36Z

It's the underpinnings, but what we probably should have in LLVM is a
function that lazily links in a module (presumably with the lazy side being
a getLazyBitcodeModule() result). Like Linker::linkInModule, but where it
only brings in the code it needs from the other module.

On Mon, Sep 15, 2014 at 7:42 AM, Dan Liew notifications@github.com wrote:

@ddunbar https://github.com/ddunbar We can probably just chuck this
branch. It was more of test to see if I could port it to LLVM3.5. If you
cherry-pick the good commits from this branch then we can try to tackle the
difficult parts in a cleaner way.

I think probably what we (LLVM) want here is for
there to be a method in LLVM that allows lazily linking in a module to
another one, giving the performance benefits of the archive style but
without the complexity overhead of actually having to have Archive reading
support.

Does getLazyBitcodeModule() serve that purpose? I remember trying to use
it the linking code I wrote but it didn't work (I couldn't read the symbol
table correctly) so I just loaded the modules the normal way.

—
Reply to this email directly or view it on GitHub
#141 (comment).

banescusebi · 2015-01-10T10:27:30Z

Hi all, is there any news about getting KLEE to work with LLVM 3.5 in the near future? I've just tried it out with the latest commit of KLEE and I get similar errors to the ones in the following mailing list message: https://www.mail-archive.com/klee-dev@imperial.ac.uk/msg01770.html��

delcypher · 2015-01-10T12:09:00Z

@banescusebi I wish I had time to work on this but right now I don't. You're quite welcome to have a go at it and I'll happily review a pull request.

ccadar · 2015-08-03T15:20:32Z

Closing this branch, as some commits were already pushed and the rest were incorporated as part of #253.

Dan Liew added 24 commits August 20, 2014 18:29

Provide --enable-cxx11 configure option to enable building with C++11.

d297858

For LLVM >= 3.5 explicitly force this to be enabled otherwise we can't compile against LLVM.

Fixed broken compilation for LLVM35 due to a change in

088fff7

``Support/Debug.h``. We need to define our own DEBUG_TYPE The LLVM change responsible for this breakage is r206822

Fix compilation error due to change in raw_fd_ostream for LLVM3.5

519e1f5

Fix compilation error for LLVM3.5 . It seems Support/CallSite.h

f7285d0

has been removed. The commit responsible for this is r202816. We should check if this header file is actually being used for older LLVM versions because I'm not convinced it is.

Fix compilation error for LLVM3.5. It seems Support/CFG.h was

98618dd

removed. The commit responsible for this is r202816 and r202827.

Fix LLVM3.5 compilation error. Support/CallSite.h has moved.

eedfa06

The commit responsible for this is r202816

Fix more breakages due to the need for us to define our own

f0be033

DEBUG_TYPE. The LLVM change responsible for this breakage is r206822

Fix another LLVM3.5 compilation error. Support/CallSite.h has moved.

48365ca

The commit responsible for this is r202816

Fix compilation error due to r199218. Maybe we should just remove this

8d41503

assertion entirely?

Fix compilation issue due to change in Object/Archive.h. The

38cc29a

commit responsible is r199751.

Start fixing LLVM3.5 build relating to changes in the Object library.

6b214c8

The commit responsible for this breakage is r211033.

Apply gross fixes for LLVM linking code for LLVM3.5

c79a9d2

This needs rewriting!

Minor fix for LLVM3.5

b8e0412

Fix LLVM3.5 compilation. This is due to r202168

edcfa60

Fix more LLVM3.5 compilation errors due to CFG.h and CallSite.h

272d920

moving.

Fix LLVM3.5 compilation due to switching to std::error_code

1f9e62c

Fix LLVM3.5 compilation a little more. Support/system_error.h

82ea60b

was removed by r210803

Another LLVM3.5 compilation fix.

09394e4

Compilation fix for LLVM3.5 caused by the removal of sys::fs::F_Binary.

694e8c2

The commit that caused this is r202052.

Fix more LLVM3.5 compilation issues.

e11095f

Fix LLVM3.5 compilation issue caused by the Verifier.h moving. This

72a2fa7

was caused by r199082

Fix kleaver compilation under LLVM3.5

766eb47

Fix segfault under LLVM3.5 . I forgot to release ownership of the

86d5e6d

MemoryBuffer from the std::unique_ptr when getLazyBitcodeModule() succesfully takes ownership.

Fixed broken assertion under LLVM3.5 . It seems in r209015 an extra

7ec56e2

field was added to the struct in the array of global constructors. I'm completly ignore the extra field because I don't know what it does and the LLVM LangRef isn't very clear at all about this.

Dan Liew added 4 commits September 12, 2014 14:21

Fix more header includes for LLVM3.5

08337e8

llvm/Linker.h -> llvm/Linker/Linker.h (r203065 responsible) llvm/Assembly/AssemblyAnnotationWriter.h -> llvm/IR/AssemblyAnnotationWriter.h (r198688 responsible) llvm/DebugInfo.h -> llvm/IR/DebugInfo.h (r203046 responsible)

Fix LLVM3.5 header include llvm/Support/PassNameParser.h

6a1de76

It was removed by r202839.

Fix #include under LLVM3.5. OwningPtr doesn't exist anymore.

2c5a25f

Fixed use of #include "llvm/Support/InstIterator.h" in LLVM3.5 which

6e3eae6

moves it to "llvm/IR/InstIterator.h" (r202814 is responsible). It turns out only InstructionInfoTable.cpp was actually using this include so I just removed the include from the other files.

ddunbar force-pushed the master branch from 1a36f7c to e87af57 Compare September 13, 2014 00:36

ccadar mentioned this pull request Mar 13, 2015

LLVM 3.5.1 compatibility #208

Closed

ccadar closed this Aug 3, 2015

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Initial LLVM3.5 support #141

Initial LLVM3.5 support #141

delcypher commented Aug 21, 2014

delcypher commented Sep 5, 2014

delcypher commented Sep 12, 2014

delcypher commented Sep 12, 2014

ddunbar commented Sep 12, 2014

delcypher commented Sep 12, 2014

ddunbar commented Sep 12, 2014

delcypher commented Sep 12, 2014

MartinNowack commented Sep 12, 2014

ddunbar commented Sep 12, 2014

ddunbar commented Sep 14, 2014

delcypher commented Sep 15, 2014

ddunbar commented Sep 15, 2014

banescusebi commented Jan 10, 2015

delcypher commented Jan 10, 2015

ccadar commented Aug 3, 2015

Initial LLVM3.5 support #141

Initial LLVM3.5 support #141

Conversation

delcypher commented Aug 21, 2014

delcypher commented Sep 5, 2014

delcypher commented Sep 12, 2014

delcypher commented Sep 12, 2014

ddunbar commented Sep 12, 2014

delcypher commented Sep 12, 2014

ddunbar commented Sep 12, 2014

delcypher commented Sep 12, 2014

MartinNowack commented Sep 12, 2014

ddunbar commented Sep 12, 2014

ddunbar commented Sep 14, 2014

delcypher commented Sep 15, 2014

ddunbar commented Sep 15, 2014

banescusebi commented Jan 10, 2015

delcypher commented Jan 10, 2015

ccadar commented Aug 3, 2015