How to update to LLVM 3.5?

[4ntoine]

No description provided.

[kripken]

Hopefully this can happen eventually. But we should rebase on llvm svn.

[tomaka]

I'm eagerly waiting for this update in order to try making the Rust programming language compatible with Emscripten 😃
(see rust-lang/rust#2235)

Someone (sorry, I don't remember the nickname) on the #emscripten IRC channel told me that he was working on it, I hope it's still the case.

[4ntoine]

I'm also waiting for it. I can do it myself if having some description on how to do it.

[juj]

Emscripten-fastcomp is not a direct fork of upstream LLVM ( https://github.com/llvm-mirror/llvm ), but a fork of Google Native Client fork of LLVM. However Google uses depot tools and their repos seem to be all over the place, so this is not reflected in the github main page of this repository that it's a fork of another repo.

I believe it's the PNaCl fork of LLVM (as opposed to the older NaCl project), and the upstream repository is probably this one: http://git.chromium.org/gitweb/?p=native_client/pnacl-llvm.git;a=tree;hb=HEAD . Although I am not completely sure, @kripken can verify.

One could try to update with different strategies:

1. Update emscripten-fastcomp up-to-date to latest Native Client upstream. If the above repo is the correct one, it looks like the upstream repo is going at LLVM 3.4, so they have somewhat never LLVM upstream in use.
2. Update upstream Native Client to latest LLVM 3.5 first, then update emscripten-fastcomp over to the updated Native Client repository. Contribute the Native Client update so that we keep in sync with it for emscripten-fastcomp.
3. Detach emscripten-fastcomp from the Native Client upstream and rebase it to the LLVM upstream repository. This would make our development pace independent of updates to PNaCl itself. Also, making this repository an explicit fork of https://github.com/llvm-mirror/llvm might be beneficial.

Also when merging, one must remember to merge both https://github.com/kripken/emscripten-fastcomp and https://github.com/kripken/emscripten-fastcomp-clang repositories.

[4ntoine]

Thanks! I think i will find free time this week to do it and report you
about my progress.

Anton.

2014-10-07 16:47 GMT+06:00 juj notifications@github.com:

Emscripten-fastcomp is not a direct fork of upstream LLVM (
https://github.com/llvm-mirror/llvm ), but a fork of Google Native Client
fork of LLVM. However Google uses depot tools and their repos seem to be
all over the place, so this is not reflected in the github main page of
this repository that it's a fork of another repo.

I believe it's the PNaCl fork of LLVM (as opposed to the older NaCl
project), and the upstream repository is probably this one:
http://git.chromium.org/gitweb/?p=native_client/pnacl-llvm.git;a=tree;hb=HEAD
. Although I am not completely sure, @kripken https://github.com/kripken
can verify.

One could try to update with different strategies:

1. Update emscripten-fastcomp up-to-date to latest Native Client
   upstream. If the above repo is the correct one, it looks like the upstream
   repo is going at LLVM 3.4, so they have somewhat never LLVM upstream in use.
2. Update upstream Native Client to latest LLVM 3.5 first, then update
   emscripten-fastcomp over to the updated Native Client repository.
   Contribute the Native Client update so that we keep in sync with it for
   emscripten-fastcomp.
3. Detach emscripten-fastcomp from the Native Client upstream and
   rebase it to the LLVM upstream repository. This would make our development
   pace independent of updates to PNaCl itself. Also, making this repository
   an explicit fork of https://github.com/llvm-mirror/llvm might be
   beneficial.

Also when merging, one must remember to merge both
https://github.com/kripken/emscripten-fastcomp and
https://github.com/kripken/emscripten-fastcomp-clang repositories.

—
Reply to this email directly or view it on GitHub
#51 (comment)
.

[kripken]

While we have been based off of PNaCl, we should probably rebase off of upstream LLVM svn at this point, as I also just said on irc. Reasons are that PNaCl lags behind LLVM svn, and our usage of PNaCl passes has decreased to a small number. So taking our new JSBackend code, plus the few PNaCl passes we need, and rebasing that on LLVM svn, seems the simplest route. However, other options are fine too if they end up easier.

[aidanhs]

Done some quick investigation with some help from git merge-base.

LLVM:

Emscripten llvm takes llvm pnacl from
 - 7026af7138fccfb256456b04b375d39b025bdb7c
the most recent llvm trunk merged into pnacl master at this point was
 - 7dfcb84fc16b3bf6b2379713b53090757f0a45f9 (r33)

emscripten-fastcomp $ git diff --shortstat 7026af7138 master # changes between pnacl and emscripten
 88 files changed, 10637 insertions(+), 2420 deletions(-)

pnacl-llvm $ git diff --shortstat 7dfcb84fc 7026af7138 # changes between trunk and pnacl
 512 files changed, 40935 insertions(+), 958 deletions(-)

LLVM pnacl after they merged trunk r34 and reapplied pnacl changes
 - fe90708cae889704bba598c5d83eb4d8f38a37c7
the most recent llvm trunk merged into pnacl master at this point was
 - 36c7806f4eacd676932ba630246f88e0e37b1cd4 (r34)

pnacl-llvm $ git diff --shortstat 36c7806f fe90708 # changes between trunk and pnacl
 476 files changed, 45266 insertions(+), 390 deletions(-)

Clang:

Emscripten clang takes clang pnacl from
 - a963b803407c9d1cac644cc425004e0ccd28fa45
the most recent clang trunk merged into pnacl master at this point was
 - ca0dd25d1d053be4557fe32691e31e728b30fad5 (r33)

emscripten-fastcomp-clang $ git diff --shortstat a963b80340 master # changes between pnacl and emscripten
 16 files changed, 493 insertions(+), 6 deletions(-)

I stopped looking at Clang because clearly the main effort is going to be moving LLVM.

After squashing all emscripten commits I did some experiments:

• cherry pick squashed commit onto vanilla r33 llvm (7dfcb84), i.e. pretending we're based of trunk
  • 16 files conflicted, mostly modifications to the (now non-existent) nacl files
  • would need to steal the appropriate nacl passes
• cherry pick squashed commit onto pnacl llvm r34 (fe90708)
  • 11 files conflicted
  • nacl passes should 'just work'

I may have a look at how easy it is to lift the appropriate passes from pnacl. Without that knowledge the easiest way looks like rebasing onto pnacl llvm r34 and creeping forward to r35 when available.

There's also the question of rebasing or merging. Merging seems like something that would work better in the long term given this 'rolling forward' will happen multiple times.

[kripken]

Interesting, thanks for looking into this.

I have heard pnacl is close to merging 3.5, at which point it might be easiest for us to base off of that, in collaboration with them.

Yes, merging is probably better.

[aidanhs]

In which case I'll have a look at getting forward to 3.4, since (in theory) that should make the transition to 3.5 easier.

[aidanhs]

I'm down to 3 conflicts of .cpp files which require more knowledge than I have. I've created a repo to do the dirty work of getting to that point.

If you clone https://github.com/aidanhs/rebase-emscripten and run script.sh it will

• do a one-time clone of this repo and pnacl repo
• undo emscripten changes (and save the diff into a patch file)
• move forward on pnacl
• apply some changes to the emscripten changes diff to make it apply more cleanly
• reapply emscripten changes

You'll be left with 3 *.rej files (patch hunks) which need to be reviewed, manually applied and deleted.
Then you can commit all changes.

The script is pretty simple and gives good guidance on what to for emscripten-fastcomp-clang.

Let me know if you have any questions.

[kripken]

Thanks! Taking a look now.

[kripken]

Ok, I merged those 3 files manually. I committed and pushed the result to pnacl-merge, https://github.com/kripken/emscripten-fastcomp/tree/pnacl-merge

I didn't try to build yet though, as I noticed that lib/Target/JSBackend (our backend) is missing. Was that supposed to happen?

[aidanhs]

In addition to the stuff it puts in the staging area, you also should end up with a bunch of untracked files, including lib/Target/JSBackend/.

Looking at your diff, it doesn't look like any files have actually been added - git add .?

[kripken]

Ok, thanks, I didn't do that important step ;)

Done now, and pushed to that branch. Starting to try to build locally.

[jfbastien]

Now that I'm back from vacation, I think I can land the PNaCl merge to 3.5 in the next few days, and then work with Alon to update Emscripten too.

[kripken]

Ok, I pushed a few commits and LLVM builds. That leaves updating clang, which should be easier, maybe.

After doing this work, I'm still a little unsure about whether rebasing or merging is going to be the best workflow, if this is to be a frequent thing. Needs more discussion.

[aidanhs]

I'll take a look at writing a script for clang. It should be much easier.

The issue with rebasing and force pushing to the same branch is that you totally lose the ability to go back to a previous version, which is bad. Pushing to a new branch could be a solution but it makes it a bit mindbending to figure out the history of changes without a guide (same change on multiple branches on different bases!? Do you need to start thinking about backporting?).

Unfortunately merges make history is a bit messy. I think I'd prefer it over the alternatives though...

[juj]

I would definitely favor merging - as long as merges flow one direction
only, they should be ok to read. If rebased, it kills the possibility to
track development between upstream LLVM, PNaCl and us, and kills bisection
opportunities if future updates cause regressions.

2014-11-18 0:59 GMT+02:00 aidanhs notifications@github.com:

I'll take a look at writing a script for clang. It should be much easier.

The issue with rebasing and force pushing to the same branch is that you
totally lose the ability to go back to a previous version, which is bad.
Pushing to a new branch could be a solution but it makes it a bit
mindbending to figure out the history of changes without a guide (same
change on multiple branches on different bases!? Do you need to start
thinking about backporting?).

Unfortunately merges make history is a bit messy. I think I'd prefer it
over the alternatives though...

—
Reply to this email directly or view it on GitHub
#51 (comment)
.

[aidanhs]

Script done. Same repo (https://github.com/aidanhs/rebase-emscripten), now imaginatively called 'script2.sh'.

There weren't any tricky merge conflicts, so you shouldn't see any .rej files listed at the end, i.e. you can just add, commit and try it out.

[aidanhs]

Just had to tweak the script, re-pull if you've already grabbed it.

[kripken]

How do you find the 'base' commit in each case? That is, the point where "emscripten" stuff starts, that you roll back to?

[aidanhs]

~/Desktop/js/emrebase/emscripten-fastcomp-clang $ git merge-base incoming pnacl-clang
a963b803407c9d1cac644cc425004e0ccd28fa45

I then use gitk to manually check the branch graph to make sure it looks sane.

[kripken]

I see, thanks.

Ok, after thinking some more, and chatting with @jfbastien about how they merge llvm into pnacl, I think the merge approach is better. I'll try to do that today. Hopefully it will be as simple as just doing the merge and then applying the diff that you made here, plus the other changes I did on top of that.

I also think we should do this on 3.4, as we are doing now, then let that stabilize for a while before tackling 3.5.

[kripken]

Ok, after the work done here so far, this was fairly easy. I got it to build ok and pushed merge-pnacl-3.4 branches to fastcomp and fastcomp-clang. Next is to go through the test suite.

[kripken]

I am running into a git issue maybe someone here can fix for me. I have a merge commit, then I wrote some commits on top of that. I want to squash the commits into the merge commit (so that bisection will work). However, e.g. git rebase -i HEAD~3 doesn't seem to work - the list of commits it gives me seems to include all the commits in the branch I merged, i.e., not just the 3 commits I asked to, instead I see thousands. Anyone know what I'm doing wrong?

[juj]

Hmm, not sure about interactive rebase, but this probably will work:

1. check out to the last commit to include
2. make a backup branch to that commit just in case (soft reset will change what the branch points to)
3. do a soft reset to the merge commit: git reset --soft hash_of_the_merge_commit
4. do an amend commit to the merge: git commit --amend -a

The soft reset will not touch the working tree, but changes the commit that the currently checked out branch points to, so it will look like as if you just authored the changes to the working tree. Then an amend commit allows you to fuse those changes into the merge commit, as if you had originally written them before committing your merge.

[aidanhs]

As always with git, there are many ways to do it. I don't specifically know how to make rebase behave, but as an example of what you could do (I tend to prefer the manual routes of diffing and applying):

emscripten-fastcomp $ git checkout merge-pnacl-3.4
emscripten-fastcomp $ git diff HEAD~3 HEAD > mydiff # make squashed diff
emscripten-fastcomp $ git checkout HEAD~3
[...]
emscripten-fastcomp $ git checkout -b mybranch # new branch from where the squashed diff should go
emscripten-fastcomp $ git apply mydiff # apply squashed diff
emscripten-fastcomp $ rm mydiff
emscripten-fastcomp $ git add .
emscripten-fastcomp $ git commit -m blah
[mybranch 3bf277f] blah
 3 files changed, 15 insertions(+), 4 deletions(-)
emscripten-fastcomp $ # check the branch looks ok
emscripten-fastcomp $ git branch -D merge-pnacl-3.4 # delete old branch
emscripten-fastcomp $ git checkout -b merge-pnacl-3.4 # recreate new branch with squashed diff
emscripten-fastcomp $ git push -f origin merge-pnacl-3.4

You could shorten this by doing git reset --hard HEAD~3 and git apply mydiff (i.e. do all your work on the merge-pnacl-3.4 branch), but the way I've outlined lets you check things over before migrating the branch back.

[aidanhs]

The reset --soft idea is pretty cool actually, probably a better approach.

[kripken]

Thanks, now I think I finally understand what --soft is good for ;)

Will try this later today, still some test failures happening.

[kripken]

Main test suite now passes. I disabled some things in non-fastcomp to get there.

Running fuzzing and the rest of the test suite now. Hopefully I can merge this to incoming tomorrow.

[kripken]

libc++ patch looks tiny and reasonable, and fixes the embind problem. Pushed to incoming. Hopefully they'll apply this upstream soon, but it's a pretty small divergence for now.

[tomaka]

Sorry for the noobish question, but I tried running ./emsdk install sdk-incoming-64bit and I'm getting the following errors:

Linking CXX executable ../../bin/opt
CMakeFiles/opt.dir/opt.cpp.o: dans la fonction « main »:
/home/pierre/Projets/emsdk_portable/clang/fastcomp/src/tools/opt/opt.cpp:404: référence indéfinie vers « llvm::initializeAddPNaClExternalDeclsPass(llvm::PassRegistry&) »
/home/pierre/Projets/emsdk_portable/clang/fastcomp/src/tools/opt/opt.cpp:405: référence indéfinie vers « llvm::initializeAllocateDataSegmentPass(llvm::PassRegistry&) »
/home/pierre/Projets/emsdk_portable/clang/fastcomp/src/tools/opt/opt.cpp:406: référence indéfinie vers « llvm::initializeBackendCanonicalizePass(llvm::PassRegistry&) »
/home/pierre/Projets/emsdk_portable/clang/fastcomp/src/tools/opt/opt.cpp:407: référence indéfinie vers « llvm::initializeCanonicalizeMemIntrinsicsPass(llvm::PassRegistry&) »
/home/pierre/Projets/emsdk_portable/clang/fastcomp/src/tools/opt/opt.cpp:408: référence indéfinie vers « llvm::initializeConstantInsertExtractElementIndexPass(llvm::PassRegistry&) »
/home/pierre/Projets/emsdk_portable/clang/fastcomp/src/tools/opt/opt.cpp:409: référence indéfinie vers « llvm::initializeExpandAllocasPass(llvm::PassRegistry&) »
/home/pierre/Projets/emsdk_portable/clang/fastcomp/src/tools/opt/opt.cpp:410: référence indéfinie vers « llvm::initializeExpandArithWithOverflowPass(llvm::PassRegistry&) »
/home/pierre/Projets/emsdk_portable/clang/fastcomp/src/tools/opt/opt.cpp:411: référence indéfinie vers « llvm::initializeExpandByValPass(llvm::PassRegistry&) »
/home/pierre/Projets/emsdk_portable/clang/fastcomp/src/tools/opt/opt.cpp:412: référence indéfinie vers « llvm::initializeExpandConstantExprPass(llvm::PassRegistry&) »
/home/pierre/Projets/emsdk_portable/clang/fastcomp/src/tools/opt/opt.cpp:413: référence indéfinie vers « llvm::initializeExpandCtorsPass(llvm::PassRegistry&) »
/home/pierre/Projets/emsdk_portable/clang/fastcomp/src/tools/opt/opt.cpp:414: référence indéfinie vers « llvm::initializeExpandGetElementPtrPass(llvm::PassRegistry&) »
/home/pierre/Projets/emsdk_portable/clang/fastcomp/src/tools/opt/opt.cpp:415: référence indéfinie vers « llvm::initializeExpandIndirectBrPass(llvm::PassRegistry&) »
/home/pierre/Projets/emsdk_portable/clang/fastcomp/src/tools/opt/opt.cpp:416: référence indéfinie vers « llvm::initializeExpandShuffleVectorPass(llvm::PassRegistry&) »
/home/pierre/Projets/emsdk_portable/clang/fastcomp/src/tools/opt/opt.cpp:417: référence indéfinie vers « llvm::initializeExpandSmallArgumentsPass(llvm::PassRegistry&) »
/home/pierre/Projets/emsdk_portable/clang/fastcomp/src/tools/opt/opt.cpp:418: référence indéfinie vers « llvm::initializeExpandStructRegsPass(llvm::PassRegistry&) »
/home/pierre/Projets/emsdk_portable/clang/fastcomp/src/tools/opt/opt.cpp:419: référence indéfinie vers « llvm::initializeExpandTlsConstantExprPass(llvm::PassRegistry&) »
/home/pierre/Projets/emsdk_portable/clang/fastcomp/src/tools/opt/opt.cpp:420: référence indéfinie vers « llvm::initializeExpandTlsPass(llvm::PassRegistry&) »
/home/pierre/Projets/emsdk_portable/clang/fastcomp/src/tools/opt/opt.cpp:421: référence indéfinie vers « llvm::initializeExpandVarArgsPass(llvm::PassRegistry&) »
/home/pierre/Projets/emsdk_portable/clang/fastcomp/src/tools/opt/opt.cpp:422: référence indéfinie vers « llvm::initializeFixVectorLoadStoreAlignmentPass(llvm::PassRegistry&) »
/home/pierre/Projets/emsdk_portable/clang/fastcomp/src/tools/opt/opt.cpp:423: référence indéfinie vers « llvm::initializeFlattenGlobalsPass(llvm::PassRegistry&) »
/home/pierre/Projets/emsdk_portable/clang/fastcomp/src/tools/opt/opt.cpp:424: référence indéfinie vers « llvm::initializeGlobalCleanupPass(llvm::PassRegistry&) »
/home/pierre/Projets/emsdk_portable/clang/fastcomp/src/tools/opt/opt.cpp:425: référence indéfinie vers « llvm::initializeGlobalizeConstantVectorsPass(llvm::PassRegistry&) »
/home/pierre/Projets/emsdk_portable/clang/fastcomp/src/tools/opt/opt.cpp:426: référence indéfinie vers « llvm::initializeInsertDivideCheckPass(llvm::PassRegistry&) »
/home/pierre/Projets/emsdk_portable/clang/fastcomp/src/tools/opt/opt.cpp:429: référence indéfinie vers « llvm::initializePNaClSjLjEHPass(llvm::PassRegistry&) »
/home/pierre/Projets/emsdk_portable/clang/fastcomp/src/tools/opt/opt.cpp:430: référence indéfinie vers « llvm::initializePromoteI1OpsPass(llvm::PassRegistry&) »
/home/pierre/Projets/emsdk_portable/clang/fastcomp/src/tools/opt/opt.cpp:431: référence indéfinie vers « llvm::initializePromoteIntegersPass(llvm::PassRegistry&) »
/home/pierre/Projets/emsdk_portable/clang/fastcomp/src/tools/opt/opt.cpp:432: référence indéfinie vers « llvm::initializeRemoveAsmMemoryPass(llvm::PassRegistry&) »
/home/pierre/Projets/emsdk_portable/clang/fastcomp/src/tools/opt/opt.cpp:433: référence indéfinie vers « llvm::initializeRenameEntryPointPass(llvm::PassRegistry&) »
/home/pierre/Projets/emsdk_portable/clang/fastcomp/src/tools/opt/opt.cpp:434: référence indéfinie vers « llvm::initializeReplacePtrsWithIntsPass(llvm::PassRegistry&) »
/home/pierre/Projets/emsdk_portable/clang/fastcomp/src/tools/opt/opt.cpp:435: référence indéfinie vers « llvm::initializeResolveAliasesPass(llvm::PassRegistry&) »
/home/pierre/Projets/emsdk_portable/clang/fastcomp/src/tools/opt/opt.cpp:436: référence indéfinie vers « llvm::initializeResolvePNaClIntrinsicsPass(llvm::PassRegistry&) »
/home/pierre/Projets/emsdk_portable/clang/fastcomp/src/tools/opt/opt.cpp:437: référence indéfinie vers « llvm::initializeRewriteAtomicsPass(llvm::PassRegistry&) »
/home/pierre/Projets/emsdk_portable/clang/fastcomp/src/tools/opt/opt.cpp:438: référence indéfinie vers « llvm::initializeRewriteLLVMIntrinsicsPass(llvm::PassRegistry&) »
/home/pierre/Projets/emsdk_portable/clang/fastcomp/src/tools/opt/opt.cpp:439: référence indéfinie vers « llvm::initializeRewritePNaClLibraryCallsPass(llvm::PassRegistry&) »
/home/pierre/Projets/emsdk_portable/clang/fastcomp/src/tools/opt/opt.cpp:440: référence indéfinie vers « llvm::initializeSandboxIndirectCallsPass(llvm::PassRegistry&) »
/home/pierre/Projets/emsdk_portable/clang/fastcomp/src/tools/opt/opt.cpp:441: référence indéfinie vers « llvm::initializeSandboxMemoryAccessesPass(llvm::PassRegistry&) »
/home/pierre/Projets/emsdk_portable/clang/fastcomp/src/tools/opt/opt.cpp:442: référence indéfinie vers « llvm::initializeStripAttributesPass(llvm::PassRegistry&) »
/home/pierre/Projets/emsdk_portable/clang/fastcomp/src/tools/opt/opt.cpp:443: référence indéfinie vers « llvm::initializeStripMetadataPass(llvm::PassRegistry&) »
/home/pierre/Projets/emsdk_portable/clang/fastcomp/src/tools/opt/opt.cpp:445: référence indéfinie vers « llvm::initializeNoExitRuntimePass(llvm::PassRegistry&) »
/home/pierre/Projets/emsdk_portable/clang/fastcomp/src/tools/opt/opt.cpp:446: référence indéfinie vers « llvm::initializeStripModuleFlagsPass(llvm::PassRegistry&) »
/home/pierre/Projets/emsdk_portable/clang/fastcomp/src/tools/opt/opt.cpp:447: référence indéfinie vers « llvm::initializeStripTlsPass(llvm::PassRegistry&) »
/home/pierre/Projets/emsdk_portable/clang/fastcomp/src/tools/opt/opt.cpp:448: référence indéfinie vers « llvm::initializeSubstituteUndefsPass(llvm::PassRegistry&) »
/home/pierre/Projets/emsdk_portable/clang/fastcomp/src/tools/opt/opt.cpp:592: référence indéfinie vers « llvm::PNaClABISimplifyAddPreOptPasses(llvm::legacy::PassManagerBase&) »
/home/pierre/Projets/emsdk_portable/clang/fastcomp/src/tools/opt/opt.cpp:639: référence indéfinie vers « llvm::PNaClABISimplifyAddPostOptPasses(llvm::legacy::PassManagerBase&) »
/home/pierre/Projets/emsdk_portable/clang/fastcomp/src/tools/opt/opt.cpp:644: référence indéfinie vers « llvm::MinSFIPasses(llvm::legacy::PassManagerBase&) »
/home/pierre/Projets/emsdk_portable/clang/fastcomp/src/tools/opt/opt.cpp:733: référence indéfinie vers « llvm::MinSFIPasses(llvm::legacy::PassManagerBase&) »
/home/pierre/Projets/emsdk_portable/clang/fastcomp/src/tools/opt/opt.cpp:692: référence indéfinie vers « llvm::PNaClABISimplifyAddPreOptPasses(llvm::legacy::PassManagerBase&) »
/home/pierre/Projets/emsdk_portable/clang/fastcomp/src/tools/opt/opt.cpp:730: référence indéfinie vers « llvm::PNaClABISimplifyAddPostOptPasses(llvm::legacy::PassManagerBase&) »
collect2: error: ld returned 1 exit status

("référence indéfinie" means "undefined reference")

I was successfully building emscripten before (including after the 3.4 merge), so I suppose that this problem comes from the 3.5 merge?
I don't know the internals of LLVM well enough to investigate this myself.

[kripken]

Yes, likely a merge issue.

Can you run configure or cmake again, then try to build? Maybe a major upgrade of LLVM needs that. Also, which of those two are you using, and with what flags?

[kripken]

@tomaka - interesting, I think I see your error on the OS X bot. Are you on OS X too? Perhaps that's why I don't see it on linux.

[ibdknox]

I'm getting build errors on OSX as well. Here's what I'm getting on two different machines that both had previous versions working:

Linking CXX executable ../../bin/opt
Undefined symbols for architecture x86_64:
  "llvm::MinSFIPasses(llvm::legacy::PassManagerBase&)", referenced from:
      _main in opt.cpp.o
  "llvm::initializeStripTlsPass(llvm::PassRegistry&)", referenced from:
      _main in opt.cpp.o
  "llvm::initializeExpandTlsPass(llvm::PassRegistry&)", referenced from:
      _main in opt.cpp.o
  "llvm::initializeExpandByValPass(llvm::PassRegistry&)", referenced from:
      _main in opt.cpp.o
  "llvm::initializeExpandCtorsPass(llvm::PassRegistry&)", referenced from:
      _main in opt.cpp.o
  "llvm::initializePNaClSjLjEHPass(llvm::PassRegistry&)", referenced from:
      _main in opt.cpp.o
  "llvm::initializePromoteI1OpsPass(llvm::PassRegistry&)", referenced from:
      _main in opt.cpp.o
  "llvm::initializeExpandAllocasPass(llvm::PassRegistry&)", referenced from:
      _main in opt.cpp.o
  "llvm::initializeExpandVarArgsPass(llvm::PassRegistry&)", referenced from:
      _main in opt.cpp.o
  "llvm::initializeGlobalCleanupPass(llvm::PassRegistry&)", referenced from:
      _main in opt.cpp.o
  "llvm::initializeNoExitRuntimePass(llvm::PassRegistry&)", referenced from:
      _main in opt.cpp.o
  "llvm::initializeStripMetadataPass(llvm::PassRegistry&)", referenced from:
      _main in opt.cpp.o
  "llvm::initializeFlattenGlobalsPass(llvm::PassRegistry&)", referenced from:
      _main in opt.cpp.o
  "llvm::initializeResolveAliasesPass(llvm::PassRegistry&)", referenced from:
      _main in opt.cpp.o
  "llvm::initializeRewriteAtomicsPass(llvm::PassRegistry&)", referenced from:
      _main in opt.cpp.o
  "llvm::initializePromoteIntegersPass(llvm::PassRegistry&)", referenced from:
      _main in opt.cpp.o
  "llvm::initializeRemoveAsmMemoryPass(llvm::PassRegistry&)", referenced from:
      _main in opt.cpp.o
  "llvm::initializeStripAttributesPass(llvm::PassRegistry&)", referenced from:
      _main in opt.cpp.o
[100%] Built target llvm-lto
  "llvm::initializeExpandIndirectBrPass(llvm::PassRegistry&)", referenced from:
      _main in opt.cpp.o
  "llvm::initializeExpandStructRegsPass(llvm::PassRegistry&)", referenced from:
      _main in opt.cpp.o
  "llvm::initializeRenameEntryPointPass(llvm::PassRegistry&)", referenced from:
      _main in opt.cpp.o
  "llvm::initializeStripModuleFlagsPass(llvm::PassRegistry&)", referenced from:
      _main in opt.cpp.o
  "llvm::initializeSubstituteUndefsPass(llvm::PassRegistry&)", referenced from:
      _main in opt.cpp.o
  "llvm::PNaClABISimplifyAddPreOptPasses(llvm::legacy::PassManagerBase&)", referenced from:
      _main in opt.cpp.o
  "llvm::initializeInsertDivideCheckPass(llvm::PassRegistry&)", referenced from:
      _main in opt.cpp.o
  "llvm::PNaClABISimplifyAddPostOptPasses(llvm::legacy::PassManagerBase&)", referenced from:
      _main in opt.cpp.o
  "llvm::initializeExpandConstantExprPass(llvm::PassRegistry&)", referenced from:
      _main in opt.cpp.o
  "llvm::initializeAllocateDataSegmentPass(llvm::PassRegistry&)", referenced from:
      _main in opt.cpp.o
Linking CXX executable ../../bin/llvm-readobj
  "llvm::initializeBackendCanonicalizePass(llvm::PassRegistry&)", referenced from:
      _main in opt.cpp.o
  "llvm::initializeExpandGetElementPtrPass(llvm::PassRegistry&)", referenced from:
      _main in opt.cpp.o
  "llvm::initializeExpandShuffleVectorPass(llvm::PassRegistry&)", referenced from:
      _main in opt.cpp.o
  "llvm::initializeReplacePtrsWithIntsPass(llvm::PassRegistry&)", referenced from:
      _main in opt.cpp.o
  "llvm::initializeExpandSmallArgumentsPass(llvm::PassRegistry&)", referenced from:
      _main in opt.cpp.o
  "llvm::initializeSandboxIndirectCallsPass(llvm::PassRegistry&)", referenced from:
      _main in opt.cpp.o
  "llvm::initializeAddPNaClExternalDeclsPass(llvm::PassRegistry&)", referenced from:
      _main in opt.cpp.o
  "llvm::initializeExpandTlsConstantExprPass(llvm::PassRegistry&)", referenced from:
      _main in opt.cpp.o
  "llvm::initializeRewriteLLVMIntrinsicsPass(llvm::PassRegistry&)", referenced from:
      _main in opt.cpp.o
  "llvm::initializeSandboxMemoryAccessesPass(llvm::PassRegistry&)", referenced from:
      _main in opt.cpp.o
  "llvm::initializeResolvePNaClIntrinsicsPass(llvm::PassRegistry&)", referenced from:
      _main in opt.cpp.o
  "llvm::initializeExpandArithWithOverflowPass(llvm::PassRegistry&)", referenced from:
      _main in opt.cpp.o
  "llvm::initializeGlobalizeConstantVectorsPass(llvm::PassRegistry&)", referenced from:
      _main in opt.cpp.o
  "llvm::initializeRewritePNaClLibraryCallsPass(llvm::PassRegistry&)", referenced from:
      _main in opt.cpp.o
  "llvm::initializeCanonicalizeMemIntrinsicsPass(llvm::PassRegistry&)", referenced from:
      _main in opt.cpp.o
  "llvm::initializeFixVectorLoadStoreAlignmentPass(llvm::PassRegistry&)", referenced from:
      _main in opt.cpp.o
  "llvm::initializeConstantInsertExtractElementIndexPass(llvm::PassRegistry&)", referenced from:
      _main in opt.cpp.o
[100%] Built target llvm-readobj
ld: symbol(s) not found for architecture x86_64
clang: error: linker command failed with exit code 1 (use -v to see invocation)
make[2]: *** [bin/opt] Error 1
make[1]: *** [tools/opt/CMakeFiles/opt.dir/all] Error 2
make[1]: *** Waiting for unfinished jobs....
Linking CXX executable ../../../../bin/clang
[100%] Built target clang
make: *** [all] Error 2
Build failed due to exception!
Working directory: /Users/chris/Downloads/emsdk_portable/clang/fastcomp/build_incoming_64
Command '['make', '-j7']' returned non-zero exit status 2
Installation failed!

[tomaka]

Are you on OS X too?

No, I'm on Ubuntu.

I cleaned the build directory and executed cmake -DLLVM_TARGETS_TO_BUILD="X86;JSBackend" manually, and still get the error.

[kripken]

Ok, maybe the difference is CMake vs configure - I use configure locally. Will test CMake tomorrow.

@ibdknox , is that error with CMake or configure?

[ibdknox]

that's with CMake

[floooh]

I'm getting the exact same errors as @ibdknox on OSX 10.10 when linking 'opt' with a fresh pull from incoming in the emsdk, I'll try on my Linux VM next. I'll see if I can investigate a bit more in a few hours.

Linking CXX executable ../../bin/opt
Undefined symbols for architecture x86_64:
"llvm::MinSFIPasses(llvm::legacy::PassManagerBase&)", referenced from:
_main in opt.cpp.o
"llvm::initializeStripTlsPass(llvm::PassRegistry&)", referenced from:
_main in opt.cpp.o
"llvm::initializeExpandTlsPass(llvm::PassRegistry&)", referenced from:
_main in opt.cpp.o
"llvm::initializeExpandByValPass(llvm::PassRegistry&)", referenced from:
_main in opt.cpp.o
"llvm::initializeExpandCtorsPass(llvm::PassRegistry&)", referenced from:
_main in opt.cpp.o
"llvm::initializePNaClSjLjEHPass(llvm::PassRegistry&)", referenced from:
_main in opt.cpp.o
"llvm::initializePromoteI1OpsPass(llvm::PassRegistry&)", referenced from:
_main in opt.cpp.o
"llvm::initializeExpandAllocasPass(llvm::PassRegistry&)", referenced from:
_main in opt.cpp.o
"llvm::initializeExpandVarArgsPass(llvm::PassRegistry&)", referenced from:
_main in opt.cpp.o
"llvm::initializeGlobalCleanupPass(llvm::PassRegistry&)", referenced from:
_main in opt.cpp.o
"llvm::initializeNoExitRuntimePass(llvm::PassRegistry&)", referenced from:
_main in opt.cpp.o
"llvm::initializeStripMetadataPass(llvm::PassRegistry&)", referenced from:
_main in opt.cpp.o
"llvm::initializeFlattenGlobalsPass(llvm::PassRegistry&)", referenced from:
_main in opt.cpp.o
"llvm::initializeResolveAliasesPass(llvm::PassRegistry&)", referenced from:
_main in opt.cpp.o
"llvm::initializeRewriteAtomicsPass(llvm::PassRegistry&)", referenced from:
_main in opt.cpp.o
"llvm::initializePromoteIntegersPass(llvm::PassRegistry&)", referenced from:
_main in opt.cpp.o
"llvm::initializeRemoveAsmMemoryPass(llvm::PassRegistry&)", referenced from:
_main in opt.cpp.o
"llvm::initializeStripAttributesPass(llvm::PassRegistry&)", referenced from:
_main in opt.cpp.o
"llvm::initializeExpandIndirectBrPass(llvm::PassRegistry&)", referenced from:
_main in opt.cpp.o
"llvm::initializeExpandStructRegsPass(llvm::PassRegistry&)", referenced from:
_main in opt.cpp.o
"llvm::initializeRenameEntryPointPass(llvm::PassRegistry&)", referenced from:
_main in opt.cpp.o
"llvm::initializeStripModuleFlagsPass(llvm::PassRegistry&)", referenced from:
_main in opt.cpp.o
"llvm::initializeSubstituteUndefsPass(llvm::PassRegistry&)", referenced from:
_main in opt.cpp.o
"llvm::PNaClABISimplifyAddPreOptPasses(llvm::legacy::PassManagerBase&)", referenced from:
_main in opt.cpp.o
"llvm::initializeInsertDivideCheckPass(llvm::PassRegistry&)", referenced from:
_main in opt.cpp.o
"llvm::PNaClABISimplifyAddPostOptPasses(llvm::legacy::PassManagerBase&)", referenced from:
_main in opt.cpp.o
"llvm::initializeExpandConstantExprPass(llvm::PassRegistry&)", referenced from:
_main in opt.cpp.o
"llvm::initializeAllocateDataSegmentPass(llvm::PassRegistry&)", referenced from:
_main in opt.cpp.o
"llvm::initializeBackendCanonicalizePass(llvm::PassRegistry&)", referenced from:
_main in opt.cpp.o
"llvm::initializeExpandGetElementPtrPass(llvm::PassRegistry&)", referenced from:
_main in opt.cpp.o
"llvm::initializeExpandShuffleVectorPass(llvm::PassRegistry&)", referenced from:
_main in opt.cpp.o
"llvm::initializeReplacePtrsWithIntsPass(llvm::PassRegistry&)", referenced from:
_main in opt.cpp.o
"llvm::initializeExpandSmallArgumentsPass(llvm::PassRegistry&)", referenced from:
_main in opt.cpp.o
"llvm::initializeSandboxIndirectCallsPass(llvm::PassRegistry&)", referenced from:
_main in opt.cpp.o
"llvm::initializeAddPNaClExternalDeclsPass(llvm::PassRegistry&)", referenced from:
_main in opt.cpp.o
"llvm::initializeExpandTlsConstantExprPass(llvm::PassRegistry&)", referenced from:
_main in opt.cpp.o
"llvm::initializeRewriteLLVMIntrinsicsPass(llvm::PassRegistry&)", referenced from:
_main in opt.cpp.o
"llvm::initializeSandboxMemoryAccessesPass(llvm::PassRegistry&)", referenced from:
_main in opt.cpp.o
"llvm::initializeResolvePNaClIntrinsicsPass(llvm::PassRegistry&)", referenced from:
_main in opt.cpp.o
"llvm::initializeExpandArithWithOverflowPass(llvm::PassRegistry&)", referenced from:
_main in opt.cpp.o
"llvm::initializeGlobalizeConstantVectorsPass(llvm::PassRegistry&)", referenced from:
_main in opt.cpp.o
"llvm::initializeRewritePNaClLibraryCallsPass(llvm::PassRegistry&)", referenced from:
_main in opt.cpp.o
"llvm::initializeCanonicalizeMemIntrinsicsPass(llvm::PassRegistry&)", referenced from:
_main in opt.cpp.o
"llvm::initializeFixVectorLoadStoreAlignmentPass(llvm::PassRegistry&)", referenced from:
_main in opt.cpp.o
"llvm::initializeConstantInsertExtractElementIndexPass(llvm::PassRegistry&)", referenced from:
_main in opt.cpp.o
ld: symbol(s) not found for architecture x86_64
clang: error: linker command failed with exit code 1 (use -v to see invocation)
make[2]: *** [bin/opt] Error 1
make[1]: *** [tools/opt/CMakeFiles/opt.dir/all] Error 2
make: *** [all] Error 2

[floooh]

I'm also getting those errors on my Linux Mint VM (compiled with "Ubuntu clang version 3.5-lubuntu1".

Just a shot in the dark, but the git log says that the merge had conflicts (among others in tools/CMakeLists.txt and tools/opt/opt.cpp), maybe something went wrong when resolving the conflicts? It looks like the the opt-tool is missing one or more link libs?

[floooh]

PS: also CMake compile here, that's what the emsdk uses automatically.

[kripken]

Thanks everyone, I did fresh builds locally with both configure and cmake, and I can confirm the cmake error now.

[kripken]

Ok, I pushed a commit that fixes this locally.

Oddly it doesn't look like a merge issue - that broken CMakeLists.txt file wasn't in the conflicts. @jfbastien , do you test CMake builds in PNaCl, or just configure?

[tomaka]

The fix works for me too.

[ibdknox]

@kripken fixed here as well.

[kripken]

Great, thanks for the confirmation @tomaka and @ibdknox .

Hmm, meanwhile I see on the bots that the windows one fails on

CMake Error at cmake/modules/HandleLLVMOptions.cmake:44 (message):
  Host Visual Studio must be at least 2012 (MSVC 17.0)

@juj, when you get back from vacation, hopefully that's not hard to upgrade?

[kripken]

Bots (OS X and linux) passed linking of opt successfully.

[jfbastien]

@kripken we test with configure on the bots because some of the tests can't be run under cmake, but some of us build with cmake locally so I expect it to work (it does locally).

Anything we should patch?

[floooh]

Compiling here on OSX too, but I got a nasty crash when trying to compile my engine for the first time, it was one of the source files, not the librarian or linker (unfortunately not reproducible on the 2nd and 3rd try). I had jekyll and a few VMs running in the background so may be memory was low, still it's the first time clang ever crashed on me.

Here's the crash log with call stack, I'll have an eye on it whether it happens again, guess that's something for the clang team to look at?

https://gist.github.com/floooh/b68a160b96775b82092a

[jfbastien]

@floooh do you have a repro? This does seem like a clang bug.

[floooh]

@jfbastien no repo so far, I compiled the whole code 4 times so far and the crash didn't happen again. if anyone wants to give it a try (on OSX or Linux, with python 2.7 in the path):

> git clone git@github.com:floooh/oryol.git
> cd ~/oryol
> ./oryol setup emsdk (this takes a while to download and compile)
> ./oryol build emscripten-make-release

I'll see if I can setup a stress test and recompile in an infinite loop over night or so...

(Update: the ./oryol setup was wrong... and the ./oryol build too)

[kripken]

@jfbastien I had to do this: bbe4691 looks like on master of pnacl-llvm you would only need to add 2 of those,

+ NaClTransforms
+ NaClAnalysis

[floooh]

PS: the PNaCl pepper_canary version also compiles so far (first "./oryol setup nacl", then "./oryol build pnacl-make-release"), @jfbastien, do you know when the nacl SDK switched to clang 3.5? I've been compiling both the emscripten (with clang 3.4) and pnacl version very frequently in the past weeks, and didn't have problems on this code base.

[jfbastien]

@kripken looks like I fixed that one here: https://codereview.chromium.org/749363002/diff/1/tools/opt/CMakeLists.txt
Though I didn't need more than MinSFI.

@floooh 3.5 will be in the SDK as of version 41. It's currently in pepper_canary, and Chrome should be branching tomorrow, hitting stable in early March.

[kripken]

OS X bot is now green, and Ubuntu looks fine so far. I think we are good, except for the windows bot which fails as mentioned before.

[waywardmonkeys]

This can be closed now?