Update tensorflow and llvm submodules.

[stellaraccident]

• tensorflow -> head
• llvm -> head (not tensorflow version because incompatible)
• llvm bazel build files: imported from tensorflow and manually patched
• adapted to new dialect registration scheme by adding calls to the registration method
• all tests pass on my linux vm with bazel (sans Vulkan due to no graphics card)
• will give cmake a shot in a minute but this has been a lot already and need to sign off

[marbre]

@stellaraccident I can take over and give CMake a try based on what you already did.

[stellaraccident]

Thanks - I was just about to email you. I really need to sign off now but was poking CMake a bit... The primary thing that changed upstream is how they are doing dialect registration. There is now an explicit mlir::registerAllDialects() in any binary that needs them (from mlir/InitAllDialects.h). It looks like this is an intermediate state, still doing static registration in some way that I haven't untangled how to depend on properly from the CMake side. There is an MLIRAllDialects target which is sufficient to get enough headers to compile, but then linking fails. One issue is that this new mechanism pulls in all MLIR dialects, whether we need them or not, and it appears that the ones we weren't using are the ones that then fail on link (FxpMathOps is one that jumps out and that I clue in on). We will probably eventually want our own, properly scoped list, but for now, just pulling them all in from upstream is better than being broken (which the most recent integrate was causing). I think we just need every binary to depend on the full set of dialects -- maybe a common list somewhere so we can at least not repeat ourselves? I'd really love to get the cmake version working again and stable so I can use it as a main mechanism of developing... believe it or not, my CMake builds are ~3-5x faster than corresponding Bazel builds and build the whole project in about 2 minutes (CMake maxes out all 64 cores in my machine, whereas Bazel rarely utilizes more than ~20). I feel like Bazel claimed to have required all of the complex build machinery for parallelism... and does not seem to deliver on local builds. I bet it is rare that big projects have an apples to apples comparison of the same build on both systems.

…

On Thu, Feb 13, 2020 at 10:32 PM Marius Brehler ***@***.***> wrote: @stellaraccident <https://github.com/stellaraccident> I can take over and give CMake a try based on what you already did. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#762>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AADYVAGSOGTO6DLOKAW3UIDRCY3IBANCNFSM4KVBRTVA> .

[stellaraccident]

+Lei Zhang <antiagainst@google.com> who was oncall this week for MLIR integration issues and may have more context on the static registration stuff. On Thu, Feb 13, 2020 at 10:50 PM Stella Laurenzo <laurenzo@google.com> wrote:

…

Thanks - I was just about to email you. I really need to sign off now but was poking CMake a bit... The primary thing that changed upstream is how they are doing dialect registration. There is now an explicit mlir::registerAllDialects() in any binary that needs them (from mlir/InitAllDialects.h). It looks like this is an intermediate state, still doing static registration in some way that I haven't untangled how to depend on properly from the CMake side. There is an MLIRAllDialects target which is sufficient to get enough headers to compile, but then linking fails. One issue is that this new mechanism pulls in all MLIR dialects, whether we need them or not, and it appears that the ones we weren't using are the ones that then fail on link (FxpMathOps is one that jumps out and that I clue in on). We will probably eventually want our own, properly scoped list, but for now, just pulling them all in from upstream is better than being broken (which the most recent integrate was causing). I think we just need every binary to depend on the full set of dialects -- maybe a common list somewhere so we can at least not repeat ourselves? I'd really love to get the cmake version working again and stable so I can use it as a main mechanism of developing... believe it or not, my CMake builds are ~3-5x faster than corresponding Bazel builds and build the whole project in about 2 minutes (CMake maxes out all 64 cores in my machine, whereas Bazel rarely utilizes more than ~20). I feel like Bazel claimed to have required all of the complex build machinery for parallelism... and does not seem to deliver on local builds. I bet it is rare that big projects have an apples to apples comparison of the same build on both systems. On Thu, Feb 13, 2020 at 10:32 PM Marius Brehler ***@***.***> wrote: > @stellaraccident <https://github.com/stellaraccident> I can take over > and give CMake a try based on what you already did. > > — > You are receiving this because you were mentioned. > Reply to this email directly, view it on GitHub > <#762>, > or unsubscribe > <https://github.com/notifications/unsubscribe-auth/AADYVAGSOGTO6DLOKAW3UIDRCY3IBANCNFSM4KVBRTVA> > . >

[marbre]

Intermediate status:

• Got a build working with all test passing, expect the tree also failing before.
  168 - iree_tools_vm_util_test (Failed)
  172 - iree_samples_custom_modules_custom_modules_test (Failed)
  173 - iree_samples_simple_embedding_simple_embedding_test (Failed)
• Further refactoring required with regard to ALWAYSLINK. The current configuration doubles the sizes of the executables (e.g. iree-opt).

[marbre]

I opened #763 as a follow up and to discuses my modifications. My branch might need rebasing (this and an earlier PR of mine are merged), but if you like feel free to take back over again.

[marbre]

I'd really love to get the cmake version working again and stable so I can use it as a main mechanism of developing... believe it or not, my CMake builds are ~3-5x faster than corresponding Bazel builds and build the whole project in about 2 minutes (CMake maxes out all 64 cores in my machine, whereas Bazel rarely utilizes more than ~20).

@OliverUrbann We need a more powerfull build machine as well. My 16GB RAM are not sufficient to run six linker processes in parallel. It actually resulted in eating up nearly my whole swap and an unusable system. 64 cores and appropriate RAM sounds adequate ;)

[antiagainst]

Yeah right now we can rely on the gigantic initAllDialects and initAllPasses. In the future we will need distilled ones with explicit register*() in binaries like what is happening with

https://github.com/llvm/llvm-project/blob/b6a9fe209992789be3ed95664d25196361cfad34/mlir/tools/mlir-opt/mlir-opt.cpp#L81-L104

[GMNGeoffrey]

This needs some surgery upstream where we have stricter checks on depending on the rule that exports the headers you use.

[marbre]

Yeah right now we can rely on the gigantic initAllDialects and initAllPasses. In the future we will need distilled ones with explicit register*() in binaries like what is happening with

https://github.com/llvm/llvm-project/blob/b6a9fe209992789be3ed95664d25196361cfad34/mlir/tools/mlir-opt/mlir-opt.cpp#L81-L104

Would you propose to fork mlir-opt.cpp and maintain an IREE specific main entry function, like IREE's translate_main.cc replacing mlir-translate.cpp?

[antiagainst]

Sorry for the late reply. Yeah I think that makes sense. mlir-opt is for testing purpose in MLIR core so it's pulling in all core dialects/passes for that. TF has its own tf-opt. Right now our iree-opt does not control the main function but I think we should.

[marbre]

Okay, will go for it after I returned to the office from C4ML. I think I will send a PR latest end of the week.