Muelu: Interoperability with Epetra and Tpetra

[bangerth]

Bug Report

@trilinos/muelu

Description

As discussed at the TUG meeting yesterday, here is the issue the deal.II project is facing with Muelu (to the best of my understanding -- @Rombur is the real expert):

• deal.II currently wraps Trilinos' Epetra libraries
• deal.II wants to use Muelu with its Epetra interface
• Muelu disables the Epetra interface if Trilinos is compiled with Tpetra and if the Tpetra GO type is not int; this was first pointed out as a problem in Trilinos, Albany: getting Albany and Trilinos to compile again using spack spack/spack#14215 and specifically this comment: Trilinos, Albany: getting Albany and Trilinos to compile again using spack spack/spack#14215 (comment)
• As a consequence, deal.II currently disables its Muelu wrapers if Muelu does not support Epetra -- see this patch: https://github.com/dealii/dealii/pull/9237/files

[cgcgcg]

@trilinos/muelu

[cgcgcg]

Ok, I see. Epetra and Tpetra can be used in the same build, but that requires the GO types to match, i.e. GO=int. That should also work in deal.ii. However, using 2 different global ordinal types at the same time is not allowed. I think that's reasonable. Does deal.ii have a required dependency on Epetra? Cause you should be able to disable Epetra and still use MueLu with Tpetra.

[bangerth]

@cgcgcg We currently still rely on Epetra. The long-term plan is to move to Tpetra, but we're not there yet.

Since Trilinos relies on its own decision in choosing what GO is, deal.II has no control over what GO is or should be. The original issue is rooted in xSDK choosing a type for GO that resulted in Muelu disabling its Epetra interfaces. As a consequence, if you build Trilinos via xSDK, you end up with a Muelu we can't use.

Where is the place inside Muelu where you decide which GO you want to use? Could Muelu's GO be chosen as (i) the one from Epetra if Tpetra is not enabled, (ii) the larger of Epetra's and Tpetra's GO type if both of these are enabled? I imagine that there must be a place in Muelu that "inherits" the GO type from one or the other place. Maybe one could tweak that "inheritance".

[cgcgcg]

We require Tpetra to be enabled for MueLu, and that MueLu uses Tpetra's GO.

Currently I don't think that within the constraints of xSDK there is a way to use MueLu with Tpetra and Epetra. xSDK wants 64 bit GO. Epetra does not provide that (there are caveats to this statement), so Epetra should be disabled in xSDK. (But for practical reasons it obviously isn't.)

I was thinking more about ways to simplify your transition. Outside of xSDK you could build deal.ii with GO=int and Epetra+Tpetra and perform bake-offs that are fair.

[bangerth]

Actually, what we typically do in large computations is use Epetra through its 64-bit interfaces. Do you have an idea how many places Muelu has that call Epetra functions? Depending on whether Tpetra uses 32- or 64-bit integers, one could call either the 32- or 64-bit Epetra functions.

[cgcgcg]

Hm, I haven't tried the Xpetra/MueLu support for 64 bit Epetra. I'll give that a shot an see what actually works.

[bangerth]

It actually works reasonably well for us. Depending on whether deal.II uses 32- or 64-bit GO types, we either call the 32- or 64-bit Epetra functions. Support in Epetra for that is complete and works well.

[GrahamBenHarper]

We require Tpetra to be enabled for MueLu, and that MueLu uses Tpetra's GO.

Is this actually on the MueLu side, or is this more related to how Xpetra handles things?

[csiefer2]

@bangerth I tried Epetra-64 support in Xpetra/MueLu a few years ago and it took me a while to even get the code to compile. The tests were failing all over the place.

You can look at this old half-done branch to see what I had to change to get MueLu to compile

https://github.com/trilinos/Trilinos/tree/xpetra-epetra64

@GrahamBenHarper Neither Xpetra nor MueLu correctly supports Epetra64 everywhere. Or at least it didn't in 2019.

[cgcgcg]

As far as I can tell, the Xpetra support for Epetra 64 doesn't build. So GO=int is the only way to have both Tpetra and Epetra enabled in MueLu.

[bangerth]

I had not realized that MueLu supports both Epetra and Tpetra via Xpetra, and that Xpetra only supports 32-bit Epetra. That's a shame, but I understand that that is not worth fixing. Thanks for talking this through with me.

I suppose there isn't much that can be done about the issue then. Should we close the issue?

[cgcgcg]

Sorry that we couldn't provide a nicer solution. Once deal.ii can make its Epetra dependency optional it should be easier to set this up.

[GrahamBenHarper]

I have a recent build of deal.II on our clusters. If I could find time, I would absolutely love to help... however, time is always the problem.

[mayrmt]

dealii/dealii#12774