-
Notifications
You must be signed in to change notification settings - Fork 2.2k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
trilinos: add adelus, aprepro and teuchos variants #28935
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Hi @prwolfe , thanks for your contribution. I've made an effort over the last year or so to trim down the number of variants in Trilinos, because each new one requires a full set of conflicts
to be duplicated from the Trilinos build logic (TriBITS) in order to gauge which combinations of variants have a hope of working. For example, I removed teuchos because it's a prerequisite to almost everything in trilinos: under what circumstances do you turn it off? I have the same question about triutils
: what is the benefit of being able to manually enable or disable it? Without the variant, the trilinos build logic will toggle it on or off as needed.
The adelus
variant needs to have conflicts
to inform spack what dependencies it has.
Finally, to reduce the combinatorial horror that is the trilinos package option system, there's been a move toward using the external seacas
package rather than the one annexed by Trilinos through the Tribits build system. Does it makes sense for your application to use that package instead of the Trilinos-vendored version?
@sethrj - Trimming the number of variants in Trilinos may be a lost cause in the long run ;->. I agree that the current package is unwieldy at best and the conflicts section is the only current defense. I have been talking with @jwillenbring about adding more generality to this file (which makes reduction difficult) and I see the need for serious unit testing for this package to enable that. Adelus does not appear to drive or conflict with any other packages. I added Teuchos because Gemma has it explicitly turned on, but I cannot say I have seen a build with it off. Can you think of one @jwillenbring? As for Seacas, I do prefer using it from Trilinos in order to reduce the number of repos needed for most builds. I am willing to change that if needed, but the Trilinos team will need to decide to not support a distributed package (not something I would recommend.) A lot of what's causing confusion to my mind is that TriBits and spack are doing a lot of the same (and potentially conflicting) configuration work. I would recommend consolidating that where we can over time. I will remove Teuchos and re-submit if it works. Thanks! Paul |
Since Gemma requires it on and spack always turns it on, I think we should be OK :)
Indeed, these two issues are intertwined. TriBITS is a framework for a "megarepo"-type build system where you always build everything at the same time. That works well if all your code is "owned" by a single entity (e.g. if you're Microsoft, Google, or 2005-era Sandia) but becomes problematic in an open source ecosystem, which is where the scientific and HPC communities have been trending for the last 10 years. Spack is a package manager that tries to handle software dependencies from a variety of sources; and TriBITS has the philosophy that everything should be built inside its single build system. The two philosophies are largely incompatible... and the TriBITS philosophy also makes agile development more difficult. If you incorporated Gemma into TriBITS you'd have to rebuild your trilinos software stack every time you change a CMake configure variable. My understanding is that SEACAS is trying to separate from Trilinos just like Kokkos is now providing a separate package. @gsjaardema is that accurate? If you are using Spack to set up dependencies for Gemma then changing from trilinos to trilinos&seacas won't be much of a stretch. But I understand wanting to keep dependencies simple... |
@sethrj |
Since Kokkos is moving toward external, and @keitat can confirm that Trilinos is trying to be more agile about external dependencies (and being a better software ecosystem player), I think external SEACAS is the direction we want to head. Looking at the Trilinos package dependency chain, it looks like there's perhaps one or two connections into SEACAS from the main trilinos, and I think no dependencies of SEACAS on trilinos? I strongly suggest we move toward simplifying the Trilinos spack package and the larger software ecosystem by removing SEACAS from Trilinos in the long term. |
Seems like a decision for the trilinos framework team (@jwillenbring) and @gsjaardema to me. |
In the meantime @prwolfe if you remove the teuchos variant and add the required conflicts for the new packages (denoting tribits package requirements) I'll approve even with the new SeaCas package. |
Perhaps @bartlettroscoe has some history on the incorporation of SEACAS into Trilinos? |
Will do – likely early tomorrow morning.
From: "Seth R. Johnson" ***@***.***>
Reply-To: spack/spack ***@***.***>
Date: Wednesday, February 16, 2022 at 2:34 PM
To: spack/spack ***@***.***>
Cc: "Wolfenbarger, Paul R" ***@***.***>, Mention ***@***.***>
Subject: [EXTERNAL] Re: [spack/spack] trilinos: add adelus, aprepro and teuchos variants (PR #28935)
In the meantime @prwolfe<https://github.com/prwolfe> if you remove the teuchos variant and add the required conflicts for the new packages (denoting tribits package requirements) I'll approve even with the new SeaCas package.
—
Reply to this email directly, view it on GitHub<#28935 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AB2OSCJD7VFKJPICU3XQWCLU3QJ5VANCNFSM5OMSUCVA>.
Triage notifications on the go with GitHub Mobile for iOS<https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android<https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.
You are receiving this because you were mentioned.Message ID: ***@***.***>
|
That question is above my pay-grade (and I had no part in that work).
The plan this FY is to allow SEACAS to be built and installed as an independent CMake project (i.e. as a Spack
That is currently true but we are working to allow TriBITS to build packages in any arbitrary collection of smaller CMake projects that you want. That way, you can build (and install) all at once or in smaller subsets (whatever makes the most sense for the use case you are working). That is the best you can get as it leads to very fast iterative development cycles using just the software you are editing. The basics of that will be complete with TriBITSPub/TriBITS#367. The ultimate goal of this work is to allow any arbitrary TriBITS package to be built and installed as its own CMake project (by finding its pre-built/pre-installed upstream dependencies with |
Yes, the Spack Trilinos/packages.py file should be updated to use TriBITS logic for addressing inner Trilinos dependencies and exposing variants for enables and disables of individual Trilinos packages. All you need is a usable cmake and python to do that. You don't need to actually configure any CMake packages. (See Reduced Package Dependency Processing and tribits_adjust_package_enables().)
I agree with that. You really can't build anything useful in Trilinos without building Teuchos. But an automated system that used TriBITS logic could run the enables/disables and provide feedback about conflicts before having to actually configure Trilinos. But any layer you put around the Trilinos CMake build is going to be inadequate for some customers because it will not expose all of the CMake cache vars (i.e. knobs) that are available. I was successful with ATDM (which included GEMMA) by exposing fairly small number of extra "knobs" as documented in: |
if '+aprepro' in spec: | ||
options.extend([ | ||
define_trilinos_enable('SEACAS', True), | ||
define_trilinos_enable('SEACASAprepro', True), | ||
]) | ||
else: | ||
options.extend([ | ||
define_trilinos_enable('SEACASAprepro', False), | ||
]) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This type of logic should not exist in this package.py file. This the type of logic that TriBITS handles automatically.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The SEACASAprepro
options need to be there so that the variants are consistent with what's actually installed... the define_trilinos_enable('SEACAS', True)
could probably be dropped though. I think this one (and several of the similar ones) can be written:
if '+aprepro' in spec: | |
options.extend([ | |
define_trilinos_enable('SEACAS', True), | |
define_trilinos_enable('SEACASAprepro', True), | |
]) | |
else: | |
options.extend([ | |
define_trilinos_enable('SEACASAprepro', False), | |
]) | |
options.append(define_trilinos_enable('SEACAS', 'aprepro')) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The issue with variants is that they have to be determined at concretization time in Spack to ensure they're compatible with all the other software in a spec (or set of specs in an environment). We can't just defer to tribits logic. The closest we could possibly do would be to parse the tribits dependencies for all the different versions of trilinos we want to support, then convert those into the appropriate variants and conflicts...
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The issue with variants is that they have to be determined at concretization time in Spack to ensure they're compatible with all the other software in a spec (or set of specs in an environment).
I see, that makes sense.
The closest we could possibly do would be to parse the tribits dependencies for all the different versions of trilinos we want to support, then convert those into the appropriate variants and conflicts...
Yes, that is what should be done. Given the generated file XML file TrilinosDependencies.xml (which could just be returned as a string to STDOUT as called inside Trilinos/package.py), Python code in Trilinos/package.py could process this and automatically generate the package enable/disable variants and provide the conflicts list (which will take some graph algorithms to give Spack what it wants and may need to be implemented carefully to be performant).
The only situations I can think of where someone might want to disable Teuchos are ones where the package the person wants to use is likely a good candidate to be split off into a separate Spack package, like Kokkos and SEACAS. |
Remove variant for secas executable "aprepro". It will always build now Remove variant for "teuchos". This is need for virtually all builds Confirm that adelus and triutils are without conflicts in either direction
@@ -126,6 +127,7 @@ class Trilinos(CMakePackage, CudaPackage, ROCmPackage): | |||
variant('tempus', default=False, description='Compile with Tempus') | |||
variant('tpetra', default=True, description='Compile with Tpetra') | |||
variant('trilinoscouplings', default=False, description='Compile with TrilinosCouplings') | |||
variant('triutils', default=False, description='Compile with Triutils') |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
As my original review requested, these new variants need conflicts
statements to attempt to mirror the TriBITS package logic in Spack (since all valid combinations of variants should produce a working binary, so that different libraries and apps can guarantee that they'll work together). Look at amesos2
as an example.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I still don't understand what the extra triutils
variant adds to the package: does your code need it downstream? If so let's just turn it on by default if it doesn't require any upstream dependencies.
@@ -539,6 +538,11 @@ def define_enable(suffix, value=None): | |||
define_from_variant('Amesos2_ENABLE_LAPACK', 'amesos2'), | |||
]) | |||
|
|||
if '+aztec' in spec and '+triutils' in spec: |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The correct way to handle this is to add a conflict saying "you cannot have aztec enabled and triutils disabled":
conflicts('+aztec', when='~triutils')
Otherwise users who want aztec will find it silently missing!
Adding a conflict will turn off all of aztec rather than just AztecOO. Seems ok to me, but may not match expectations. |
Am I missing something? There is no "Aztec" trilinos package other than AztecOO; the spack option is named This question may be moot because looking at the actual tribits dependencies, However, with when('~kokkos'):
conflicts('+adelos')
... |
https://github.com/trilinos/Trilinos/blob/master/packages/aztecoo/cmake/Dependencies.cmake#L3 Question/comment to @sethrj : It seems like we don't have Is Perhaps we should have this conversation elsewhere, but I support the idea of not trying to replicate Tribits dependencies within Trilinos in Spack, and only using Spack to handle conflicts with missing TPLs. Then, if the build fails because of a conflicting combination of packages in Trilinos, then let the user read the error message and turn on/off the appropriate package combinations. |
Correct. However, there's a (not very stable or well used?) feature to build a spack package's test at install time, and the package has an attribute
No: besides the proliferation of options,
The ideal spack package will never concretize to a configuration that won't build: we always want to prevent an non-installable trilinos. Imagine the use case of building a bunch of dependencies in an environment only to find out your build fails: not ideal. The only problem with this philosophy is clingo's variant conflict error messages, which at present frankly are garbage. |
I think all the comments have been addressed here. Is there a mod that we need to make on this? |
conflicts('~thyra', when='+stratimikos') | ||
conflicts('+aztec', when='~fortran') | ||
conflicts('+aztec', when='~triutils') |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Aztec doesn't require triutils. Only its tests do, and we aren't building them.
@@ -126,6 +127,7 @@ class Trilinos(CMakePackage, CudaPackage, ROCmPackage): | |||
variant('tempus', default=False, description='Compile with Tempus') | |||
variant('tpetra', default=True, description='Compile with Tpetra') | |||
variant('trilinoscouplings', default=False, description='Compile with TrilinosCouplings') | |||
variant('triutils', default=False, description='Compile with Triutils') |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I still don't understand what the extra triutils
variant adds to the package: does your code need it downstream? If so let's just turn it on by default if it doesn't require any upstream dependencies.
@sethrj - triutils is in fact required for this code. I can turn it on by default and most code will never notice as it very small. Are you saying you done want to add a variant? Something like
instead? |
Yep, exactly! Thanks! |
Hi all, rereading the comments here and trying to get a sense for what was said: So @prwolfe, you would like to add @sethrj, you don't want a Since the dependencies for
where Triutils are needed for tests but not for building normally, can we add the variant |
I am actually ok with always building triutils. It's 32 source and 8 header files, so adds little to the build time. In the long run we need more coordination with whatever we have in the Trilinos system, but that is some time away and I am good with this for the foreseeable future. I set it all to running over the weekend and will try to submit the changes today, but I am involved in the ASC S3C conference and that is taking most of my time. Sorry about that. Paul |
Gemma uses a different stack than most FE codes. All new variants are off by default and this shoudl not affect current users.
Remove variant for secas executable "aprepro". It will always build now Remove variant for "teuchos". This is need for virtually all builds Confirm that adelus and triutils are without conflicts in either direction
There are no dependencies between adelus and any other package, and this is the the only required dependency I could find for triutils other than a test requirement (and this package does not enable tests.)
Basicaklly uses a conflict statement to remove aztec when triutils is not present.
Per the github discussion this will simply be on by default.
d32e973
to
810cbf8
Compare
All, My apologies but I had to rebase onto develop due to a problem I caused in git (basically the old merge of develop into the branch was incorrectly done.) That caused my hand testing to fail during cling bootstrapping which of course is not acceptable. I believe this matches everyone's requests. Paul |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Quite satisfactory, thanks for working with us on this.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Quite satisfactory! Thanks for working with us on this.
Gemma uses a different stack than most FE codes. All new
variants are off by default and this shoudl not affect
current users.