Petsc names #208

Merged
merged 21 commits into from Mar 13, 2014

Conversation

Projects
None yet
6 participants
Owner

roystgnr commented Feb 21, 2014

This adds two libMesh command line options:

With --solver_system_names, solver objects get handed the name of the System they'll be solving. PETSc solvers use this to set a prefix for their command-line options; e.g. -ksp_type becomes -Poisson_ksp_type for a Poisson system. This is useful for independently tweaking solvers for decoupled problems.

With --solver_variable_names, solver objects get a chance to query the System for what variables are named what. PETSc solvers use this to set pc_fieldsplit variables. This is useful for doing things like Schur complement preconditioning on coupled problems.

I'm not sure the API for this is as clean as it could be. Worst of all: the new init() API argument will break (albeit with an easily-fixed compile-time error) any 3rd party codes subclassing LinearSolver or NonlinearSolver. I'll wait a week before merging to see if anyone has complaints or suggestions.

Results of testing c136a66 using libmesh recipe:

Failed on: linux

View the results here: https://www.moosebuild.com/view_job/25

Owner

friedmud commented Feb 26, 2014

Hmmmm - those test failures look like real failures. It appears that somehow the preconditioner isn't really getting set properly now...

Owner

roystgnr commented Feb 26, 2014

This is quite interesting - I'm not surprised to see a failure (I had to make some major changes today to make this more generally useful, and committed a checkpoint before testing), but this isn't the type of failure I was expecting.

Owner

friedmud commented Feb 26, 2014

I'm pretty busy over the next 2 days so I don't know when I'll get to do some hands-on testing of this.

Maybe @jwpeterson can work this in the background tomorrow to see what's up.

Owner

roystgnr commented Feb 26, 2014

Gah, I should have been more clear: although I wasn't expecting this precise sort of failure, I can still replicate a problem with my own apps and I ought to be able to work from there. No need to suck you guys in, although I do really appreciate having the extra information from CI to work from.

Although, if I'd remembered that moosebuild was going to be checking my work, I might have made a petsc_names_scratch branch before checkpointing a known-bad state! :-P Do you have moosebuild working on non-PR branches too, and if so how do I get to the results pages?

Owner

friedmud commented Feb 26, 2014

Unfortunately - for now it's still only on PRs. You will be able to use it on branches soon (that's the next thing that will be added).

Glad you're finding it useful - I'm hoping that it's going to help smooth things out between all of the libMesh/MOOSE community's various projects. We'll continue to ramp it up and add features as we go...

Contributor

karpeev commented Feb 26, 2014

In PETSc DM is the right object to serve up nontrivial splits to the various preconditioners.
In particular, by implementing DMCreateFieldDecomposition() a DM implementation can
define essentially arbitrary splits for PCFieldSplit. DMCreateDomainDecomposition() does
the same for PCASM/PCGASM. Then an implementation-specific API and/or command-line
options allow the user to configure the specific splits.

For example, I have DMCreateField/DomainDecomposition() implemented for DMMoose in MOOSE, which allows the user to name the splits and specify which mesh blocks and/or variables go into each split. The original intent was to develop this in DMlibMesh with some MOOSE-specific stuff layered on top, but with the resource constraints at the time it all ended up in DMMoose.

DMMoose uses little MOOSE-specific code, so it would be easy to move all that over to DMlibMesh, making it more useful, than it is at the moment. I can do this, but I wanted to see first whether it would clash with any petsc_auto_fieldsplit stuff.

A more complete implementation of DMlibMesh would also defining geometric coarsening/refinement and the attendant restriction/interpolation for PCMG. That would be very useful, but has been on the back burner for me for lack of time. It would be good to revive that.

Results of testing 81db871 using libmesh recipe:

Failed on: linux

View the results here: https://www.moosebuild.com/view_job/52

Owner

roystgnr commented Feb 26, 2014

"/home/gastdr/moosebuild/libmesh_build/moose/framework/src/outputs/Checkpoint.C:132:46: error: 'libMeshEnums' has not been declared" is a bit worrisome. Is that passing with libmesh master? And what's running for the remainder of the tests, if MOOSE didn't build?

Owner

jwpeterson commented Feb 26, 2014

On Wed, Feb 26, 2014 at 4:10 PM, roystgnr notifications@github.com wrote:

"/home/gastdr/moosebuild/libmesh_build/moose/framework/src/outputs/Checkpoint.C:132:46:
error: 'libMeshEnums' has not been declared" is a bit worrisome. Is that
passing with libmesh master? And what's running for the remainder of the
tests, if MOOSE didn't build?

The fix for this should be merged already, so that error message might be a
little bit out of date.

John

Owner

friedmud commented Feb 26, 2014

Yeah it should be fixed - I'm going to double check the recipe to make sure
it's doing the right thing...

Derek

On Wed, Feb 26, 2014 at 3:20 PM, jwpeterson notifications@github.comwrote:

On Wed, Feb 26, 2014 at 4:10 PM, roystgnr notifications@github.com
wrote:

"/home/gastdr/moosebuild/libmesh_build/moose/framework/src/outputs/Checkpoint.C:132:46:
error: 'libMeshEnums' has not been declared" is a bit worrisome. Is that
passing with libmesh master? And what's running for the remainder of the
tests, if MOOSE didn't build?

The fix for this should be merged already, so that error message might be a
little bit out of date.

John

Reply to this email directly or view it on GitHubhttps://github.com/libMesh/libmesh/pull/208#issuecomment-36190669
.

Results of testing 81db871 using libmesh recipe:

Failed on: linux

View the results here: https://www.moosebuild.com/view_job/52

Owner

friedmud commented Feb 27, 2014

Ok - I fixed up the build recipe (wasn't enough cleaning going on) and made it rebuild. The current failure can be seen here: https://www.moosebuild.com/view_result/57

It looks much better than before - only 6 MOOSE tests failing - and almost all of those look to be an initialization order issue with the preconditioners (somehow the solver is getting initialized before the preconditioner is set)....

Owner

roystgnr commented Feb 27, 2014

As of bd84082 this is working for me; now we just need to hammer out what it's breaking in Moose. And now that I can no longer replicate a failure I'm going to need some hints at where to look. Which System subclass(es) are the failing tests using? You're almost certainly right that the problem is my new code overzealously initializing the solver before you get a chance to do so; that's the problem I fixed in LinearImplicitSystem. But the only other place I could see that being an issue is in FEMSystem, which you're not using.

Owner

jwpeterson commented Feb 27, 2014

On Thu, Feb 27, 2014 at 2:20 PM, roystgnr notifications@github.com wrote:

As of bd84082 bd84082 this is
working for me; now we just need to hammer out what it's breaking in Moose.
And now that I can no longer replicate a failure I'm going to need some
hints at where to look. Which System subclass(es) are the failing tests
using? You're almost certainly right that the problem is my new code
overzealously initializing the solver before you get a chance to do so;
that's the problem I fixed in LinearImplicitSystem. But the only other
place I could see that being an issue is in FEMSystem, which you're not
using.

I'll take a look at the tests which are failing.

Would it have anything to do with the PetscPreconditioner class?

John

Owner

roystgnr commented Feb 27, 2014

I suspect the mechanism is that codes rely on calling init with a
PetscMatrix argument, but that is a null op if the solver has already
been initialized. This branch has to initialize solvers somewhere to
pass them system and variable name information, but if it does so too
early then it's precluding the subsequent init.

Owner

roystgnr commented Feb 27, 2014

I might need to force push a rebased version of this shortly; looks like a previous rebase caught a commit mid-bug from the libMeshEnums stuff.

roystgnr added some commits Feb 25, 2014

@roystgnr roystgnr Prepare to factor out petsc_auto_fieldsplit code 7243ee5
@roystgnr roystgnr Factor out petsc_auto_fieldsplit code da78f04
@roystgnr roystgnr Trying to add --solver_group_foo options
Not working yet, but need to checkpoint here.
228140f
@roystgnr roystgnr Use consistent name for old PETSc case a7f74c6
@roystgnr roystgnr Postpone library-level init() as late as possible
That way we don't preclude a physics-aware init(Matrix) from user
code.

This will hopefully fix some MOOSE regressions
15a91c1
@roystgnr roystgnr Got Schur complement fieldsplit working
The two last catches:

libMesh::on_command_line didn't work the way I had assumed, so I
replaced it with code that does.

For multiplicative and schur fieldsplits, PETSc really cares what
order you declare fields in.  But we can get arbitrary order by simply
putting everything in a group, because std::map orders groups
alphabetically and libMesh declares group fields in container order.
cfa6af9
Owner

jwpeterson commented Feb 27, 2014

On Thu, Feb 27, 2014 at 3:55 PM, roystgnr notifications@github.com wrote:

I might need to force push a rebased version of this shortly; looks like a
previous rebase caught a commit mid-bug from the libMeshEnums stuff.

That's fine, I haven't committed anything locally.

John

Results of testing cfa6af9 using libmesh recipe:

Failed on: linux

View the results here: https://www.moosebuild.com/view_job/72

Results of testing bd84082 using libmesh recipe:

Failed on: linux

View the results here: https://www.moosebuild.com/view_job/71

Member

permcody commented Feb 28, 2014

@roystgnr , don't let those 6 tests hold you up if you think everything is good on your end. Those PBP (Physics-based preconditioners) are fairly old and we might just need to change some call order in MOOSE. The public repository is imminent...

Owner

roystgnr commented Feb 28, 2014

No worries; this is an urgent capability for us to have for testing, but not an urgent one to have in master; the current users are git-friendly.

Although I might consider app-breaking changes if I understand exactly why the apps are breaking and can document the workaround, we're not at that point yet.

I am greatly looking forward to the public MOOSE repo.

roystgnr added some commits Feb 28, 2014

@roystgnr roystgnr local_variable_indices bugfix
This fixes a case of local_variable_indices returning a non-local
index, and adds an assert to see that it doesn't happen again.

-pc_type_fieldsplit works correctly in parallel now.
ac4b3b9
@roystgnr roystgnr Remove unused variable 8c3cc6d

Results of testing ac4b3b9 using libmesh recipe:

Failed on: linux

View the results here: https://www.moosebuild.com/view_job/85

Results of testing 8c3cc6d using libmesh recipe:

Failed on: linux

View the results here: https://www.moosebuild.com/view_job/86

@roystgnr roystgnr Postpone automatic NonlinearSolver initialization
This should increase backwards compatibility of the petsc_names
changes, and in particular should hopefully fix more of the MOOSE
regressions.
5718971

Results of testing 5718971 using libmesh recipe:

Passed on: linux

View the results here: https://www.moosebuild.com/view_job/329

Owner

roystgnr commented Mar 12, 2014

Looks like MOOSE is happy after that last fix. (which was to a bug in this branch; not a workaround for anything wrong in MOOSE). Assuming my own tests don't show new problems and nobody objects, I'll merge tomorrow.

@roystgnr roystgnr added a commit that referenced this pull request Mar 13, 2014

@roystgnr roystgnr Merge pull request #208 from libMesh/petsc_names
Petsc names - support for system-specific PETSc options and named-variable PETSc fieldsplit options.
5c25f1c

@roystgnr roystgnr merged commit 5c25f1c into master Mar 13, 2014

1 check passed

default Successfully passed all tests
Details

jwpeterson deleted the petsc_names branch Mar 14, 2014

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment