solving linear problem fails in parallel (Object is in wrong state)

[tkarna]

This example is adapted from the mixed element preconditioning example. It works fine in sequential. In parallel it fails on the second solve (e.g. in a loop for time dependent system).

from firedrake import *
n = 30
mesh = UnitSquareMesh(n, n)
V1 = FunctionSpace(mesh, 'RT', 1)
V2 = FunctionSpace(mesh, 'DG', 0)
W = V1 * V2
lmbda = 1
u, p = TrialFunctions(W)
v, q = TestFunctions(W)
f = Function(V2)
f.interpolate(Expression('1e-7')) # need to set nonzero value

a = (p*q - q*div(u) + lmbda*inner(v, u) + div(v)*p)*dx
L = f*q*dx

solution = Function(W)
solver_parameters={'ksp_type': 'cg',
                    'pc_type': 'fieldsplit',
                    'pc_fieldsplit_type': 'schur',
                    'pc_fieldsplit_schur_fact_type': 'FULL',
                    'fieldsplit_0_ksp_type': 'cg',
                    'fieldsplit_1_ksp_type': 'cg'}
# works in parallel
solve(a == L, solution, solver_parameters=solver_parameters)
# fails
solve(a == L, solution, solver_parameters=solver_parameters)

Error stack:

Traceback (most recent call last):
  File "mixed_pc_test.py", line 27, in <module>
    solve(a == L, solution, solver_parameters=solver_parameters)
  File "[...]/firedrake/solving.py", line 900, in solve
    _solve_varproblem(*args, **kwargs)
  File "[...]/firedrake/solving.py", line 930, in _solve_varproblem
    solver.solve()
  File "[...]/firedrake/solving.py", line 283, in solve
    self.snes.solve(None, v)
  File "SNES.pyx", line 413, in petsc4py.PETSc.SNES.solve (src/petsc4py.PETSc.c:133749)
petsc4py.PETSc.Error: error code 73
[0] SNESSolve() line 3794 in /tmp/pip_build_root/petsc/src/snes/interface/snes.c
[0] SNESSolve_KSPONLY() line 43 in /tmp/pip_build_root/petsc/src/snes/impls/ksponly/ksponly.c
[0] KSPSolve() line 458 in /tmp/pip_build_root/petsc/src/ksp/ksp/interface/itfunc.c
[0] KSPSolve_CG() line 218 in /tmp/pip_build_root/petsc/src/ksp/ksp/impls/cg/cg.c
[0] KSP_MatMult() line 204 in /tmp/pip_build_root/petsc/include/petsc-private/kspimpl.h
[0] MatMult() line 2263 in /tmp/pip_build_root/petsc/src/mat/interface/matrix.c
[0] Object is in wrong state
[0] Not for unassembled matrix

I'm using up-to-date firedrake (2450fb9), petsc4py (c912a42) and petsc (d6d4e9d).

[dham]

Hi Thomas,

I suspect this is a known issue (#271) in the assembly cache which I have not yet figured out a good solution for.

Could you try setting:

parameters["assembly_cache"]["enabled"] = False

at the top of your script?

[tkarna]

Hi David,

Thanks for the tip, with that setting it runs fine. Should I consider this as a workaround for the time being?

[dham]

Yes please. I'll have a chat with the others later today and we'll decide whether to actually put the workaround in master or whether we have a better solution.

[tkarna]

OK, sounds good. Thanks.

[wence-]

Thanks for this, this was not an instance of #271, but a different problem. I believe it is fixed by fix/parallel_mixed_assembly_cache. So you could try that, or wait til it makes it into master.

[tkarna]

Thanks for the fix, I confirm that it solves the issue.

[kynan]

Hence closing.