As the number of computational cores increases, the convergence performance of the adjoint equation deteriorates.

[z-g-h]

Dear Professor,

I'm working on a problem using DAFOAM and have encountered an issue with the convergence of the adjoint equation in parallel computations. Specifically, I've observed that as the number of computational cores increases, the convergence performance of the adjoint equation worsens. For instance, with 4 cores, the adjoint equation converges quickly,

but when scaling up to 12 cores, the convergence slows significantly.

This behavior appears to be related to the 1-level ASM preconditioner in PETSc. I'm wondering whether switching to either the GAMG or 2-level ASM preconditioner could help accelerate the convergence of the adjoint equation under multi-core computations. Additionally, if they do prove effective, would it be feasible to implement these preconditioners for adjoint equation solver within PETSc ?

Thanks,
Genghui Zhu

[friedenhe]

Hmm, that is very strange. We did not observe such behavior before. When you use 12 cores, did the adjoint convergence stall happen at the first adjoint solution or later solutions? Yes, you can increase ASM to 2. Check here. Also, check the DAFoam website FAQ on how to improve adjoint convergence.

[z-g-h]

Thank you for your reply. The adjoint convergence stalls occur in the first adjoint solution.

I have tested increasing the asmOverlap parameter to 4, and it does help improve convergence. Specifically, the convergence stalls initially but then accelerates quickly after reaching a certain residual value.

However, when working with a large number of cells, I need to increase both the asmOverlap parameter and the pcfillLevel parameter, which consumes a significant amount of memory.

By the 2-level ASM I mentioned earlier, I mean an approach similar to the multigrid method—one that constructs a coarse space to eliminate coarse-scale error components. This approach seems promising for addressing the issue (as referenced in chapter 4), but I'm not sure how to implement it in PETSc, and I would appreciate any suggestions you might have regarding 2-level ASM.
Dolean et al. - 2015 - An Introduction to Domain Decomposition Methods Algorithms, Theory, and Parallel Implementation.pdf

[z-g-h]

I have re-tested my case, which involves a subsonic turbine. I found that when I switched the solver from DATurboFoam to DARhoSimpleFoam, convergence remains good regardless of the number of cores used. It seems there might be an issue with the preconditioner (PC) constructed by DATurboFoam. Upon examining the eigenvalues during GMRES computation, I noticed that when using DATurboFoam, there are extremely small eigenvalues—even on the order of 1e-6.
Moreover, I’ve observed that in DATurboFoam, certain terms of the pressure equation are omitted in constructing the transonic dRdWPC to improve convergence.

dafoam/src/adjoint/DAResidual/DAResidualTurboFoam.C

Lines 174 to 178 in 2583572

	// for PC we do not include the div(phid, p) term, this improves the convergence
	if (isPC && daOption_.getOption<label>("transonicPCOption") == 1)
	{
	pEqn -= fvm::div(phid, p_, divPhidPScheme);
	}

Given this, would it be feasible to improve the PC’s convergence by omitting certain terms of equations under subsonic conditions in a similar manner?

[friedenhe]

@z-g-h Regarding the 2-level ASM, I believe Petsc has an interface, but due to the limited manpower, we won't be able to implement it in the near future. Regarding the PC for DATurbFoam, yes, we observe that its convergence is less robust than DARhoSimpleFoam. For subsonic flow, I suggest you use DARhoSimpleFoam. But for transonic flow, DATurboFoam should be better. We haven't thoroughly tested the effect of dropping certain terms in the pEqn to improve convergence for subsonic conditions, but that is definitely sth you can try. If you find something, please let us know or submit a pull request.

[z-g-h]

The reason is that increasing the number of parallel cores degrades the quality of the ILU preconditioner, as off-diagonal blocks emerge, as illustrated in the spy plot.

Using the KAHIP method for parallel decomposition could improve convergence performance. This is because KAHIP is capable of yielding a superior decomposition, thereby enhancing the diagonal properties of the preconditioner.

[friedenhe]

@z-g-h Thanks very much for the feedback! This is very helpful for improving adjoint convergence for large-scale case running in parallel! We have added this feature into a recent commit.