You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
There seems to be some issues with tutorial case specifically on DArhosimpleCfoam.
All optimisation were testing using lastest DaFoam V5.0.0 on HPC's with different partitions and node sizes
I was trying to run all the DArhoSimpleCFoam tutorial to understand the adjoint convergence pattern. There seems to be some issues with adjoint convergence even with the tutorial case. I tested all 3 cases CRM_Wing, Onera and DPW4. Except DPW4 none of the other two seems to converge with the default settings.
CRM_Wing
I have to mention there was some issue in meshing pyHyp in the HPC so i meshed it in my local PC and copied the polyMesh to the case file
This was all default settings I just made the maxIterations to 6000
GMRES Restart: 2000
ASM Overlap: 1
Global PC Iters: 0
Local PC Iters: 1
Mat ReOrdering: natural
ILU PC Fill Level: 2
GMRES Max Iterations: 6000
GMRES Relative Tolerance: 1e-05
GMRES Absolute Tolerance: 1e-14
Solving Linear Equation... 1894.91 s
Main iteration 0 KSP Residual norm 7.478947251507e-03 2615.28 s.
Main iteration 100 KSP Residual norm 7.474016499348e-03 2664.11 s.
Main iteration 200 KSP Residual norm 7.473134051868e-03 2713.38 s.
Main iteration 300 KSP Residual norm 7.472496829662e-03 2764.02 s.
Main iteration 400 KSP Residual norm 7.471859770433e-03 2816.15 s.
Main iteration 500 KSP Residual norm 7.471222874112e-03 2869.39 s.
Main iteration 600 KSP Residual norm 7.470586140628e-03 2924.03 s.
Main iteration 700 KSP Residual norm 7.469949569913e-03 2981.28 s.
Main iteration 800 KSP Residual norm 7.469313161898e-03 3039.52 s.
Main iteration 900 KSP Residual norm 7.468676916513e-03 3098.99 s.
Main iteration 1000 KSP Residual norm 7.468040833688e-03 3159.20 s.
Main iteration 1100 KSP Residual norm 7.467404913356e-03 3221.27 s.
Main iteration 1200 KSP Residual norm 7.466769155445e-03 3286.18 s.
Main iteration 1300 KSP Residual norm 7.466133559888e-03 3352.81 s.
Main iteration 1400 KSP Residual norm 7.465498126615e-03 3421.06 s.
Main iteration 1500 KSP Residual norm 7.464862855558e-03 3491.60 s.
I increased the ILU and ASM to 2
GMRES Restart: 2000
ASM Overlap: 2
Global PC Iters: 0
Local PC Iters: 1
Mat ReOrdering: nd
ILU PC Fill Level: 2
GMRES Max Iterations: 6000
GMRES Relative Tolerance: 1e-05
GMRES Absolute Tolerance: 1e-14
Solving Linear Equation... 1520.96 s
Main iteration 0 KSP Residual norm 7.478939350680e-03 2726.42 s.
Main iteration 100 KSP Residual norm 7.478935470584e-03 2788.92 s.
Main iteration 200 KSP Residual norm 7.478917345250e-03 2852.14 s.
Main iteration 300 KSP Residual norm 7.478900329958e-03 2916.32 s.
Main iteration 400 KSP Residual norm 7.478883314783e-03 2981.73 s.
Main iteration 500 KSP Residual norm 7.478866299724e-03 3048.58 s.
Main iteration 600 KSP Residual norm 7.478849284781e-03 3117.52 s.
Main iteration 700 KSP Residual norm 7.478832269955e-03 3187.07 s.
Main iteration 800 KSP Residual norm 7.478815255244e-03 3259.59 s.
Main iteration 900 KSP Residual norm 7.478798240650e-03 3332.61 s.
Main iteration 1000 KSP Residual norm 7.478781226171e-03 3408.50 s.
Main iteration 1100 KSP Residual norm 7.478764211809e-03 3485.14 s.
Then i switch the fvSchemes to upwind for div(phi,U), div(phid,p) with ILU. ASM - 2
GMRES Restart: 2000
ASM Overlap: 2
Global PC Iters: 0
Local PC Iters: 1
Mat ReOrdering: nd
ILU PC Fill Level: 2
GMRES Max Iterations: 6000
GMRES Relative Tolerance: 1e-05
GMRES Absolute Tolerance: 1e-14
Solving Linear Equation... 1338.89 s
Main iteration 0 KSP Residual norm 7.476403545991e-03 1491.11 s.
Main iteration 100 KSP Residual norm 7.467640653922e-03 1549.96 s.
Main iteration 200 KSP Residual norm 7.434405960640e-03 1612.16 s.
Main iteration 300 KSP Residual norm 7.407615238857e-03 1678.36 s.
Main iteration 400 KSP Residual norm 7.387065089186e-03 1747.10 s.
Main iteration 500 KSP Residual norm 7.353987098249e-03 1819.45 s.
Main iteration 600 KSP Residual norm 7.329169368963e-03 1893.75 s.
Main iteration 700 KSP Residual norm 7.280724909714e-03 1972.89 s.
Main iteration 800 KSP Residual norm 7.219377355706e-03 2053.55 s.
Main iteration 900 KSP Residual norm 7.142154162747e-03 2137.83 s.
Main iteration 1000 KSP Residual norm 7.087758297414e-03 2225.20 s.
Main iteration 1100 KSP Residual norm 7.021461405432e-03 2317.34 s.
Main iteration 1200 KSP Residual norm 6.976567477442e-03 2412.40 s.
Main iteration 1300 KSP Residual norm 6.941950047016e-03 2510.51 s.
Main iteration 1400 KSP Residual norm 6.895235261259e-03 2612.37 s.
Main iteration 1500 KSP Residual norm 6.823520923884e-03 2717.90 s.
Main iteration 1600 KSP Residual norm 6.770539897469e-03 2827.06 s.
Main iteration 1700 KSP Residual norm 6.721366458903e-03 2938.63 s.
Main iteration 1800 KSP Residual norm 6.642427851340e-03 3054.27 s.
Main iteration 1900 KSP Residual norm 6.576821855539e-03 3172.17 s.
Main iteration 2000 KSP Residual norm 6.534695486253e-03 3293.78 s.
Main iteration 2100 KSP Residual norm 6.534695486219e-03 3352.47 s.
Main iteration 2200 KSP Residual norm 6.534695485823e-03 3414.63 s.
Main iteration 2300 KSP Residual norm 6.534695483527e-03 3479.14 s.
Main iteration 2400 KSP Residual norm 6.534695479829e-03 3548.68 s.
Main iteration 2500 KSP Residual norm 6.534695474640e-03 3619.43 s.
Main iteration 2600 KSP Residual norm 6.534695470102e-03 3694.80 s.
Main iteration 2700 KSP Residual norm 6.534695458409e-03 3771.49 s.
Main iteration 2800 KSP Residual norm 6.534695440378e-03 3853.94 s.
Main iteration 2900 KSP Residual norm 6.534695409617e-03 3938.93 s.
Main iteration 3000 KSP Residual norm 6.534695375298e-03 4027.51 s.
Main iteration 3100 KSP Residual norm 6.534695224378e-03 4119.35 s.
Main iteration 3200 KSP Residual norm 6.534694981088e-03 4213.97 s.
Main iteration 3300 KSP Residual norm 6.534694387186e-03 4310.25 s.
Main iteration 3400 KSP Residual norm 6.534692216359e-03 4411.79 s.
Main iteration 3500 KSP Residual norm 6.534686956837e-03 4516.31 s.
Main iteration 3600 KSP Residual norm 6.534661972040e-03 4624.64 s.
Main iteration 3700 KSP Residual norm 6.534576351206e-03 4735.26 s.
Never converged
Onera_Wing
Onera_M6_Wing-SnappyHexMesh- All default tutorial settings
Solver Type: gmres
GMRES Restart: 1000
ASM Overlap: 1
Global PC Iters: 0
Local PC Iters: 1
Mat ReOrdering: natural
ILU PC Fill Level: 1
GMRES Max Iterations: 1000
GMRES Relative Tolerance: 0.0001
GMRES Absolute Tolerance: 1e-14
Solving Linear Equation... 947.41 s
Main iteration 0 KSP Residual norm 1.014975657480e-02 1018.56 s.
Main iteration 100 KSP Residual norm 1.014975657477e-02 1060.45 s.
Main iteration 200 KSP Residual norm 1.014975657477e-02 1106.66 s.
Main iteration 300 KSP Residual norm 1.014975657477e-02 1152.74 s.
Main iteration 400 KSP Residual norm 1.014975657477e-02 1201.70 s.
Main iteration 500 KSP Residual norm 1.014975657477e-02 1252.69 s.
Main iteration 600 KSP Residual norm 1.014975657477e-02 1306.30 s.
Main iteration 700 KSP Residual norm 1.014975657477e-02 1362.71 s.
Main iteration 800 KSP Residual norm 1.014975657477e-02 1421.38 s.
Main iteration 900 KSP Residual norm 1.014975657477e-02 1482.83 s.
Main iteration 1000 KSP Residual norm 1.014975657477e-02 1546.47 s.
Main iteration 1000 KSP Residual norm 1.014975657477e-02 1546.74 s Completed! Total iterations: 1000. PetscConvergedReason: -3. 1546.74 s
Residual tolerance not satisfied, solution failed!
the default tutorial wasnt converging, this was wth SnappyHexMesh
Then i switch the fvSchemes to upwind for div(phi,U), div(phid,p) with ILU. ASM - 2
I dont which one helped when incresing the ILU or switching to upwind but it worked. Just wanted to mention this here. if someone new tries this it would help.
GMRES Restart: 1000
ASM Overlap: 1
Global PC Iters: 0
Local PC Iters: 1
Mat ReOrdering: natural
ILU PC Fill Level: 2
GMRES Max Iterations: 1000
GMRES Relative Tolerance: 0.0001
GMRES Absolute Tolerance: 1e-14
Solving Linear Equation... 197.15 s
Main iteration 0 KSP Residual norm 1.014469015953e-02 690.90 s.
Main iteration 63 KSP Residual norm 9.082773158709e-07 722.90 s Completed! Total iterations: 63. PetscConvergedReason: 2. 722.9 s
Residual tolerance satisfied, solution finished!
Computing d[aero_residuals]/d[aero_vol_coords]^T * psi 723.0599999999999 s
Computing d[aero_residuals]/d[patchV]^T * psi 725.47 s
Computing d[CL]/d[aero_states]^T * psi 726.55 s
Computing d[CL]/d[aero_vol_coords]^T * psi 726.63 s
Computing d[CL]/d[patchV]^T * psi 727.13 s
Latest solution time 0.0001 less than 1, not renamed.
Solving Linear Equation... 727.2 s
Main iteration 0 KSP Residual norm 1.039898147074e-01 727.21 s.
Main iteration 67 KSP Residual norm 8.235637892428e-06 756.37 s Completed! Total iterations: 67. PetscConvergedReason: 2. 756.37 s
Residual tolerance satisfied, solution finished!
When i tried the pyHyp version ( the same issue with pyHyp in the HPC ) the primal seems to work well so i didnt suspect the mesh might be the issue but it can also be the case.
Onera_M6_Wing-PyHyp - All Default settings ran on 20 CPU in one node with 64GB Ram
Solver Type: gmres
GMRES Restart: 1000
ASM Overlap: 1
Global PC Iters: 0
Local PC Iters: 1
Mat ReOrdering: natural
ILU PC Fill Level: 1
GMRES Max Iterations: 1000
GMRES Relative Tolerance: 0.0001
GMRES Absolute Tolerance: 1e-14
Solving Linear Equation... 2279.35 s
Primary job terminated normally, but 1 process returned
a non-zero exit code. Per user-direction, the job has been aborted.
mpiexec noticed that process rank 14 with PID 195195 on node node143 exited on signal 9 (Killed).
Onera_M6_Wing-PyHyp - All Default settings ran on 20 CPU in 2 node with 64GB Ram each
Solver Type: gmres
GMRES Restart: 1000
ASM Overlap: 1
Global PC Iters: 0
Local PC Iters: 1
Mat ReOrdering: natural
ILU PC Fill Level: 1
GMRES Max Iterations: 1000
GMRES Relative Tolerance: 0.0001
GMRES Absolute Tolerance: 1e-14
Solving Linear Equation... 587.38 s
Primary job terminated normally, but 1 process returned
a non-zero exit code. Per user-direction, the job has been aborted.
mpiexec noticed that process rank 7 with PID 0 on node node179 exited on signal 9 (Killed).
slurmstepd: error: Detected 2 oom-kill event(s) in StepId=3370.batch. Some of your processes may have been killed by the cgroup out-of-memory handler.
Both ran out of memory - for 37k cells I used 64GB , 128GB ram respectively.
DPW4
DPW4 case works fine so i dont think I have a major compilation issues
DPW4 - Default fvSchemes but used nd and ASM and ILU 2 - works fine
GMRES Restart: 2000
ASM Overlap: 2
Global PC Iters: 0
Local PC Iters: 1
Mat ReOrdering: nd
ILU PC Fill Level: 2
GMRES Max Iterations: 6000
GMRES Relative Tolerance: 1e-05
GMRES Absolute Tolerance: 1e-14
Solving Linear Equation... 2343.61 s
Main iteration 0 KSP Residual norm 6.882407371012e-03 2645.09 s.
Main iteration 100 KSP Residual norm 4.477090036643e-03 2724.56 s.
Main iteration 200 KSP Residual norm 2.242989304830e-04 2807.99 s.
Main iteration 300 KSP Residual norm 7.529208022692e-06 2895.35 s.
Main iteration 400 KSP Residual norm 1.617210552715e-06 2986.51 s.
Main iteration 500 KSP Residual norm 1.341288205115e-06 3082.02 s.
Main iteration 600 KSP Residual norm 4.675592890766e-07 3181.49 s.
Main iteration 700 KSP Residual norm 3.137283790770e-07 3285.51 s.
Main iteration 800 KSP Residual norm 1.672251865516e-07 3393.74 s.
Main iteration 859 KSP Residual norm 6.755619671626e-08 3460.91 s Completed! Total iterations: 859. PetscConvergedReason: 2. 3460.91 s
Residual tolerance satisfied, solution finished!
Computing d[aero_residuals]/d[aero_vol_coords]^T * psi 3461.53 s
Computing d[aero_residuals]/d[patchV]^T * psi 3463.62 s
Computing d[CL]/d[aero_states]^T * psi 3469.22 s
Computing d[CL]/d[aero_vol_coords]^T * psi 3469.42 s
Computing d[CL]/d[patchV]^T * psi 3470.22 s
Latest solution time 0.0001 less than 1, not renamed.
Solving Linear Equation... 3470.45 s
Main iteration 0 KSP Residual norm 3.996523193994e-02 3470.47 s.
Main iteration 100 KSP Residual norm 3.103135466961e-02 3551.77 s.
Main iteration 200 KSP Residual norm 2.611675097729e-03 3636.96 s.
Main iteration 300 KSP Residual norm 9.191504166849e-05 3725.99 s.
Main iteration 400 KSP Residual norm 3.395599927045e-05 3819.15 s.
Main iteration 500 KSP Residual norm 2.631575968070e-05 3916.70 s.
Main iteration 600 KSP Residual norm 1.173080812103e-05 4018.36 s.
Main iteration 700 KSP Residual norm 2.975987797344e-06 4124.77 s.
Main iteration 788 KSP Residual norm 3.933333154229e-07 4221.52 s Completed! Total iterations: 788. PetscConvergedReason: 2. 4221.52 s
Residual tolerance satisfied, solution finished!
Computing d[aero_residuals]/d[aero_vol_coords]^T * psi 4222.05 s
Computing d[aero_residuals]/d[patchV]^T * psi 4223.7 s
Computing d[CMY]/d[aero_states]^T * psi 4228.71 s
Computing d[CMY]/d[aero_vol_coords]^T * psi 4229.24 s
Computing d[CMY]/d[patchV]^T * psi 4230.87 s
Latest solution time 0.0001 less than 1, not renamed.
Solving Linear Equation... 4231.08 s
Main iteration 0 KSP Residual norm 8.582237831085e-02 4231.10 s.
Main iteration 100 KSP Residual norm 5.327334512464e-02 4311.49 s.
Main iteration 200 KSP Residual norm 1.098839628518e-02 4396.74 s.
Main iteration 300 KSP Residual norm 2.673713515769e-04 4485.96 s.
Main iteration 400 KSP Residual norm 5.143336365822e-05 4579.55 s.
Main iteration 500 KSP Residual norm 4.118456647598e-05 4677.22 s.
Main iteration 600 KSP Residual norm 2.648278681934e-05 4779.95 s.
Main iteration 700 KSP Residual norm 7.289332476310e-06 4885.94 s.
Main iteration 793 KSP Residual norm 8.563865965162e-07 4989.15 s Completed! Total iterations: 793. PetscConvergedReason: 2. 4989.15 s
Residual tolerance satisfied, solution finished!
Computing d[aero_residuals]/d[aero_vol_coords]^T * psi 4989.71 s
Computing d[aero_residuals]/d[patchV]^T * psi 4991.36 s
Fuselage only case with my own mesh:
I made my own case to converge too but it takes a lot of time to converge.
GMRES Restart: 2000
ASM Overlap: 2
Global PC Iters: 0
Local PC Iters: 1
Mat ReOrdering: nd
ILU PC Fill Level: 2
GMRES Max Iterations: 6000
GMRES Relative Tolerance: 1e-05
GMRES Absolute Tolerance: 1e-14
Solving Linear Equation... 438.59 s
Main iteration 0 KSP Residual norm 4.627779785159e-02 547.81 s.
Main iteration 100 KSP Residual norm 4.297427553024e-02 598.57 s.
Main iteration 200 KSP Residual norm 4.044975505490e-02 650.89 s.
Main iteration 300 KSP Residual norm 4.007992573566e-02 704.73 s.
Main iteration 400 KSP Residual norm 3.961851191266e-02 761.95 s.
Main iteration 500 KSP Residual norm 3.925206474714e-02 823.51 s.
Main iteration 600 KSP Residual norm 3.892740846860e-02 886.00 s.
Main iteration 700 KSP Residual norm 3.862389265612e-02 951.54 s.
Main iteration 800 KSP Residual norm 3.735477939942e-02 1020.14 s.
Main iteration 900 KSP Residual norm 3.283145453496e-02 1092.43 s.
Main iteration 1000 KSP Residual norm 1.691081147286e-02 1165.93 s.
Main iteration 1100 KSP Residual norm 1.424811440751e-02 1244.88 s.
Main iteration 1200 KSP Residual norm 7.971463026040e-03 1326.10 s.
Main iteration 1300 KSP Residual norm 5.785497525780e-03 1409.17 s.
Main iteration 1400 KSP Residual norm 3.719856486417e-03 1494.84 s.
Main iteration 1500 KSP Residual norm 3.491911425905e-03 1583.66 s.
Main iteration 1600 KSP Residual norm 2.212460759752e-03 1677.30 s.
Main iteration 1700 KSP Residual norm 1.220821391440e-03 1772.56 s.
Main iteration 1800 KSP Residual norm 9.299593945216e-04 1869.56 s.
Main iteration 1900 KSP Residual norm 9.058616468088e-04 1969.51 s.
Main iteration 2000 KSP Residual norm 6.860726923258e-04 2073.84 s.
Main iteration 2100 KSP Residual norm 6.857984643318e-04 2122.35 s.
Main iteration 2200 KSP Residual norm 6.845706124816e-04 2175.28 s.
Main iteration 2300 KSP Residual norm 6.827801823044e-04 2230.25 s.
Main iteration 2400 KSP Residual norm 6.820637592422e-04 2288.19 s.
Main iteration 2500 KSP Residual norm 6.798229099414e-04 2350.38 s.
Main iteration 2600 KSP Residual norm 6.735072550497e-04 2415.25 s.
Main iteration 2700 KSP Residual norm 6.701328861451e-04 2482.53 s.
Main iteration 2800 KSP Residual norm 6.643231034277e-04 2553.19 s.
Main iteration 2900 KSP Residual norm 6.575963567621e-04 2624.94 s.
Main iteration 3000 KSP Residual norm 6.480460373426e-04 2700.54 s.
Main iteration 3100 KSP Residual norm 6.364503417456e-04 2777.95 s.
Main iteration 3200 KSP Residual norm 5.801047480017e-04 2859.66 s.
Main iteration 3300 KSP Residual norm 5.616929820161e-04 2942.08 s.
Main iteration 3400 KSP Residual norm 3.843751054285e-04 3029.00 s.
Main iteration 3500 KSP Residual norm 2.498082809234e-04 3118.39 s.
Main iteration 3600 KSP Residual norm 1.597154370664e-04 3210.71 s.
Main iteration 3700 KSP Residual norm 1.060344700378e-04 3306.47 s.
Main iteration 3800 KSP Residual norm 8.351047562573e-05 3404.86 s.
Main iteration 3900 KSP Residual norm 6.010383825797e-05 3506.79 s.
Main iteration 4000 KSP Residual norm 3.747821399429e-05 3611.41 s.
Main iteration 4100 KSP Residual norm 3.741542550164e-05 3663.06 s.
Main iteration 4200 KSP Residual norm 3.716349669959e-05 3717.45 s.
Main iteration 4300 KSP Residual norm 3.709758512119e-05 3774.19 s.
Main iteration 4400 KSP Residual norm 3.676696627831e-05 3833.61 s.
Main iteration 4500 KSP Residual norm 3.664371155861e-05 3895.58 s.
Main iteration 4600 KSP Residual norm 3.646659191785e-05 3959.47 s.
Main iteration 4700 KSP Residual norm 3.618258050409e-05 4026.42 s.
Main iteration 4800 KSP Residual norm 3.613653254141e-05 4096.43 s.
Main iteration 4900 KSP Residual norm 3.599632663952e-05 4167.29 s.
Main iteration 5000 KSP Residual norm 3.579278370329e-05 4244.25 s.
Main iteration 5100 KSP Residual norm 3.445441961804e-05 4321.02 s.
Main iteration 5200 KSP Residual norm 2.988696686892e-05 4401.16 s.
Main iteration 5300 KSP Residual norm 2.431664348810e-05 4486.21 s.
Main iteration 5400 KSP Residual norm 1.812248326263e-05 4573.12 s.
Main iteration 5500 KSP Residual norm 1.569776476815e-05 4664.52 s.
Main iteration 5600 KSP Residual norm 1.497439974401e-05 4756.98 s.
Main iteration 5700 KSP Residual norm 1.425306972007e-05 4852.40 s.
Main iteration 5800 KSP Residual norm 1.178476098808e-05 4950.53 s.
Main iteration 5900 KSP Residual norm 8.895587702316e-06 5050.87 s.
Main iteration 6000 KSP Residual norm 5.784482010810e-06 5155.40 s.
Main iteration 6000 KSP Residual norm 5.784482010810e-06 5155.71 s Completed! Total iterations: 6000. PetscConvergedReason: -3. 5155.71 s
Residual tolerance satisfied, solution finished!
barely convegred the Tol which I prescribed wasnt met
Can someone please confirm whether its my HPC or mesh issue or there is a actual problem in the tutorial.
Also, Any inputs on how to speedup the Adjoint convergence would be appreciated. I tried all the things mentioned in forum and also on the website and I dont know whether i should consider trying yplus above 60. but I havent tried above 45.
reacted with thumbs up emoji reacted with thumbs down emoji reacted with laugh emoji reacted with hooray emoji reacted with confused emoji reacted with heart emoji reacted with rocket emoji reacted with eyes emoji
Uh oh!
There was an error while loading. Please reload this page.
Hello Everyone,
There seems to be some issues with tutorial case specifically on DArhosimpleCfoam.
All optimisation were testing using lastest DaFoam V5.0.0 on HPC's with different partitions and node sizes
I was trying to run all the DArhoSimpleCFoam tutorial to understand the adjoint convergence pattern. There seems to be some issues with adjoint convergence even with the tutorial case. I tested all 3 cases CRM_Wing, Onera and DPW4. Except DPW4 none of the other two seems to converge with the default settings.
I have to mention there was some issue in meshing pyHyp in the HPC so i meshed it in my local PC and copied the polyMesh to the case file
This was all default settings I just made the maxIterations to 6000
GMRES Restart: 2000
ASM Overlap: 1
Global PC Iters: 0
Local PC Iters: 1
Mat ReOrdering: natural
ILU PC Fill Level: 2
GMRES Max Iterations: 6000
GMRES Relative Tolerance: 1e-05
GMRES Absolute Tolerance: 1e-14
Solving Linear Equation... 1894.91 s
Main iteration 0 KSP Residual norm 7.478947251507e-03 2615.28 s.
Main iteration 100 KSP Residual norm 7.474016499348e-03 2664.11 s.
Main iteration 200 KSP Residual norm 7.473134051868e-03 2713.38 s.
Main iteration 300 KSP Residual norm 7.472496829662e-03 2764.02 s.
Main iteration 400 KSP Residual norm 7.471859770433e-03 2816.15 s.
Main iteration 500 KSP Residual norm 7.471222874112e-03 2869.39 s.
Main iteration 600 KSP Residual norm 7.470586140628e-03 2924.03 s.
Main iteration 700 KSP Residual norm 7.469949569913e-03 2981.28 s.
Main iteration 800 KSP Residual norm 7.469313161898e-03 3039.52 s.
Main iteration 900 KSP Residual norm 7.468676916513e-03 3098.99 s.
Main iteration 1000 KSP Residual norm 7.468040833688e-03 3159.20 s.
Main iteration 1100 KSP Residual norm 7.467404913356e-03 3221.27 s.
Main iteration 1200 KSP Residual norm 7.466769155445e-03 3286.18 s.
Main iteration 1300 KSP Residual norm 7.466133559888e-03 3352.81 s.
Main iteration 1400 KSP Residual norm 7.465498126615e-03 3421.06 s.
Main iteration 1500 KSP Residual norm 7.464862855558e-03 3491.60 s.
I increased the ILU and ASM to 2
GMRES Restart: 2000
ASM Overlap: 2
Global PC Iters: 0
Local PC Iters: 1
Mat ReOrdering: nd
ILU PC Fill Level: 2
GMRES Max Iterations: 6000
GMRES Relative Tolerance: 1e-05
GMRES Absolute Tolerance: 1e-14
Solving Linear Equation... 1520.96 s
Main iteration 0 KSP Residual norm 7.478939350680e-03 2726.42 s.
Main iteration 100 KSP Residual norm 7.478935470584e-03 2788.92 s.
Main iteration 200 KSP Residual norm 7.478917345250e-03 2852.14 s.
Main iteration 300 KSP Residual norm 7.478900329958e-03 2916.32 s.
Main iteration 400 KSP Residual norm 7.478883314783e-03 2981.73 s.
Main iteration 500 KSP Residual norm 7.478866299724e-03 3048.58 s.
Main iteration 600 KSP Residual norm 7.478849284781e-03 3117.52 s.
Main iteration 700 KSP Residual norm 7.478832269955e-03 3187.07 s.
Main iteration 800 KSP Residual norm 7.478815255244e-03 3259.59 s.
Main iteration 900 KSP Residual norm 7.478798240650e-03 3332.61 s.
Main iteration 1000 KSP Residual norm 7.478781226171e-03 3408.50 s.
Main iteration 1100 KSP Residual norm 7.478764211809e-03 3485.14 s.
Then i switch the fvSchemes to upwind for div(phi,U), div(phid,p) with ILU. ASM - 2
GMRES Restart: 2000
ASM Overlap: 2
Global PC Iters: 0
Local PC Iters: 1
Mat ReOrdering: nd
ILU PC Fill Level: 2
GMRES Max Iterations: 6000
GMRES Relative Tolerance: 1e-05
GMRES Absolute Tolerance: 1e-14
Solving Linear Equation... 1338.89 s
Main iteration 0 KSP Residual norm 7.476403545991e-03 1491.11 s.
Main iteration 100 KSP Residual norm 7.467640653922e-03 1549.96 s.
Main iteration 200 KSP Residual norm 7.434405960640e-03 1612.16 s.
Main iteration 300 KSP Residual norm 7.407615238857e-03 1678.36 s.
Main iteration 400 KSP Residual norm 7.387065089186e-03 1747.10 s.
Main iteration 500 KSP Residual norm 7.353987098249e-03 1819.45 s.
Main iteration 600 KSP Residual norm 7.329169368963e-03 1893.75 s.
Main iteration 700 KSP Residual norm 7.280724909714e-03 1972.89 s.
Main iteration 800 KSP Residual norm 7.219377355706e-03 2053.55 s.
Main iteration 900 KSP Residual norm 7.142154162747e-03 2137.83 s.
Main iteration 1000 KSP Residual norm 7.087758297414e-03 2225.20 s.
Main iteration 1100 KSP Residual norm 7.021461405432e-03 2317.34 s.
Main iteration 1200 KSP Residual norm 6.976567477442e-03 2412.40 s.
Main iteration 1300 KSP Residual norm 6.941950047016e-03 2510.51 s.
Main iteration 1400 KSP Residual norm 6.895235261259e-03 2612.37 s.
Main iteration 1500 KSP Residual norm 6.823520923884e-03 2717.90 s.
Main iteration 1600 KSP Residual norm 6.770539897469e-03 2827.06 s.
Main iteration 1700 KSP Residual norm 6.721366458903e-03 2938.63 s.
Main iteration 1800 KSP Residual norm 6.642427851340e-03 3054.27 s.
Main iteration 1900 KSP Residual norm 6.576821855539e-03 3172.17 s.
Main iteration 2000 KSP Residual norm 6.534695486253e-03 3293.78 s.
Main iteration 2100 KSP Residual norm 6.534695486219e-03 3352.47 s.
Main iteration 2200 KSP Residual norm 6.534695485823e-03 3414.63 s.
Main iteration 2300 KSP Residual norm 6.534695483527e-03 3479.14 s.
Main iteration 2400 KSP Residual norm 6.534695479829e-03 3548.68 s.
Main iteration 2500 KSP Residual norm 6.534695474640e-03 3619.43 s.
Main iteration 2600 KSP Residual norm 6.534695470102e-03 3694.80 s.
Main iteration 2700 KSP Residual norm 6.534695458409e-03 3771.49 s.
Main iteration 2800 KSP Residual norm 6.534695440378e-03 3853.94 s.
Main iteration 2900 KSP Residual norm 6.534695409617e-03 3938.93 s.
Main iteration 3000 KSP Residual norm 6.534695375298e-03 4027.51 s.
Main iteration 3100 KSP Residual norm 6.534695224378e-03 4119.35 s.
Main iteration 3200 KSP Residual norm 6.534694981088e-03 4213.97 s.
Main iteration 3300 KSP Residual norm 6.534694387186e-03 4310.25 s.
Main iteration 3400 KSP Residual norm 6.534692216359e-03 4411.79 s.
Main iteration 3500 KSP Residual norm 6.534686956837e-03 4516.31 s.
Main iteration 3600 KSP Residual norm 6.534661972040e-03 4624.64 s.
Main iteration 3700 KSP Residual norm 6.534576351206e-03 4735.26 s.
Never converged
Onera_M6_Wing-SnappyHexMesh- All default tutorial settings
Solver Type: gmres
GMRES Restart: 1000
ASM Overlap: 1
Global PC Iters: 0
Local PC Iters: 1
Mat ReOrdering: natural
ILU PC Fill Level: 1
GMRES Max Iterations: 1000
GMRES Relative Tolerance: 0.0001
GMRES Absolute Tolerance: 1e-14
Solving Linear Equation... 947.41 s
Main iteration 0 KSP Residual norm 1.014975657480e-02 1018.56 s.
Main iteration 100 KSP Residual norm 1.014975657477e-02 1060.45 s.
Main iteration 200 KSP Residual norm 1.014975657477e-02 1106.66 s.
Main iteration 300 KSP Residual norm 1.014975657477e-02 1152.74 s.
Main iteration 400 KSP Residual norm 1.014975657477e-02 1201.70 s.
Main iteration 500 KSP Residual norm 1.014975657477e-02 1252.69 s.
Main iteration 600 KSP Residual norm 1.014975657477e-02 1306.30 s.
Main iteration 700 KSP Residual norm 1.014975657477e-02 1362.71 s.
Main iteration 800 KSP Residual norm 1.014975657477e-02 1421.38 s.
Main iteration 900 KSP Residual norm 1.014975657477e-02 1482.83 s.
Main iteration 1000 KSP Residual norm 1.014975657477e-02 1546.47 s.
Main iteration 1000 KSP Residual norm 1.014975657477e-02 1546.74 s
Completed! Total iterations: 1000. PetscConvergedReason: -3. 1546.74 s
Residual tolerance not satisfied, solution failed!
the default tutorial wasnt converging, this was wth SnappyHexMesh
Then i switch the fvSchemes to upwind for div(phi,U), div(phid,p) with ILU. ASM - 2
I dont which one helped when incresing the ILU or switching to upwind but it worked. Just wanted to mention this here. if someone new tries this it would help.
GMRES Restart: 1000
ASM Overlap: 1
Global PC Iters: 0
Local PC Iters: 1
Mat ReOrdering: natural
ILU PC Fill Level: 2
GMRES Max Iterations: 1000
GMRES Relative Tolerance: 0.0001
GMRES Absolute Tolerance: 1e-14
Solving Linear Equation... 197.15 s
Main iteration 0 KSP Residual norm 1.014469015953e-02 690.90 s.
Main iteration 63 KSP Residual norm 9.082773158709e-07 722.90 s
Completed! Total iterations: 63. PetscConvergedReason: 2. 722.9 s
Residual tolerance satisfied, solution finished!
Computing d[aero_residuals]/d[aero_vol_coords]^T * psi 723.0599999999999 s
Computing d[aero_residuals]/d[patchV]^T * psi 725.47 s
Computing d[CL]/d[aero_states]^T * psi 726.55 s
Computing d[CL]/d[aero_vol_coords]^T * psi 726.63 s
Computing d[CL]/d[patchV]^T * psi 727.13 s
Latest solution time 0.0001 less than 1, not renamed.
Solving Linear Equation... 727.2 s
Main iteration 0 KSP Residual norm 1.039898147074e-01 727.21 s.
Main iteration 67 KSP Residual norm 8.235637892428e-06 756.37 s
Completed! Total iterations: 67. PetscConvergedReason: 2. 756.37 s
Residual tolerance satisfied, solution finished!
When i tried the pyHyp version ( the same issue with pyHyp in the HPC ) the primal seems to work well so i didnt suspect the mesh might be the issue but it can also be the case.
Onera_M6_Wing-PyHyp - All Default settings ran on 20 CPU in one node with 64GB Ram
Solver Type: gmres
GMRES Restart: 1000
ASM Overlap: 1
Global PC Iters: 0
Local PC Iters: 1
Mat ReOrdering: natural
ILU PC Fill Level: 1
GMRES Max Iterations: 1000
GMRES Relative Tolerance: 0.0001
GMRES Absolute Tolerance: 1e-14
Solving Linear Equation... 2279.35 s
Primary job terminated normally, but 1 process returned
a non-zero exit code. Per user-direction, the job has been aborted.
mpiexec noticed that process rank 14 with PID 195195 on node node143 exited on signal 9 (Killed).
Onera_M6_Wing-PyHyp - All Default settings ran on 20 CPU in 2 node with 64GB Ram each
Solver Type: gmres
GMRES Restart: 1000
ASM Overlap: 1
Global PC Iters: 0
Local PC Iters: 1
Mat ReOrdering: natural
ILU PC Fill Level: 1
GMRES Max Iterations: 1000
GMRES Relative Tolerance: 0.0001
GMRES Absolute Tolerance: 1e-14
Solving Linear Equation... 587.38 s
Primary job terminated normally, but 1 process returned
a non-zero exit code. Per user-direction, the job has been aborted.
mpiexec noticed that process rank 7 with PID 0 on node node179 exited on signal 9 (Killed).
slurmstepd: error: Detected 2 oom-kill event(s) in StepId=3370.batch. Some of your processes may have been killed by the cgroup out-of-memory handler.
Both ran out of memory - for 37k cells I used 64GB , 128GB ram respectively.
DPW4 case works fine so i dont think I have a major compilation issues
DPW4 - Default fvSchemes but used nd and ASM and ILU 2 - works fine
GMRES Restart: 2000
ASM Overlap: 2
Global PC Iters: 0
Local PC Iters: 1
Mat ReOrdering: nd
ILU PC Fill Level: 2
GMRES Max Iterations: 6000
GMRES Relative Tolerance: 1e-05
GMRES Absolute Tolerance: 1e-14
Solving Linear Equation... 2343.61 s
Main iteration 0 KSP Residual norm 6.882407371012e-03 2645.09 s.
Main iteration 100 KSP Residual norm 4.477090036643e-03 2724.56 s.
Main iteration 200 KSP Residual norm 2.242989304830e-04 2807.99 s.
Main iteration 300 KSP Residual norm 7.529208022692e-06 2895.35 s.
Main iteration 400 KSP Residual norm 1.617210552715e-06 2986.51 s.
Main iteration 500 KSP Residual norm 1.341288205115e-06 3082.02 s.
Main iteration 600 KSP Residual norm 4.675592890766e-07 3181.49 s.
Main iteration 700 KSP Residual norm 3.137283790770e-07 3285.51 s.
Main iteration 800 KSP Residual norm 1.672251865516e-07 3393.74 s.
Main iteration 859 KSP Residual norm 6.755619671626e-08 3460.91 s
Completed! Total iterations: 859. PetscConvergedReason: 2. 3460.91 s
Residual tolerance satisfied, solution finished!
Computing d[aero_residuals]/d[aero_vol_coords]^T * psi 3461.53 s
Computing d[aero_residuals]/d[patchV]^T * psi 3463.62 s
Computing d[CL]/d[aero_states]^T * psi 3469.22 s
Computing d[CL]/d[aero_vol_coords]^T * psi 3469.42 s
Computing d[CL]/d[patchV]^T * psi 3470.22 s
Latest solution time 0.0001 less than 1, not renamed.
Solving Linear Equation... 3470.45 s
Main iteration 0 KSP Residual norm 3.996523193994e-02 3470.47 s.
Main iteration 100 KSP Residual norm 3.103135466961e-02 3551.77 s.
Main iteration 200 KSP Residual norm 2.611675097729e-03 3636.96 s.
Main iteration 300 KSP Residual norm 9.191504166849e-05 3725.99 s.
Main iteration 400 KSP Residual norm 3.395599927045e-05 3819.15 s.
Main iteration 500 KSP Residual norm 2.631575968070e-05 3916.70 s.
Main iteration 600 KSP Residual norm 1.173080812103e-05 4018.36 s.
Main iteration 700 KSP Residual norm 2.975987797344e-06 4124.77 s.
Main iteration 788 KSP Residual norm 3.933333154229e-07 4221.52 s
Completed! Total iterations: 788. PetscConvergedReason: 2. 4221.52 s
Residual tolerance satisfied, solution finished!
Computing d[aero_residuals]/d[aero_vol_coords]^T * psi 4222.05 s
Computing d[aero_residuals]/d[patchV]^T * psi 4223.7 s
Computing d[CMY]/d[aero_states]^T * psi 4228.71 s
Computing d[CMY]/d[aero_vol_coords]^T * psi 4229.24 s
Computing d[CMY]/d[patchV]^T * psi 4230.87 s
Latest solution time 0.0001 less than 1, not renamed.
Solving Linear Equation... 4231.08 s
Main iteration 0 KSP Residual norm 8.582237831085e-02 4231.10 s.
Main iteration 100 KSP Residual norm 5.327334512464e-02 4311.49 s.
Main iteration 200 KSP Residual norm 1.098839628518e-02 4396.74 s.
Main iteration 300 KSP Residual norm 2.673713515769e-04 4485.96 s.
Main iteration 400 KSP Residual norm 5.143336365822e-05 4579.55 s.
Main iteration 500 KSP Residual norm 4.118456647598e-05 4677.22 s.
Main iteration 600 KSP Residual norm 2.648278681934e-05 4779.95 s.
Main iteration 700 KSP Residual norm 7.289332476310e-06 4885.94 s.
Main iteration 793 KSP Residual norm 8.563865965162e-07 4989.15 s
Completed! Total iterations: 793. PetscConvergedReason: 2. 4989.15 s
Residual tolerance satisfied, solution finished!
Computing d[aero_residuals]/d[aero_vol_coords]^T * psi 4989.71 s
Computing d[aero_residuals]/d[patchV]^T * psi 4991.36 s
I made my own case to converge too but it takes a lot of time to converge.
GMRES Restart: 2000
ASM Overlap: 2
Global PC Iters: 0
Local PC Iters: 1
Mat ReOrdering: nd
ILU PC Fill Level: 2
GMRES Max Iterations: 6000
GMRES Relative Tolerance: 1e-05
GMRES Absolute Tolerance: 1e-14
Solving Linear Equation... 438.59 s
Main iteration 0 KSP Residual norm 4.627779785159e-02 547.81 s.
Main iteration 100 KSP Residual norm 4.297427553024e-02 598.57 s.
Main iteration 200 KSP Residual norm 4.044975505490e-02 650.89 s.
Main iteration 300 KSP Residual norm 4.007992573566e-02 704.73 s.
Main iteration 400 KSP Residual norm 3.961851191266e-02 761.95 s.
Main iteration 500 KSP Residual norm 3.925206474714e-02 823.51 s.
Main iteration 600 KSP Residual norm 3.892740846860e-02 886.00 s.
Main iteration 700 KSP Residual norm 3.862389265612e-02 951.54 s.
Main iteration 800 KSP Residual norm 3.735477939942e-02 1020.14 s.
Main iteration 900 KSP Residual norm 3.283145453496e-02 1092.43 s.
Main iteration 1000 KSP Residual norm 1.691081147286e-02 1165.93 s.
Main iteration 1100 KSP Residual norm 1.424811440751e-02 1244.88 s.
Main iteration 1200 KSP Residual norm 7.971463026040e-03 1326.10 s.
Main iteration 1300 KSP Residual norm 5.785497525780e-03 1409.17 s.
Main iteration 1400 KSP Residual norm 3.719856486417e-03 1494.84 s.
Main iteration 1500 KSP Residual norm 3.491911425905e-03 1583.66 s.
Main iteration 1600 KSP Residual norm 2.212460759752e-03 1677.30 s.
Main iteration 1700 KSP Residual norm 1.220821391440e-03 1772.56 s.
Main iteration 1800 KSP Residual norm 9.299593945216e-04 1869.56 s.
Main iteration 1900 KSP Residual norm 9.058616468088e-04 1969.51 s.
Main iteration 2000 KSP Residual norm 6.860726923258e-04 2073.84 s.
Main iteration 2100 KSP Residual norm 6.857984643318e-04 2122.35 s.
Main iteration 2200 KSP Residual norm 6.845706124816e-04 2175.28 s.
Main iteration 2300 KSP Residual norm 6.827801823044e-04 2230.25 s.
Main iteration 2400 KSP Residual norm 6.820637592422e-04 2288.19 s.
Main iteration 2500 KSP Residual norm 6.798229099414e-04 2350.38 s.
Main iteration 2600 KSP Residual norm 6.735072550497e-04 2415.25 s.
Main iteration 2700 KSP Residual norm 6.701328861451e-04 2482.53 s.
Main iteration 2800 KSP Residual norm 6.643231034277e-04 2553.19 s.
Main iteration 2900 KSP Residual norm 6.575963567621e-04 2624.94 s.
Main iteration 3000 KSP Residual norm 6.480460373426e-04 2700.54 s.
Main iteration 3100 KSP Residual norm 6.364503417456e-04 2777.95 s.
Main iteration 3200 KSP Residual norm 5.801047480017e-04 2859.66 s.
Main iteration 3300 KSP Residual norm 5.616929820161e-04 2942.08 s.
Main iteration 3400 KSP Residual norm 3.843751054285e-04 3029.00 s.
Main iteration 3500 KSP Residual norm 2.498082809234e-04 3118.39 s.
Main iteration 3600 KSP Residual norm 1.597154370664e-04 3210.71 s.
Main iteration 3700 KSP Residual norm 1.060344700378e-04 3306.47 s.
Main iteration 3800 KSP Residual norm 8.351047562573e-05 3404.86 s.
Main iteration 3900 KSP Residual norm 6.010383825797e-05 3506.79 s.
Main iteration 4000 KSP Residual norm 3.747821399429e-05 3611.41 s.
Main iteration 4100 KSP Residual norm 3.741542550164e-05 3663.06 s.
Main iteration 4200 KSP Residual norm 3.716349669959e-05 3717.45 s.
Main iteration 4300 KSP Residual norm 3.709758512119e-05 3774.19 s.
Main iteration 4400 KSP Residual norm 3.676696627831e-05 3833.61 s.
Main iteration 4500 KSP Residual norm 3.664371155861e-05 3895.58 s.
Main iteration 4600 KSP Residual norm 3.646659191785e-05 3959.47 s.
Main iteration 4700 KSP Residual norm 3.618258050409e-05 4026.42 s.
Main iteration 4800 KSP Residual norm 3.613653254141e-05 4096.43 s.
Main iteration 4900 KSP Residual norm 3.599632663952e-05 4167.29 s.
Main iteration 5000 KSP Residual norm 3.579278370329e-05 4244.25 s.
Main iteration 5100 KSP Residual norm 3.445441961804e-05 4321.02 s.
Main iteration 5200 KSP Residual norm 2.988696686892e-05 4401.16 s.
Main iteration 5300 KSP Residual norm 2.431664348810e-05 4486.21 s.
Main iteration 5400 KSP Residual norm 1.812248326263e-05 4573.12 s.
Main iteration 5500 KSP Residual norm 1.569776476815e-05 4664.52 s.
Main iteration 5600 KSP Residual norm 1.497439974401e-05 4756.98 s.
Main iteration 5700 KSP Residual norm 1.425306972007e-05 4852.40 s.
Main iteration 5800 KSP Residual norm 1.178476098808e-05 4950.53 s.
Main iteration 5900 KSP Residual norm 8.895587702316e-06 5050.87 s.
Main iteration 6000 KSP Residual norm 5.784482010810e-06 5155.40 s.
Main iteration 6000 KSP Residual norm 5.784482010810e-06 5155.71 s
Completed! Total iterations: 6000. PetscConvergedReason: -3. 5155.71 s
Residual tolerance satisfied, solution finished!
barely convegred the Tol which I prescribed wasnt met
Can someone please confirm whether its my HPC or mesh issue or there is a actual problem in the tutorial.
Also, Any inputs on how to speedup the Adjoint convergence would be appreciated. I tried all the things mentioned in forum and also on the website and I dont know whether i should consider trying yplus above 60. but I havent tried above 45.
Thanks in advance!
All reactions