Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

PyAMG Error during SU2-PyAMG run #948

Closed
aydinmirac opened this issue Apr 24, 2020 · 51 comments
Closed

PyAMG Error during SU2-PyAMG run #948

aydinmirac opened this issue Apr 24, 2020 · 51 comments

Comments

@aydinmirac
Copy link

Hello everyone,

We are trying to use SU2-pyamg on the server.
We followed the installation steps on SU2 website and did not face any compilation error.

Steps we followed:

  1. Download the file with git ("Install a custom version of SU2" section in https://pyamg.saclay.inria.fr/pyamg.html)
  2. cd SU2
  3. ./bootstrap
  4. ./configure --prefix=$HOME/SU2 --enable-mpi --with-cc=/opt/intel/compilers_and_libraries_2019.5.281/linux/mpi/intel64/bin/mpicc --with-cxx=/opt/intel/compilers_and_libraries_2019.5.281/linux/mpi/intel64/bin/mpicxx CXXFLAGS="-O3"
  5. make -j 8
  6. make install

Then we added the following lines to bashrc file:

export PYTHONPATH="/usr/lib64/python2.7/site-packages"
export SU2_RUN="$HOME/su2-installed/bin"
export SU2_HOME="$HOME/SU2"
export PATH=$PATH:$SU2_RUN
export PYTHONPATH=$PYTHONPATH:$SU2_RUN

We downloaded example file from pyamg website:
https://pyamg.saclay.inria.fr/pyamgexamples.html (Adaptive 3D ONERA M6 wing example)

During running, we encountered the following error:

$HOME/su2-installed/bin/mesh_adaptation_amg.py -f adap_ONERAM6.cfg -n 4 > out.txt &
[1] 166739
[user@host pyamg-su-test]$ Traceback (most recent call last):
File "/home/eduline/su2-installed/bin/mesh_adaptation_amg.py", line 111, in
main()
File "/home/eduline/su2-installed/bin/mesh_adaptation_amg.py", line 72, in main
options.save_all )
File "/home/eduline/su2-installed/bin/mesh_adaptation_amg.py", line 101, in mesh_adaptation_amg
SU2.run.amg(config)
File "/home/eduline/su2-installed/bin/SU2/run/amg.py", line 212, in amg
pyamg.su2_to_libmeshb(config_amg['adap_back'], "", "amg_back")
UnboundLocalError: local variable 'pyamg' referenced before assignment

Python version is 2.7.5 and we installed pyamg by running "sudo pip install pyamg".

Any help would be appreciated.

Best regards,

Mirac
@bmunguia
Copy link
Member

bmunguia commented Apr 24, 2020 •
edited
Loading

Hi Mirac,

The pyamg website is based on an fork of Vic's that's very out of date. The branch feature_adap handles all the pyamg and GMF I/O library installation if you just follow the usual installation instructions (either using preconfigure.py or the meson scripts) in the v7 instructions. Just keep in mind that it's a bit experimental still. The most up-to-date version of pyamg is included, but note that it requires python 3.7 or greater.

Brian

@antares190
Copy link

Hi @bmunguia ,
I was in the same situation as @miracaydin1, so thanks for the hint. Nevertheless I'm having some problems. I followed your instructions to install the branch:

  1. downloaded the feature_adapt_sst branch.
  2. Used meson to configure the build
    ./meson.py build -Denable-autodiff=true -Denable-directdiff=true
  3. Added the environment variables to the .bashrc
  4. ninja build.
    ./ninja -C build install

The build didn't show any errors, only few warnings during meson ( gcc1: warning: command line option ‘-Wno-non-virtual-dtor’ is valid for C++/ObjC++ but not for C).

When I try to run the mesh_adaption (or even run simply the solver through parallel_computation.py):

$SU2_RUN/mesh_adaptation_amg.py -f invCyl.cfg -n 6

It immediately shows an error (without even showing the SU2 splash screen) :
Traceback (most recent call last):
File "/usr/local/bin/mesh_adaptation_amg.py", line 38, in <module>
import SU2
File "/usr/local/bin/SU2/__init__.py", line 14, in <module>
from SU2 import amginria
File "/usr/local/bin/SU2/amginria/__init__.py", line 4, in <module>
from .interface import *
File "/usr/local/bin/SU2/amginria/interface.py", line 41, in <module>
import _amgio as amgio
ImportError: No module named _amgio

Am I missing any dependencies?
Please note that I previously compiled SU2 master branch without issues, and have already installed mpich, numpy, scipy.
Any help would be really appreciated.
Thanks

@bmunguia
Copy link
Member

The _amgio extension should be built by default. I haven't had any issues on any of the machines I've built on, so I appreciate the feedback.

Just curious, are you running with python >=3.7? The shebang in all the SU2 python scripts

#!/usr/bin/env python

uses whatever python is set to in your environment variables, but pyamg/_amgio will only build/run with python >=3.7. Could you also check if the _amgio extension was installed in your site-packages (probably located in ~/.local/lib/python3.x/site-packages)?

If it didn't build/install, you could go into extensions/AMGIO/su2io and run

python3 setup.py build_ext && python3 setup.py install

If it did build/install, try the command

python3 $SU2_RUN/mesh_adaptation_amg.py -f invCyl.cfg -n 6

@antares190
Copy link

antares190 commented Apr 25, 2020 •
edited
Loading

Hi @bmunguia ,
thanks for the fast reply, I'm sorry, the Python version was 3.6. I installed the 3.7.5, and made sure to reconfigure (had an issue with the gnome-terminal, but solved).
I re-compiled everything, but unfortunately the error persists, so I checked .local/python3.7.5./site-packages/pyamg but no _amgio extension was there.
When I move in external/AMGIO/su2io/ and run :
python3 setup.py build_ext && python3 setup.py install
An error occours:

running build_ext
building '_amgio' extension
swigging ./sources/Python3.7/amgio_py.i to ./sources/Python3.7/amgio_py_wrap.c
swig -python -o ./sources/Python3.7/amgio_py_wrap.c ./sources/Python3.7/amgio_py.i
x86_64-linux-gnu-gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -g -fwrapv -O2 -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -I/usr/include/python3.7m -c ./sources/Python3.7/amgio_py.c -o build/temp.linux-x86_64-3.7/./sources/Python3.7/amgio_py.o -std=c99 -Wno-unused-variable -Wno-unused-result -O3
./sources/Python3.7/amgio_py.c:2:10: fatal error: Python.h: No such file or directory
 #include "Python.h"
          ^~~~~~~~~~
compilation terminated.
error: command 'x86_64-linux-gnu-gcc' failed with exit status 1

Shall I remove python2.7 as well?
Thanks

During compilation the following warning appears:

cc1: warning: command line option ‘-Wno-non-virtual-dtor’ is valid for C++/ObjC++ but not for C
../externals/AMGIO/sources/libmeshb7.c: In function ‘GmfGetLinTab’:
../externals/AMGIO/sources/libmeshb7.c:1391:35: warning: format ‘%lld’ expects argument of type ‘long long int *’, but argument 3 has type ‘int64_t * {aka long int *}’ [-Wformat=]
             safe_fscanf(msh->hdl, "%lld", &IntTab[ (*IntCpt)++ ], msh->err);
                                   ^       ~~~~~~~~~~~~~~~~~~~~~~
../externals/AMGIO/sources/libmeshb7.c:475:23: note: in definition of macro ‘safe_fscanf’
       if( fscanf(hdl, format, ptr) != 1 ) \
                       ^~~~~~
../externals/AMGIO/sources/libmeshb7.c: In function ‘GmfSetLinTab’:
../externals/AMGIO/sources/libmeshb7.c:1462:35: warning: format ‘%lld’ expects argument of type ‘long long int’, but argument 3 has type ‘int64_t {aka long int}’ [-Wformat=]
             fprintf(msh->hdl, "%lld ", LngTab[ LngCpt++ ]);
                                ~~~^    ~~~~~~~~~~~~~~~~~~
                                
%ld

Moreover later on the script (even if it asks me for permissions, and I provide the password):


Installing _amgio.
Installation failed due to insufficient permissions.
Attempting to use polkit to gain elevated privileges...
Installing _amgio.
Installing pyAMG for Linux.
pyAMG installation failed

@bmunguia
Copy link
Member

I just updated the build to use the --user flag so it shouldn't require permissions. Can you pull and try installing again? Just run the same meson command as before with --reconfigure

As for the missing header, I think that's an issue with your python setup. Don't remove python2.7 from your system, since a lot of packages still depend on it. How did you install python? If you run locate Python.h does it return anything?

@antares190
Copy link

Thanks for the commit, I've installed it using your hints (I also changed some aliases to always point to python3), and now parallel_computation.py it is running properly.
mesh_adaption_amg.py runs, until it complains about Ncorners in the SU2 mesh.
## WARNING: MISSING SU2 MESH FILE KEYWORD: NCORNERS=.
and then mesh_adapt fails, I can't find any reference to NCORNERS in *.su2 mesh files by the way.

As far as locate Python.h that's the output :

/home/antares/Downloads/SALOME-9.3.0-UB18.04-SRC/BINARIES-UB18.04/Python/include/python3.6/Python.h
/home/antares/Downloads/SALOME-9.3.0-UB18.04-SRC/BINARIES-UB18.04/gmsh/include/FieldPython.h
/home/antares/Downloads/SALOME-9.3.0-UB18.04-SRC/BINARIES-UB18.04/gmsh/include/simpleFunctionPython.h
/home/antares/Downloads/ThirdParty-v1806/ParaView-v5.5.2/VTK/Utilities/Python/vtkPython.h
/home/antares/OpenFOAM/ThirdParty-v1806/ParaView-v5.5.2/VTK/Utilities/Python/vtkPython.h
/usr/include/python3.6m/Python.h

Thanks

@bmunguia
Copy link
Member

bmunguia commented Apr 25, 2020 •
edited
Loading

The corners warning is fine. I added the ability to specify corner points for sharp features that the user wants to maintain. For example, in an airfoil test case I want to make sure the sharp trailing edge at node 512 is maintained, so I can manually add

NCORNERS= 1
1 512

to the end of my mesh. It's not necessary, though, and AMG performs ridge detection to identify any features anyway.

I assume since it built and (kinda) ran that you managed to fix your python setup through your aliases?

What was the specific error that you got? I probably should have also mentioned that I've changed a lot of options since updating from Vic's old code. Here are the list of required options (values from my test case):

% Type of sensor used for adaptation
% Options include GOAL (adjoint-based), MACH, PRES
PYADAP_SENSOR= GOAL
% Objective function used for goal-oriented adaptation
OBJECTIVE_FUNCTION= LIFT
% Maximum cell size for adaptation
PYADAP_HMAX= 500.0
% Minimum cell size for adaptation
PYADAP_HMIN= 1e-9
% Gradation factor (typically 1.2-1.8)
PYADAP_HGRAD= 1.8
% Norm used for adaptation
% 1.0 or 2.0 recommended for inviscid flows
% 4.0 recommeneded for viscid
ADAP_NORM= 4.0
% Approximate mesh size (NPOI) at each level
PYADAP_COMPLEXITY= (30000, 60000, 120000)
% Number of adaptations performed at each level
PYADAP_SUBITE= (3, 3, 3)

Some optional options:

% Numerical method for Hessian reconstruction in goal-oriented adaptation
% Currently only Green-Gauss
NUM_METHOD_HESS= GREEN_GAUSS
% Value for ITER for the flow solver at each level
PYADAP_FLOW_ITER= (2999, 4999, 6999)
% Value for ITER for the adjoint solver at each level
PYADAP_ADJ_ITER= (2999, 4999, 6999)
% Value for CFL_NUMBER for the adjoint solver at each level
PYADAP_CFL= (10.0, 15.0, 20.0)
% Background mesh containing fine surface representation
% Can be either a .su2 or .meshb (GMF) file
% If not specified, the baseline mesh is used
PYADAP_BACK= rae2822_euler_40960_split.meshb
% Invert the background mesh orientation
% Sometimes required for 3D meshes, since AMG doesn't perform
% a check that the orientation of the background and foreground
% meshes match
PYADAP_INV_BACK= NO
% Don't use ridge detection if corners specified
PYADAP_RDG= NO

@bmunguia
Copy link
Member

bmunguia commented Apr 25, 2020 •
edited
Loading

FYI, just so I can keep debugging the RANS adaptation in feature_adap_sst, I've merged it into feature_adap. You should use feature_adap instead from here on out since feature_adap_sst will be pretty volatile. I'll modify my comment above in case anybody references this thread.

@antares190
Copy link

Thanks for the reply, I'll use feature_adap. Also very appreciated the required and optional command list.
As far as the run is concerned, it exits during the first mesh adaption (no error on the console though). The amg.out is quite long and there are a lot of info.

Let me briefly summarize the case I'm testing:

  • Supersonic inviscid 2D axialsymmetric cylinder.
  • mesh adaption Mach based.
  • no shape optimization.
  • mesh adaption (PYADAP_FLOW_ITER= (2999, 4999, 6999))
  • coarse mesh (PYADAP_COMPLEXITY= (20000, 60000, 120000))
  • ITER = 10000.

I would expect then the solver to start, reach iteration number 2999, adapt the flow mesh, then run until 4999, adapt, 6999 adapt, reach 10000 iterations.
In the :
adapt/ini/log.out
as far as I can tell it reaches 10000 iterations, which is quite unexpected, maybe my problem arise due to mismatching options in the *.cfg file, is there any applicable *.cfg file to start from?

@bmunguia
Copy link
Member

Yeah the config is a bit confusing. The initial CFD run (in adap/ini) uses the value specified in ITER (unless RESTART_SOL= YES, then it runs one iteration just to get force coefficients).

After that, it runs a series of adaptations based on the settings in PYADAP_SUBITE, PYADAP_FLOW_ITER, and PYADAP_COMPLEXITY.

  1. AMG interpolates the solution
  2. A simulation is run for PYADAP_FLOW_ITER[0] iterations (2999 in your case)
  3. An adaptation is performed with a target size of PYADAP_COMPLEXITY[0] (20000 in your case)

This is done PYADAP_SUBITE[0] times. Then it moves on to the next mesh level.

I admit that I haven't really debugged the feature-based (e.g. Mach-based) adaptation much. I found a couple bugs in the python scripts and just updated them, so could you pull again and rerun ninja?

The AMG output is pretty long, but since it's developed at Inria, I don't have any control over that. I guess as long as none of the lines are error statements (usually complaining about invalid projections or negative eigenvalues of the metric from my experience) it should be fine. The only actual information I get from amg.log is usually in the last line

NbSensor 30518 msh->npobk 30518 NbrTyp 19 msh->npoin 52477

which just tells you how many points are in the old vs. new mesh, but you could just look at the .su2 files for that.

Another important note is that the python version of AMG currently only supports triangle and tetrahedral volume elements.

@antares190
Copy link

Sorry for the delay, I managed to clone the feature_adap branch, and updated it.

Now during the computation I think it manages to go through the first adap, and creates the ite0 folder so basically my understanding is that right now it is working (still have some convergence problems).

Is this branch providing similar convergence rates as SU2 7.0.3 ?

Thank you very much for the support!

@bmunguia
Copy link
Member

It's basically 7.0.3, but I've modified the gradient correction, viscous flux Jacobian, and pretty much the entire SST solver. Are you using any of those? If you give me more info about your case and the convergence problems, I can look into it

@antares190
Copy link

Thanks Brian,
my bad, I misinterpreted the carpets / logs structure. I successfully ran a test of a very simple case. In the results there are some small artifacts, but the shocks are resolution improved consistently after mesh adaption.
Thanks for the support

Comparison
ComparisonMesh

@juanjosealonso
Copy link
Member

juanjosealonso commented Apr 29, 2020 via email

@antares190
Copy link

Thanks for the hint, just uploaded a screenshot of the mesh adaption!
Now I'm trying to run a simple 3D case, but unfortunately the amg.out fails during the first try (in the adap/ini folder):

  ## 16739 TRIANGLE(S) DISCARDED 
 fefloa_Python2Mesh : 3d mesh on input 
 fefloa_Python2Mesh : msh->NbrVer 14368 
 fefloa_Python2Mesh : msh->NbrTet 58189 
 fefloa_Python2Mesh : leaving with 14368 ver. 16740 tri. 0 edg. 
  -- Maximal memory 
  Maximum number of Points                  53800000 
  Maximun number of Bnd Points              10760000
  Maximum number of Triangles               21520000 
  Maximum number of Tetrahedra             295900000 
  Allocated Memory 50.105 Gb 
  Physical Memory  62.729 Gb
  bounding box x: -100 200 y: -150 150 z: -150 150 
  ## WARNING: REMOVING DUPLICATED FACE 12357 12359 12358 
  ## WARNING: REMOVING DUPLICATED FACE 12357 12359 12358 
  ## WARNING: REMOVING DUPLICATED FACE 12357 12359 12358 
  ## WARNING: REMOVING DUPLICATED FACE 12357 12359 12358 
  ## WARNING: REMOVING DUPLICATED FACE 12357 12359 12358 
  ## WARNING: REMOVING DUPLICATED FACE 12357 12359 12358 
  ## WARNING: REMOVING DUPLICATED FACE 12357 12359 12358 
  ## WARNING: REMOVING DUPLICATED FACE 12357 12359 12358 
  ## WARNING: REMOVING DUPLICATED FACE 12357 12359 12358 
  ## WARNING: REMOVING DUPLICATED FACE 12357 12359 12358 
  ## WARNING: REMOVING DUPLICATED FACE 12357 12359 12358 
  ## WARNING: REMOVING DUPLICATED FACE 12357 12359 12358 
  ## WARNING: REMOVING DUPLICATED FACE 12357 12359 12358 
  ## WARNING: REMOVING DUPLICATED FACE 12357 12359 12358 
  ## WARNING: REMOVING DUPLICATED FACE 12357 12359 12358 
       14368 Vertex 
           3 Boundary Vertices   
       58189 Tetrahedra   
           1 Triangles  
           0 Edges  
           1 Surface(s) 
  % Output subdom.meshb Mesh ndimn=3, mpoin=14368, melem=58189, mface=1
 Corners 0 Required 0 
   ## ERROR : TETRAHEDRON             2  BOUNDARY FACE            1           2
           9  IS MISSING 
   ## ERROR : TETRAHEDRON             4  BOUNDARY FACE            3           7
        2324  IS MISSING 
···
  ## ERROR: FEFLO.A: 16740 BOUNDARY FACES ARE MISSING 
  CHECK YOUR MESH 

  ## FATAL ERROR: FEFLOA: INITIAL OR CURRENT MESH HAS AN INCONSISTANT TOPOLOGY 

  Thank you for using feflo.a

Which in turn creates an invalid amg.su2 mesh in the adap/ite0 folder, which triggers the fault.
I'm a bit puzzled as the domain is very simple, and the log.out is going well (with no complains of SU2 about any negative volume or trias with wrong normals). The mesh is only made of TRIAS and TETRAS, it should work in principle.
Am I missing something in the setup / mesh? I know you have little control on the amg.out, but maybe there some special hint that I'm missing.

Btw the final error in the terminal is but I think the problem arises before reaching flo.csv file:

Traceback (most recent call last):
  File "/usr/local/bin/mesh_adaptation_amg.py", line 111, in <module>
    main()
  File "/usr/local/bin/mesh_adaptation_amg.py", line 72, in main
    options.save_all    )
  File "/usr/local/bin/mesh_adaptation_amg.py", line 101, in mesh_adaptation_amg
    SU2.run.amg(config)
  File "/usr/local/bin/SU2/run/amg.py", line 464, in amg
    os.rename(cur_solfil, cur_solfil_ini)
FileNotFoundError: [Errno 2] No such file or directory: 'flo.csv' -> 'flo_ini.csv'

Thanks!

@bmunguia
Copy link
Member

bmunguia commented May 6, 2020

I just tested the 3D adaptation on an ONERA M6 and it seems to be working fine for me.

Screen Shot 2020-05-06 at 2 05 56 PM

I just updated AMGIO to include surface/volume element orientation checks, not sure if that'll help with your issue.

If the updated code doesn't fix things, can you upload your initial mesh, flow restart, and the flo.meshb and flo.solb files from adap/ini?

@antares190
Copy link

Hi Brian,
I'm sorry for the late reply, I've been using the feature_adap branch in 2D and 3D with success in the last weeks, so thanks for the amazing work.

After your last commit I recompiled everything, moreover now I'm using $SU2_RUN/SU2_DEF before running mesh_adaption_amg, so to be sure that element orientation is as expected.

The tool is working nice, once the smoothing, grad, adaption values are tuned, thank you very much!

After a bit of use may I add pair of feature request, that in my opinion would improve the overall (already remarkable) quality of the feature_adap .
Feature request:

  1. The possibility to have boundary layer elements (QUAD PENTA) with the mesh adaption would be very much appreciated. If not possible, wouldn't it be possible to force a frozen interface between boundary layer elements?
  2. Add a "region of interest" where the refinement takes place. Sometimes it is not required to have sharp shocks resolved far away from the body.
  3. Do you plan to merge this feature in the master branch?

Bugs (or at least unexpected behavior):

  1. When I use the restart option, even if there is a restart_flow.dat, in the base folder, apparently the mesh_adaption_amg does not create a link into the \adap\ini\ folder, and I've to manually copy it (quite fast), to effectively enable the restart option.

Clarification

  1. PYADAP_BACK option is really interesting, but I don't get it completely: shall I mesh the complete flowfield with higher detail (refined mesh), does this option handle only the body surface (or only the volume inside the body of interest, not the flowfield)? Is there any plan to use geometric files such as step or iges, to have the proper curvatures defined?

@chesiv
Copy link

chesiv commented Dec 2, 2020

Hi everyone,
I was tring to use the mesh adaptation feature but the whole procedure is not clear to me.
Following the previous indications I:

  1. downloaded the feature_adapt branch.

  2. downloaded the Adaptive 2D NACA 0012 example (https://pyamg.saclay.inria.fr/pyamgexamples.html#2dnaca): adap_NACA0012.cfg, mesh_NACA0012_inv.su2, NACA0012_ini.dat.

  3. added the following lines to the config:

% Type of sensor used for adaptation
% Options include GOAL (adjoint-based), MACH, PRES
PYADAP_SENSOR= GOAL
% Objective function used for goal-oriented adaptation
OBJECTIVE_FUNCTION= LIFT
% Maximum cell size for adaptation
PYADAP_HMAX= 500.0
% Minimum cell size for adaptation
PYADAP_HMIN= 1e-9
% Gradation factor (typically 1.2-1.8)
PYADAP_HGRAD= 1.8
% Norm used for adaptation
% 1.0 or 2.0 recommended for inviscid flows
% 4.0 recommeneded for viscid
ADAP_NORM= 4.0
% Approximate mesh size (NPOI) at each level
PYADAP_COMPLEXITY= (30000, 60000, 120000)
% Number of adaptations performed at each level
PYADAP_SUBITE= (3, 3, 3)
  1. run the following command: $ python3 SU2_RUN/mesh_adaptation_amg.py -f adap_ONERAM6.cfg -n 8.

The response I have obtained is the following:

SU2-AMG Anisotropic Mesh Adaptation

Mesh adaptation options:
PYADAP_COMPLEXITY : (30000, 60000, 120000)
PYADAP_SUBITE : (2, 2, 2)
PYADAP_SENSOR : MACH
PYADAP_HMAX : 500.0
PYADAP_HMIN : 1e-9
PYADAP_HGRAD : 1.8
PYADAP_FLOW_ITER : (2999, 4999, 6999)
PYADAP_ADJ_ITER : (2999, 4999, 6999)
PYADAP_CFL : (10.0, 15.0, 20.0)
PYADAP_RDG : NO

The ./adap folder was deleted

Generating GMF background surface mesh.
## WARNING: MISSING SU2 MESH FILE KEYWORD: NCORNERS=.
Initial CFD solution is provided.

As expected, a ./ADAP folder was created, containing these docs: amg_back_meshb, config_CFD.cfd, log.err, log.out (sizes 0 byte), mesh_NACA0012_inv.su2 (link to the mesh), NACA0012_ini.dat (link to the file.dat).

Did I perform the procedure correctly? Or did I miss something?
If it is correct and complete, which file contains the adaptive mesh? How can I use it since there is no new .su2 file?

Thank you for the big help

@timjim333
Copy link

@chesiv Did you ever get this working? =) I'd be interested to hear your workflow for newbies!

@antares190
Copy link

@chesiv (and also @timjim333) I confirm that the output you described is the expected one. In the *.cfg file I also make sure to add CSV tabular output management, as well as ASCII outputs.
TABULAR_FORMAT= CSV
OUTPUT_FILES= RESTART_ASCII

The python script to run the case with mesh adaptation is :
$ python3 SU2_RUN/mesh_adaptation_amg.py -f adap_ONERAM6.cfg -n 8
The python script handles all the folder structure, initialization, mesh adaptation(s), sub iteration(s).

  1. At first it removes the contents of an adap folder.
  2. creates adap\ini and starts the initialization run (with the original mesh).
  3. creates adap\ite0 and based on the solution of the initialization run adapts the mesh with the given goal. As @bmunguia explained, have a look at the end of amg.out if anything doesn't go as expected.

The only actual information I get from amg.log is usually in the last line
NbSensor 30518 msh->npobk 30518 NbrTyp 19 msh->npoin 52477
which just tells you how many points are in the old vs. new mesh, but you could just look at the .su2 files for that.

  1. repeats the subite as specified by the user.
  2. At the end of the run the final results will be copied to the parent directory, as well as an history_adap.csv file which will store the output of the final iteration of the initialization and each mesh adaptation.

To answer @chesiv question, in the adap\ite0 you will find an adap.su2 file which is the adapted mesh. The configuration file is modified (so to get the new mesh and restart from the previous iterations solution.

@timjim333
Copy link

timjim333 commented Dec 27, 2020 •
edited
Loading

Thanks for the walkthrough, @antares190 ! Actually, I also seem to be having some issues installing this branch successfully (but I'm not sure if it is also present in v7.0.8).

When I try to import pyamg it crashes with:

In [1]: import pyamg
---------------------------------------------------------------------------
ImportError                               Traceback (most recent call last)
<ipython-input-1-1160da809073> in <module>()
----> 1 import pyamg

ImportError: libpython3.7m.so.1.0: cannot open shared object file: No such file or directory

@bmunguia I realised that when I try to compile this branch, it seems to pick up the wrong python paths. (Though I'm not 100% sure if this is the cause.)

i.e. running:
CC=$(which mpicc) CXX=$(which mpicxx) ./meson.py build -Denable-autodiff=true -Denable-directdiff=true -Denable-pywrapper=true --prefix=/opt/su2/SU2v7_adap -Dwith-mpi=enabled -Denable-mkl=true -Dmkl_root=/opt/intel/compilers_and_libraries_2020/linux/mkl

reports:

Installing _amgio.
The Meson build system
Version: 0.52.999
Source dir: /opt/su2/SU2_src
Build dir: /opt/su2/SU2_src/build
Build type: native build
Project name: SU2
Project version: 7.0.3 "Blackbird"
C compiler for the host machine: /usr/lib64/openmpi/bin/mpicc (gcc 8.3.1 "gcc (GCC) 8.3.1 20191121 (Red Hat 8.3.1-5)")
C linker for the host machine: GNU ld.bfd 2.30-79
C++ compiler for the host machine: /usr/lib64/openmpi/bin/mpicxx (gcc 8.3.1 "g++ (GCC) 8.3.1 20191121 (Red Hat 8.3.1-5)")
C++ linker for the host machine: GNU ld.bfd 2.30-79
Host machine cpu family: x86_64
Host machine cpu: x86_64
Program python3 found: YES (/home/tjim/anaconda3/envs/sound/bin/python)
Run-time dependency MPI for c found: YES 4.0.3
Run-time dependency MPI for cpp found: YES 4.0.3
WARNING: Use the 'pic' kwarg instead of passing '-fpic' manually to 'meshb7'
Message: Boost sources found.
Run-time dependency mkl-static-lp64-seq found: YES 2020 Update 4
Program swig found: YES (/home/tjim/anaconda3/envs/sound/bin/swig)
Message: Using mpi4py from /home/tjim/anaconda3/envs/sound/lib/python3.7/site-packages/mpi4py/include
Run-time dependency python3 found: YES 3.6
Program install.sh found: YES (/bin/sh /opt/su2/SU2_src/SU2_PY/pySU2/install.sh)
Message: -------------------------------------------------------------------------
         |    ___ _   _ ___                                                      |
         |   / __| | | |_  )   Release 7.0.3 "Blackbird"                         |
         |   \__ \ |_| |/ /                                                      |
         |   |___/\___//___|   Meson Configuration Summary                       |
         |                                                                       |
         -------------------------------------------------------------------------
         
         Option          Value
         ---------------------
         TecIO:          true
         CGNS:           true
         AD (reverse):   true
         AD (forward):   true
         Python Wrapper: true
         Intel-MKL:      true
         OpenBlas:       false
         PaStiX:         false

         Please be sure to add the $SU2_HOME and $SU2_RUN environment variables,
         and update your $PATH (and $PYTHONPATH if applicable) with $SU2_RUN

         Based on the input to this configuration, add these lines to your .bashrc file:

         export SU2_RUN=/opt/su2/SU2v7_adap/bin
         export SU2_HOME=/opt/su2/SU2_src
         export PATH=$PATH:$SU2_RUN
         export PYTHONPATH=$PYTHONPATH:$SU2_RUN

         Use './ninja -C build install' to compile and install SU2

Build targets in project: 19
Found ninja-1.10.0.git at /opt/su2/SU2_src/ninja

Where the Run-time dependency python3 is reported as the system v3.6 whereas I'm running 3.7.8. Could it be that amg was installed into the system python? But before running meson, I had activated my python conda environment and mpi4py seems to be picked up from the right one. I'm just not sure why it also finds 3.6. Can you give me some tips for troubleshooting this?
Thanks very much!

@champagneproblems
Copy link

@timjim333 it could be an issue with shared libraries, which ultimately is a path issue, as you suspected.
To verify it try opening a python shell, checking the version displayed (should be your 3.7.8) and try import pyamg in the shell. If the "No such file or directory" error shows up, it means pyamg was not correctly installed; otherwise it is in fact installed.

In the latter case, you can check if your python is configured to enable shared libraries. To do it, according to this, in the python shell you can type
import sysconfig
sysconfig.get_config_vars('Py_ENABLE_SHARED')
If the result is [0], ie "False", you have to build python with the option --enable-shared.
Now, I don't know how to do accomplish that with conda, but with pyenv it involves installing the desired python version with CONFIGURE_OPTS=--enable-shared pyenv install <version>

To recap, a "workflow for newbies" which appears to be working for me is

  1. install pyenv either from source or from your distibution repositories, if available. IMPORTANT: make sure the environment variables are correctly set and pyenv init is added to your shell, as per the installation instructions.
  2. install the desired python version (for example 3.7.8), enabling shared libraries:
    CONFIGURE_OPTS=--enable-shared pyenv install 3.7.8
  3. clone/download the feature_adap branch of SU2;
  4. in the SU2 folder, configure meson
    ./meson.py build
    and make sure to update the environment variables in your .bashrc as indicated by meson's output.
  5. use ninja to build and install SU2
    ./ninja -C build install;
    if you run into a permission error, specify an installation directory passing the --prefix option to meson, as explained here.

These passages, at least for me, result in a running SU2 with mesh adaptation, although I have made no tests on running actual simulations.

That being said, the involvement of pyenv is due to the fact that the provided python wheel for pyamg is for python 3.7 only. If I understand correctly, the library itself is proprietary, sadly, so I assume @bmunguia has access either to the source files or to someone who can package them. Depending on the actual compatibility of the code, it would be nice if the wheel was updated to the latest python version, or at least the dependency on 3.7.* was removed.

@chesiv you can also try checking the logs in the log.err and log.out files to see if anything has gone wrong. Keep in mind all SU2 output is redirected to them, so you might not get any useful information on your terminal.

@timjim333
Copy link

Thanks for the tips so far! By any chance do you know which part of the configuration/path search that results in the line that says:

Run-time dependency python3 found: YES 3.6

I can't figure out why it picks up my environment 3.7 (which has the mpi4py installed) correctly, but then somehow pulls in 3.6! Thanks very much!

@timjim333
Copy link

timjim333 commented Jan 5, 2021 •
edited
Loading

Hi, @champagneproblems thank. I managed to get my amg to find the libpython3.7m.so.1.0 in the end - for Conda environments, it seems that the libraries are included but aren't on the LD_LIBRARY_PATH by default, so one has to export first. (I'm not sure if this is really a great workaround since the whole Python environment lib directory is now in the path, but it seems to work for now.)

I'm getting an issue now where the script seems to crash while Running initial CFD solution:

(test_env) tim@super:/work/projects/mesh_adapt_testing/A000i_adap/2_cfd$ LD_LIBRARY_PATH=$CONDA_PREFIX/lib:$LD_LIBRARY_PATH SU2_MPI_COMMAND="mpirun --use-hwthread-cpus -n %i %s" python /opt/su2/SU2v7_adap/bin/mesh_adaptation_amg.py -f inv_test.cfg -n 40 > mesh_adapt.out
  ## WARNING: MISSING SU2 MESH FILE KEYWORD: NCORNERS=.
Traceback (most recent call last):
  File "/opt/su2/SU2v7_adap/bin/mesh_adaptation_amg.py", line 112, in <module>
    main()
  File "/opt/su2/SU2v7_adap/bin/mesh_adaptation_amg.py", line 73, in main
    options.save_all    )
  File "/opt/su2/SU2v7_adap/bin/mesh_adaptation_amg.py", line 102, in mesh_adaptation_amg
    SU2.run.amg(config)
  File "/opt/su2/SU2v7_adap/bin/SU2/run/amg.py", line 220, in amg
    SU2_CFD(config_cfd)
  File "/opt/su2/SU2v7_adap/bin/SU2/run/interface.py", line 112, in CFD
    run_command( the_Command )
  File "/opt/su2/SU2v7_adap/bin/SU2/run/interface.py", line 292, in run_command
    raise exception(message)
RuntimeError: Path = /work/projects/mesh_adapt_testing/A000i_adap/2_cfd/adap/ini/,
Command = mpirun --use-hwthread-cpus -n 40 /opt/su2/SU2v7_adap/bin/SU2_CFD config_CFD.cfg
SU2 process returned error '254'
cmap[28922] == -1
cmap[28853] == -1
cmap[28858] == -1
cmap[28966] == -1
cmap[28989] == -1
cmap[28943] == -1
cmap[28915] == -1
cmap[28950] == -1
cmap[28923] == -1
cmap[28871] == -1
cmap[28941] == -1
cmap[28887] == -1
cmap[28889] == -1
cmap[37] == -1
cmap[32342] == -1
cmap[24652] == -1
cmap[50507] == -1
cmap[51850] == -1
--------------------------------------------------------------------------
Primary job  terminated normally, but 1 process returned
a non-zero exit code. Per user-direction, the job has been aborted.
--------------------------------------------------------------------------
--------------------------------------------------------------------------
mpirun detected that one or more processes exited with non-zero status, thus causing
the job to be terminated. The first process to do so was:

  Process name: [[57069,1],11]
  Exit code:    254
--------------------------------------------------------------------------

When I check the log.out, I see that it occurs when calling ParMETIS (log.err is empty):

------------------- Geometry Preprocessing ( Zone 0 ) -------------------
Three dimensional problem.
7860157 grid points before partitioning.
11486658 volume elements before partitioning.
3 surface markers.
32148 boundary elements in index 0 (Marker = BODY).
76160 boundary elements in index 1 (Marker = FARFIELD).
291440 boundary elements in index 2 (Marker = SYMMETRY).
Executing the partitioning functions.
Building the graph adjacency structure.
Calling ParMETIS...

I figured that that this might not be an AMG issue but a 7.0.3 build issue, so I tried running mpirun --use-hwthread-cpus -n 40 /opt/su2/SU2v7_adap/bin/SU2_CFD inv_test.cfg on its own and also got a similar error:

------------------- Geometry Preprocessing ( Zone 0 ) -------------------
Three dimensional problem.
7860157 grid points before partitioning.
11486658 volume elements before partitioning.
3 surface markers.
32148 boundary elements in index 0 (Marker = BODY).
76160 boundary elements in index 1 (Marker = FARFIELD).
291440 boundary elements in index 2 (Marker = SYMMETRY).
Executing the partitioning functions.
Building the graph adjacency structure.
Key 0 not found!
--------------------------------------------------------------------------
Primary job  terminated normally, but 1 process returned
a non-zero exit code. Per user-direction, the job has been aborted.
--------------------------------------------------------------------------
--------------------------------------------------------------------------
mpirun detected that one or more processes exited with non-zero status, thus causing
the job to be terminated. The first process to do so was:

  Process name: [[54379,1],4]
  Exit code:    254
--------------------------------------------------------------------------

Do you know what Key 0 might be referring to? Thanks and regards.

@timjim333
Copy link

Ok, I went back to try varying the .cfg settings on the simple v7.0.3 repo TestCases/euler/naca0012 case and managed to get mesh_adaptation_amg.py to run successfully. It seems that one bit of advice from here is no longer true, i.e. one needs to set:

% Write binary restart files (YES, NO)
WRT_BINARY_RESTART= YES
%
% Read binary restart files (YES, NO)
READ_BINARY_RESTART= YES

in order for the amg mesh adaptation to function.

However, for some reason, I can't get it working for my actual mesh of interest. When using the same settings, I get a different ParMETIS error (from the adap/ini/log.out):

------------------- Geometry Preprocessing ( Zone 0 ) -------------------
Three dimensional problem.
4929018 grid points before partitioning.
7406196 volume elements before partitioning.
3 surface markers.
18040 boundary elements in index 0 (Marker = BODY).
50968 boundary elements in index 1 (Marker = FARFIELD).
284054 boundary elements in index 2 (Marker = SYMMETRY).
Executing the partitioning functions.
Building the graph adjacency structure.
[ 1] ***ASSERTION failed on line 207 of file ../externals/parmetis/libparmetis/comm.c:sendind[i] >= firstvtx && sendind[i] < lastvtx
[ 1] 361316 123226 246452
[1609857970.901920] [super:1060888:0]           sock.c:344  UCX  ERROR recv(fd=62) failed: Connection reset by peer

The only difference that I can think of is that my flow is fully supersonic and my mesh is an unstructured core with a structured collar - might this be causing the issue? Kind regards.

@timjim333
Copy link

timjim333 commented Jan 5, 2021 •
edited
Loading

I made a fully unstructured version of the mesh and it seems I can now pass the Geometry Preprocessing stage. I'm now having a different issue that I do not know if it is related. After 250 iterations, the simulation hangs, with the following output in the /adap/ini/log.out:

|         249|   -7.427827|    0.000000|    0.024016|    0.000401|  8.4160e+09|
|         250|   -7.390279|    0.000000|    0.024016|    0.000401|  8.4160e+09|
+-----------------------------------------------------------------------+
|        File Writing Summary       |              Filename             |
+-----------------------------------------------------------------------+
|SU2 restart                        |restart_flow.dat                   |
|Paraview binary                    |flow.vtk                           |
|Paraview binary surface            |surface_flow.vtk                   |
[1609863350.417715] [super:1071838:0]           sock.c:344  UCX  ERROR recv(fd=50) failed: Bad address
[1609863350.417771] [super:1071838:0]           sock.c:344  UCX  ERROR recv(fd=25) failed: Connection reset by peer
[1609863350.418024] [super:1071838:0]           sock.c:344  UCX  ERROR sendv(fd=-1) failed: Bad file descriptor

UPDATE: Bizarrely, if I attempt to run the adaptation script again, it now fails earlier with the same UCX Connection reset error:

------------------- Geometry Preprocessing ( Zone 0 ) -------------------
Three dimensional problem.
774736 grid points before partitioning.
4416440 volume elements before partitioning.
3 surface markers.
18040 boundary elements in index 0 (Marker = BODY).
44930 boundary elements in index 1 (Marker = FARFIELD).
177930 boundary elements in index 2 (Marker = SYMMETRY).
Executing the partitioning functions.
Building the graph adjacency structure.
[1609864445.066242] [super:1075233:0]           sock.c:344  UCX  ERROR recv(fd=98) failed: Connection reset by peer
[1609864445.066058] [super:1075147:0]           sock.c:344  UCX  ERROR recv(fd=54) failed: Connection reset by peer

Is this a bug?

UPDATE: This seems to have gone away pulling a different version OpenMPI and recompiling.

@timjim333
Copy link

I was running some tests using a simple geometry but for the refinement seems to be chewing up the geometry. Has anyone who has worked on 3D geometries experienced this or figured out a way to avoid this?

I was using the following settings:

% ------------- MESH ADAPTATION PARAMETER ------------%
%
% Type of sensor used for adaptation
% Options include GOAL (adjoint-based), MACH, PRES
PYADAP_SENSOR= MACH
% Maximum cell size for adaptation
PYADAP_HMAX= 500.0
% Minimum cell size for adaptation
PYADAP_HMIN= 1e-9
% Gradation factor (typically 1.2-1.8)
PYADAP_HGRAD= 1.8
% Norm used for adaptation
% 1.0 or 2.0 recommended for inviscid flows
% 4.0 recommeneded for viscid
ADAP_NORM= 2.0
% Approximate mesh size (NPOI) at each level
PYADAP_COMPLEXITY= (1000000, 1300000, 1500000)
% Number of adaptations performed at each level
PYADAP_SUBITE= (2, 2, 2)

refine

I noticed a few comments up, there is the PYADAP_BACK setting, which sounds like it might solve this, but how should one use this? Should the geometry alone be re-meshed to a high resolution and the whole volume exported as a SU2 block? Or should it be just a surface mesh? (It seems that Pointwise is not able to output only the surface mesh in SU2 format.)

@antares190
Copy link

antares190 commented Jan 17, 2021 •
edited
Loading

I experienced the issue you described and tried different strategies to mitigate it.
The back mesh was promising, but after setting it up properly (yes, full mesh, only surfaces need to be refined, volume and boundaries may be kept coarse/high growth ratio), it doesn’t help much. I think this behavior is related to the inability of the adaptation to either keep a tbd element size along curvatures / have a max element size on the surface.
I bet the nodes of the adapted grid are still on the surface but huge elements (indicating no particular gradient there) ultimately deteriorate the surface mesh quality.

As a way around to this I separated into different zones/groups the parts of the surface mesh I wanted to “freeze”, this approach works as the mesh adaptation always respect/keeps the original mesh edge position. The approach above obviously helps mitigating the issue, surface mesh quality is still highest in the initialization mesh.

As this method only mitigates the problem, please share your experience! Any other indication/suggestion is very welcome!

@timjim333
Copy link

@antares190 running some more tests, I'm not sure that the adapted grid is still on the surface any more, especially on the trailing edge. Or perhaps it is, but it has lost associativity with its neighbour? This also seems to be the case for the fuselage, where cells seem to be jumping to the opposite side of the aircraft. Did you experience this issue too?
interior_ite1

Regarding separating zones, do you mean through SU2's multizone? In this case, did you apply the aircraft surface as a separate zone? Or a 'core' region around the aircraft? Thanks for your thoughts! Kind regards, Tim

@timjim333
Copy link

Also, @bmunguia - how should I submit a patch to amg.py? I have a fix for it not working with meshes placed in relative paths. Kind regards.

@timjim333
Copy link

@TobiKattmann - I was just reading your slides on multizone, when I noticed that you mention python scripts not working for multizone, and I guess that would include the mesh adaptation scripts... is this still the case? Thank you!

@antares190
Copy link

@antares190 running some more tests, I'm not sure that the adapted grid is still on the surface any more, especially on the trailing edge. Or perhaps it is, but it has lost associativity with its neighbour? This also seems to be the case for the fuselage, where cells seem to be jumping to the opposite side of the aircraft. Did you experience this issue too?
interior_ite1

Regarding separating zones, do you mean through SU2's multizone? In this case, did you apply the aircraft surface as a separate zone? Or a 'core' region around the aircraft? Thanks for your thoughts! Kind regards, Tim

When I was referring to separated zones I mean in the mesh, so I'm not referring to SU2 multizone. This is an example I took online (it is apparently a structured hexa mesh, so not the best example for mesh adaptation, hexa not supported), but I hope it explains the different zones at mesher level.
By defining different zones for top and bottom of the wing (and wing tip), the edges, leading and trailing as well as the one on the tip will be maintained after mesh adaptation.
In your case I would also suggest to split the body in body top and body bottom so to prevent (or reduce) artifacts in the trailing cone.

image

@timjim333
Copy link

Thank you. I see, so you set separate markers, e.g. MARKER_EULER= ( WingTop, WingBottom ), and where the marker boundaries meet, the mesh points on that interface is unchanged; did I understand that correctly? I guess that while this will reduce the artefacts on the LE/TE it still results in a departure from the design shape at the portions away from the boundary edges. As you mentioned before, there might not be a way to solve this without mapping back to an iges or parametric file.

I suppose another approach might be to work the other way around? Perhaps one can limit the max allowable adapted size to match the 'worst case' allowable surface mesh? I just realised that may be a contributing factor to why the points jumpy across the fuselage; PYADAP_HMAX may be set excessively large.

If I set this limit to less than the average surface mesh cell size, will it attempt to refine everything over the limit instantly? Or will it only use the limit for 'new' cells that are generated during adaptation? Thanks for your thoughts.

@antares190
Copy link

Yes, I set separate markers. I don't think that mesh nodes aren't unchanged, but the edge topology is respected. You are right also when you refer to departure from the design shape (mostly driven by excessively big TRIAS on the surface; I suspect all the surface nodes still lay on the original, initialization surface mesh); there is still departure.
I tried to reduce the PYADAP_HMAX but as it affects all the mesh (including volume elements), in order to reach the desired refinement on the surface, the number of cells easily explodes.

If I set this limit to less than the average surface mesh cell size, will it attempt to refine everything over the limit instantly? Or will it only use the limit for 'new' cells that are generated during adaptation? Thanks for your thoughts.

I believe it will affect all the mesh, but yet again, it may be also influenced by the numbers of PYADAP_SUBITE specified in the config file. It would be great to have different parameters for volume mesh dimensions and surface mesh dimensions.

@timjim333
Copy link

I just tried reducing PYADAP_HMAX and the AMG run time has increased hugely, so I think this ties in with your experience and is likely due to the massive increase in the number of volume cells. Have you noticed much difference in setting PYADAP_SUBITE to different values? Is 2 or 3 passes any better than a single sub-iteration? Other than generating meshes of different sizes, is there much point to setting more than one main iteration?

@TobiKattmann
Copy link
Contributor

@TobiKattmann - I was just reading your slides on multizone, when I noticed that you mention python scripts not working for multizone, and I guess that would include the mesh adaptation scripts... is this still the case? Thank you!

Hi @timjim333, I have to admit I am not fully (i.e. not 😉) aware of the mesh adaption capabilities of/with SU2. And there is quite a bit to read in this issue which i won't really have time to until the weekend but here's already my quicktake:
If you can run the intended feature (with multizone) manually (i.e. a c++ binary) and it works, then probably the python script 'just' sets up some folders and does clever choices for parameters that are put into new .cfg files which are run afterwards. So just orchestrating. There it is likely that it is not fixed with multizone by now but that is usually not too difficult to fix ... it is tedious though.

If you think that I understand what you want to do specifically with the posts in this thread then I will read through that and answer again, otherwise I would not mind a specification 👍

@fredericodpc
Copy link

Hi @timjim333, any luck with the surface mesh deterioration issue?
I've managed to successfully run the Onera M6 case, with a sufficiently refined volume mesh around the shock wave region, however the wing surface mesh does not follow this adaptation. Does anyone by any chance have a working .cfg file for the M6 case? I may be setting something up incorrectly.
Thanks a lot!

@timjim333
Copy link

Hi @TobiKattmann sorry, I must have missed your message somehow! Thanks for the input, maybe when the mesh adaptation strategy has matured a little more (or is better documented), it can be a bit more easily used. This is also in response to @fredericodpc Right now, I don't know if it is a bug or I'm just unsure of how to correctly use it, but I also have not been able to get the mesh to respect the wing surface. I've also not heard from the branch author either, so I don't know if this is being maintained publically anymore. =( It's a shame because it looked really promising for optimising the cost/accuracy of supersonic solutions!

Ideally, I think one should be able to pass in a representation of the actual surface (either through an stl surface or as a very fine mesh), which should be used as a reference surface. As a minimum, we should be able to set a maximum surface mesh turning angle or some other surface-local control.

@fredericodpc
Copy link

Hi @timjim333, thanks for your reply. Well, let's wait to hear from the branch author, there's probably a way to keep/refine the surface mesh, even judging by one picture posted on this issue. Thanks again!

@timjim333
Copy link

@fredericodpc if you find a way I'd be interested to hear how you did it! In my testing, it seems to work for 2D but not 3D.

@bigfooted bigfooted mentioned this issue Jan 27, 2022
Open
@49takashi 49takashi mentioned this issue Jul 5, 2022
Closed
@nair-subra
Copy link

Hi everyone,
I'm really interesting by the adaptitve mesh feature and i am wondering if the branch is still alive or if anyone have done any progress before starting to play with this branch
regards

@msahinae00
Copy link

Hi all,
When l run initial su2 mesh file with python3 ~/SU2-feature_adap_sst/externals/AMGIO/su2io/su2_to_gmf.py -m ite4.su2 -o ite4 and l provide this file as adap_back binary file, it works nicely in 2D. However, this approach does not work in 3D. My understanding is that in 3D, this file should contain third order triangles with edges. This is at least what l saw in the python2 version of the pyamg (meshp2/3.meshb files). I have tried Gmsh to create third order triangles as adap_back file, this approach is not working (mat due to missing edges or non-closed volume). I have also provided meshp3.meshb output files from python2 version, this is not working either. Can someone tell what this file should look like in 3D and how/where we can obtain this adap_back file (GMSH, pointwise, etc).
Best

@casfra96
Copy link

Hi all,
I'm trying to install the SU2_feature_adap branch. I'm working on WSL 2 with Ubuntu 20.04.
I have also tried the entire procedure on a PC with Ubuntu 20.04 installed, but I'm getting the same errors.

I'm following the procedure described in comments above, but when I try to run ./meson.py build, I get an error:

Installing _amgio.
Installing pyAMG for Linux.
pyAMG installation failed.

In fact, I'm having problems installing pyamg. (_amgio seems to be installed correctly).
So, I tried to install it manually to understand what the problem might be. Moving to the subdirectory "/externals/AMGIO/pyamg/Python3", there are two .whl files (one for Linux and one for macOS). I tried to install the Linux version with the following command:
sudo pip3.11 install pyamg-1.0.0-cp37-cp37m-linux_x86_64.whl
I get the error:
ERROR: pyamg-1.0.0-cp37-cp37m-linux_x86_64.whl is not a supported wheel on this platform.

I have also tried Python 3.7 and 3.8 versions without success.
This is the output of the command uname -a:

Linux DESKTOP-3OCOBTM 5.10.16.3-microsoft-standard-WSL2 #1 SMP Fri Apr 2 22:23:49 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux

I seem to meet the requirements of the .whl file. Does anyone have any ideas on what the problem might be?

@msahinae00
Copy link

Can you try python 3.7.7? This is what l have used.

@casfra96
Copy link

Can you try python 3.7.7? This is what l have used.

It is working! Tanks!

@msahinae00
Copy link

It looks like pointwise Mesh Link frame have a support for both mesh link and database (CAD) and it includes both edge data and corners ("Edge2" for edges and begining and end vertices of each edge data set corresponds to corner data.)
https://www.pointwise.com/doc/user-manual/file/export/cae/kestrel.html
https://github.com/pointwise/MeshLink library also provides some info for mesh to CAD projection.

Therefore adap_back mesh file can be prepared in GAMMA mesh file format including corners, edges, triangles, tets and vertices. But l am not sure about the correctness of su2 files created at each refinement level since corner and edge data will be lost when they are written in su2 format. It may be a good idea to use the GAMMA mesh file format for pyamg mesh adaptation levels and use su2 mesh file format only when it is needed for SU2 runs.

@msahinae00
Copy link

Also pyamg.write_mesh correctly write the mesh file but when l convert its solution file to ascii,with libmesh there are excessively large numbers inside the file. It is most likely that there are some errors with it (solve part).

@msahinae00
Copy link

A small note: All the face elements in the adap_back meshb file should be arranged in a way that their normal vector should point outwards for all surfaces and then adap_back option can work correctly. Also you may deactivate geometry enrichment (corners/edges) if these data are provided within the gamma mesh format. But we run refinement on gamma mesh format rather than su2 mesh file. Generally pointwise writes face elements with mixed normal vectors (inward/outward) for su2 format. May be l don't know pointwise well enough to get normals correctly.

@nirvananas
Copy link

Hello everyone

I am working with SU2 on the feature_adap branch and have come across an issue during the execution of the mesh adaptation script. After following the standard installation procedure and verifying the installation (the exact same steps that @chesiv presented), I encountered a problem when running the mesh_adaptation_amg.py script.

The command used was:

python3 $SU2_RUN/mesh_adaptation_amg.py -f adapt_Mesh.cfg -n 4

The script initiates the mesh adaptation process as expected, with the following output:

SU2-AMG Anisotropic Mesh Adaptation

Mesh adaptation options:
PYADAP_COMPLEXITY : (30000, 60000, 120000)
PYADAP_SUBITE : (2, 2, 2)
PYADAP_SENSOR : GOAL
PYADAP_HMAX : 500.0
PYADAP_HMIN : 1e-9
PYADAP_HGRAD : 1.8

./adap exists. Removing old mesh adaptation in 10s.
The ./adap folder was deleted

Generating GMF background surface mesh.
Initial CFD solution is provided.

However, no adap\ite0 folder gets created.
The log.err file in the adap/ini directory provides this traceback:

Traceback (most recent call last):
  File ".../mesh_adaptation_amg.py", line 111, in <module>
    ...
RuntimeError: ...
Error in TokenizeString(): two or more options before an "=" sign in the configuration file.
terminate called after throwing an instance of 'int'
...
*** Process received signal ***
Signal: Aborted (6)
Signal code:  (-6)
...

The critical part of this error seems to be the Error in TokenizeString() message, suggesting an issue with parsing the configuration file. This error typically indicates a syntax problem, such as having multiple options on a single line without proper separation by an equal sign (=). However, upon reviewing adap/ini/config_CFD.cfg, was unable to find any lines that clearly violated this syntax rule.

Could this error be indicative of a more subtle issue within the configuration file, or might it be related to specific aspects of the feature_adap branch? Any insights, suggestions, or guidance on how to troubleshoot this error would be greatly appreciated.

If anyone has a working adapt_mesh.cfg file for a similar setup or for the feature_adap branch, I would be very interested in seeing it. Understanding the configuration details of a working example could be highly beneficial in resolving my issue.

Thank you for your help!

@antares190
Copy link

antares190 commented Dec 11, 2023 •
edited
Loading

Hi,
I never experienced that error, at least I don't recall anything similar. The file I share below is a "config_CFD.cfg" taken from from the adap/ini/ folder. I hope it helps.

SOLVER= EULER
KIND_TURB_MODEL= NONE
MATH_PROBLEM= DIRECT
RESTART_SOL= NO
AXISYMMETRIC= NO
MACH_NUMBER= 2.0
AOA= 0.0
SIDESLIP_ANGLE= 0.0
FREESTREAM_OPTION= TEMPERATURE_FS
FREESTREAM_TEMPERATURE= 185
FREESTREAM_PRESSURE= 2000.
REF_ORIGIN_MOMENT_X= 0.00
REF_ORIGIN_MOMENT_Y= 0.00
REF_ORIGIN_MOMENT_Z= 0.00
REF_LENGTH= 1
REF_AREA= 1
REF_DIMENSIONALIZATION= FREESTREAM_VEL_EQ_ONE
MARKER_EULER= ( Body, FarField )
MARKER_SUPERSONIC_INLET= ( Inlet, 185, 2000, 550, 0.0, 0 )
MARKER_OUTLET= ( Outlet, 2000 )
MARKER_PLOTTING= ( Body )
MARKER_MONITORING= ( Body )
NUM_METHOD_GRAD= WEIGHTED_LEAST_SQUARES
CFL_NUMBER= 0.75
CFL_ADAPT= NO
CFL_ADAPT_PARAM= ( 0.1, 2.0, 0.5, 5 )
RK_ALPHA_COEFF= ( 0.66667, 0.66667, 1.000000 )
ITER= 5000
VENKAT_LIMITER_COEFF= 0.1
ADJ_SHARP_LIMITER_COEFF= 3.0
REF_SHARP_EDGES= 3.0
SENS_REMOVE_SHARP= NO
LINEAR_SOLVER= BCGSTAB
LINEAR_SOLVER_PREC= ILU
LINEAR_SOLVER_ERROR= 1E-6
LINEAR_SOLVER_ITER= 10
MGLEVEL= 0
MGCYCLE= W_CYCLE % was V_CYCLE
MG_PRE_SMOOTH= ( 1, 2, 2, 3 )
MG_POST_SMOOTH= ( 0, 0, 0, 0 )
MG_CORRECTION_SMOOTH= ( 0, 0, 0, 0 )
MG_DAMP_RESTRICTION= .7  % was 0.7
MG_DAMP_PROLONGATION= .7 % was 0.7
CONV_NUM_METHOD_FLOW= AUSM
MUSCL_FLOW= YES
SLOPE_LIMITER_FLOW= VENKATAKRISHNAN
JST_SENSOR_COEFF= ( 0.5, 0.02 )
TIME_DISCRE_FLOW= EULER_IMPLICIT
CONV_NUM_METHOD_TURB= SCALAR_UPWIND
MUSCL_TURB= NO
SLOPE_LIMITER_TURB= VENKATAKRISHNAN
TIME_DISCRE_TURB= EULER_IMPLICIT
MESH_FILENAME= MyMesh.su2
MESH_FORMAT= SU2
MESH_OUT_FILENAME= mesh_out.su2
OUTPUT_FILES= (RESTART_ASCII, PARAVIEW, SURFACE_PARAVIEW)
SOLUTION_FILENAME= restart_flow.dat
SOLUTION_ADJ_FILENAME= solution_adj.dat
TABULAR_FORMAT= CSV
CONV_FILENAME= history
HISTORY_OUTPUT= ITER, RMS_RES, AERO_COEFF, FLOW_COEFF
RESTART_FILENAME= restart_flow.csv
RESTART_ADJ_FILENAME= restart_adj.dat
VOLUME_FILENAME= flow
VOLUME_ADJ_FILENAME= adjoint
GRAD_OBJFUNC_FILENAME= of_grad.dat
SURFACE_FILENAME= surface_flow
SURFACE_ADJ_FILENAME= surface_adjoint
WRT_SOL_FREQ= 50.0
WRT_CON_FREQ= 1
SCREEN_OUTPUT= (INNER_ITER, WALL_TIME, RMS_DENSITY, LIFT, DRAG)
ADAP_NORM= 1.5
WRT_BINARY_RESTART= NO
READ_BINARY_RESTART= NO
NUM_METHOD_HESS= GREEN_GAUSS
PYADAP_INV_BACK= YES
DV_VALUE_NEW= 0
DV_VALUE_OLD= 0
OPT_ITERATIONS= 100
OPT_ACCURACY= 1e-10
OPT_RELAX_FACTOR= 1.0
OPT_GRADIENT_FACTOR= 1.0
OPT_BOUND_UPPER= 10000000000.0
OPT_BOUND_LOWER= -10000000000.0
OPT_COMBINE_OBJECTIVE= NO
OPT_CONSTRAINT= NONE
VALUE_OBJFUNC_FILENAME= of_eval.dat
BREAKDOWN_FILENAME= forces_breakdown.dat
REYNOLDS_NUMBER= 0.0
TARGET_CL= 0.0
MULTIPOINT_WEIGHT= (1.0)
MULTIPOINT_MACH_NUMBER= (2.0)
MULTIPOINT_AOA= (0.0)
MULTIPOINT_SIDESLIP_ANGLE= (0.0)
MULTIPOINT_REYNOLDS_NUMBER= (0.0)
MULTIPOINT_TARGET_CL= (0.0)
MULTIPOINT_FREESTREAM_PRESSURE= (2000.)
MULTIPOINT_FREESTREAM_TEMPERATURE= (185)
MULTIPOINT_OUTLET_VALUE= ( 2000 )
MULTIPOINT_MESH_FILENAME= (MyMesh.su2)
OBJECTIVE_FUNCTION= DRAG
DV_KIND= FFD_SETTING
DV_PARAM= ( 1, 0.5) 
DEFINITION_DV= ( 30 , 1.0 | WING  | 0.0 , 0.05  )
NUMBER_PART= 64
VOLUME_OUTPUT= COORDINATES, SOLUTION, PRIMITIVE
COMPUTE_METRIC= NO
DV_VALUE= 0

I understand you installed it on the version 7.0.3 (to my knowledge it is the latest supported by the feature_adap branch).

@nirvananas
Copy link

nirvananas commented Dec 14, 2023 •
edited
Loading

Hi,

Thank you for your quick answer, It indeed seems that I had some parameters that were passed that did not match su2 version 7.0.3

I am experiencing an issue while running the mesh adaptation process in SU2. The process starts but fails during the mesh generation phase, leading to a FileNotFoundError. I am looking for assistance in resolving this error.
`
SU2-AMG Anisotropic Mesh Adaptation

Mesh adaptation options:
PYADAP_COMPLEXITY : (100, 200, 300)
PYADAP_SUBITE : (2, 2, 2)
PYADAP_SENSOR : MACH
PYADAP_HMAX : 200
PYADAP_HMIN : 1e-8
PYADAP_HGRAD : 1.3
PYADAP_RESIDUAL_REDUCTION : (3, 3, 3)
PYADAP_FLOW_ITER : (500, 500, 500)

./adap exists. Removing old mesh adaptation in 10s.
The ./adap folder was deleted

Generating GMF background surface mesh.
Initial CFD solution is provided.

Starting mesh adaptation process.

Iteration 0 - Mesh size coefficient 100.0
(1/2) Generating adapted mesh using AMG
AMG done:
Running CFD
Traceback (most recent call last):
File "/path/to/SU2/install/bin/mesh_adaptation_amg.py", line 111, in
main()
File "/path/to/SU2/install/bin/mesh_adaptation_amg.py", line 72, in main
options.save_all )
File "/path/to/SU2/install/bin/mesh_adaptation_amg.py", line 101, in mesh_adaptation_amg
SU2.run.amg(config)
File "/path/to/SU2/install/bin/SU2/run/amg.py", line 466, in amg
os.rename(cur_solfil, cur_solfil_ini)
FileNotFoundError: [Errno 2] No such file or directory: 'flo.dat' -> 'flo_ini.dat'`

and it0 folder is created, with some log.err and log.out, but they are empty,

I would appreciate any guidance or suggestions on how to resolve this issue. Thank you!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests