Electromagnetism

[haryegechi]

Hi!
I am checking your MFEM as a newbie. I was thinking of checking TM modes for a cylindrical pillbox. How can I do it without MPI?
Thanks
Best wishes
Babai

[mlstowell]

Hello, Babai,

Thank you for your interest in MFEM. Unfortunately, the only eigenvalue solvers we currently support in MFEM do require MPI. However, if you are familiar with ARPACK, it can certainly be used with MFEM it will just require a bit more programming.

Specifically, you could use MFEM to assemble the matrices that you need (a mass matrix built using the VectorFEMassIntegrator and a stiffness matrix using the CurlCurlIntegrator). ARPACK then operates through a reverse communication interface. This means that your code repeatedly calls an ARPACK routine and this routine instructs your code to perform a Matrix-Vector multiply or a solve, etc. until the eigenvalues have converged.

Does this sound like something that could meet your needs?

Thanks,
Mark

[martemyev]

Hi All,

I've used the JADAMILU (http://homepages.ulb.ac.be/~jadamilu/) eigensolver some time ago. It's quite fast - you can find the comparisons with ARPACK on the JADAMILU's website. I'm not sure if it works with matrices produced by curl-curl operator - check the documentation. Also, the code is not open-source, and the libraries are provided for some combinations of OS and compiler - basically, there is no library for Mac OS X, otherwise you should be okay. If the library looks suitable for your needs I can help create a wrapper and an example of usage with MFEM matrices - in my opinion, it should be MUCH easier than using ARPACK via the reverse communication interface.

Cheers,
Mikhail

[haryegechi]

Dear Mark and Mikhail,

thanks a lot for ideas. I need opensource stuff. I did not know Arpack or jadamilu. I am just looking at them. However, since I am newbie, I am fumbling on the usage to solve wave equation for pillbox for FEM010 mode so that I may check my RF cavity in the next stage.

Best,
Babai

[haryegechi]

Hello!

I tried to work with openMP. I have installed metis and hypre as per your advice. I compiled MFEM using make parallel -j. Then I wanted to see make check which shows following error.

Parallel example [ mpirun -np 4 ex1p ... ]: Current memory used: 384 bytes
Maximum memory used: 384 bytes
***Memory allocation failed for SetupCtrl: maxvwgt. Requested size: 3779571220488 bytes

Current memory used: 384 bytes
Maximum memory used: 384 bytes
***Memory allocation failed for SetupCtrl: maxvwgt. Requested size: 3779571220488 bytes

Mesh::GeneratePartitioning: error in METIS_PartGraphKway!
Current memory used: 384 bytes
Maximum memory used: 384 bytes
***Memory allocation failed for SetupCtrl: maxvwgt. Requested size: 3779571220488 bytes

Mesh::GeneratePartitioning: error in METIS_PartGraphKway!
Current memory used: 384 bytes
Maximum memory used: 384 bytes
***Memory allocation failed for SetupCtrl: maxvwgt. Requested size: 3779571220488 bytes

Mesh::GeneratePartitioning: error in METIS_PartGraphKway!
Mesh::GeneratePartitioning: error in METIS_PartGraphKway!
FAILED
make[1]: Leaving directory '/home/vega/decrypted/ELECTROMAG/MFEM/examples'

I wanted to work in my laptop having 8GB RAM. Please guide.

Thanks and Regards

[mlstowell]

Hello, Babai,

I do most of my development on an older MacBook Pro with 8GB so I expect your laptop will work fine. Does your laptop run Windows, Linux, or Mac OSx?

Another opensource eigenvalue solver you might consider is FEAST from UMass Amherst.

Now on the the current issue. The amount of memory requested here is not reasonable so it must indicate an error. There are a number of possibilities. I would start by checking METIS to make sure its tests succeed. Unfortunately the version I use, 4.0.1, doesn't seem to have a "make check" target but it does have a README file in the Graphs subdirectory that shows several simple tests that you can run.

I'm also suspicious of openMP. I believe that many of us typically use MPICH which has some important differences from openMP and it's possible that MFEM and openMP may not be as fully tested. You may want to stick with the serial version initially. Both ARPACK and FEAST have serial versions so you should be able to create a simple eigensolver using MFEM and one of these packages.

Best wishes,
Mark

[haryegechi]

Dear Mark,

Thanks for your prompt reply. I am using OpenSuse leap 42.1 - 64 bit. I tested METIS for graph using
`[vega@Aditya: metis-5.1.0 1:08 Jul 16, 2016]$ build/Linux-x86_64/programs/graphchk graphs/test.mgraph

---

METIS 5.0 Copyright 1998-13, Regents of the University of Minnesota
(HEAD: , Built on: Jul 16 2016, 00:55:38)
size of idx_t: 32bits, real_t: 32bits, idx_t *: 64bits

Graph Information ---------------------------------------------------
Name: graphs/test.mgraph, #Vertices: 766, #Edges: 1314

Checking Graph... ---------------------------------------------------
The format of the graph is correct!

---

[vega@Aditya: metis-5.1.0 1:08 Jul 16, 2016]$
`

This means METIS seems to be OK. You may be right. MFEM works with MPICH. I have got ARPACK and FEAST while I am really confused to write the algorithm for solving curlcurl wave equation.

By the by, I have tested openMP benchmarking codes which are running OK.

Best
Babai

[mlstowell]

Hi, Babai,

I expect openMP would run just fine on its own. The trouble is usually that openMP defines the MPI_Comm as a struct whereas in MPICH this is just an integer. It's possible that MFEM or another parallel library might treat the MPI_Comm object as an integer which can cause real problems if it is in fact a struct. I don't know that this is a problem with MFEM but I have seen this before in other parallel libraries and I'm not aware of any validation tests of MFEM with openMP so it is possible.

I can try to walk you through the process of writing the algorithm for the curl curl eigenvalue problem. You should start with ex13p which solves this eigenvalue problem in parallel. The serial version would look quite similar. The main differences would be that most object names that start with "Par" should have that prefix removed e.g. ParGridFunction would become GridFunction, next any references to MPI should be removed, linear algebra objects like HypreParVector and HypreParMatrix would be replaced with Vector and SparseMatrix respectively. The harder part would be interfacing with ARPACK or FEAST rather than Hypre. I can probably give you more advice when you get to this stage.

Best wishes,
Mark

[haryegechi]

Dear Mark,
Thanks a lot.
I am grateful to you for kindly sharing time for me. I am trying to follow your advices and shall report you the status.

Best Regards
Babai

[v-dobrev]

Hi Babai,

I work a lot on MFEM and I do use OpenMPI rather than mpich, so I can confirm that everything works fine with OpenMPI both on linux and macosx. I see that you used version 5 of metis, which requires additional flag when configuring MFEM: did you set MFEM_USE_METIS_5=YES? If not, try this:

make config MFEM_USE_MPI=YES MFEM_USE_METIS_5=YES
make -j

Another (maybe simpler) option is create you own config file using

cp -p config/defaults.mk config/user.mk

and then edit config/user.mk to set the above options. You will probably need to set the paths to hypre and metis too, e.g. look for the HYPRE_* and METIS_* variables in the file.

Hope this helps,
Veselin

[haryegechi]

Dear Veselin,

I followed that part as it was mentioned in the "Building MFEM" page. The file .ex1p was compiled. The error appeared while it ran. The error mentioned "current memory used : 392 bytes memory allocation failed for SetupCtrl: ctrl->tpwgts, Requested size: 169324790677512 bytes."

The error later mentioned another error "Mesh::GeneratePartitioning: error in METIS_PartGraphKway!
MPI_ABORT was invoked on rank 1 in communicator MPI_COMM_WORLD
with errorcode 1."

I tested METIS as I discussed with Mark and found METIS reports no error while tested with its example graph.

The path for METIS and HYPRE have also been set.

Best
Babai

[haryegechi]

@mark,

Hi!

I was trying to go through your steps. I see that in ex13p, you have serial mesh steps for UniformRefinement() in code segment #4 while parallel Mesh in #5. If I go for only serial stuffs, then do I need to go through #5 ?

In #6, HYPRE has been used. How may I use ARPACK here?

best,
Babai

[v-dobrev]

Hi Babai,

The instructions in the "Building MFEM" page do not explicitly mention the option MFEM_USE_METIS_5 which is required when building with Metis v5. If you did not set MFEM_USE_METIS_5 to YES, then that is the reason you get the run-time error.

Veselin

Edit: I updated the Building MFEM page with short instructions for building parallel MFEM with METIS 5.

[haryegechi]

Hi Veselin,

I have changed those options at user.mk for MFEM_USE_METIS_5= YES is done.

I did not change in default.mk.

Best,
Babai.

On 17 Jul 2016 04:53, "Veselin Dobrev" notifications@github.com wrote:

Hi Babai,

The instructions in the "Building MFEM" page do not explicitly mention the
option MFEM_USE_METIS_5 which is required when building with Metis v5.
If you did not set MFEM_USE_METIS_5 to YES, then that is the reason you
get the run-time error.

Veselin

—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
#72 (comment), or mute
the thread
https://github.com/notifications/unsubscribe-auth/ATgcn4-WPJcD18sgEO_6dN1eRzFxBLLjks5qWWf7gaJpZM4JMLhK
.

[v-dobrev]

Hmm, then the problem with OpenMPI must be somewhere else. I'll try to reproduce it in a VM.

When you tried MPICH, were you able to run ex13p? For example, try any of the sample runs in the comments at the beginning of ex13p.cpp. If that works, you probably don't need to write a serial version of ex13p using another eigensolver.

[haryegechi]

I have openMP and I tried ex13p.cpp with that. I do not have MPICH.
From the error that I reported earlier, I feel that the issue is use with virtual memory / swap memory allocation by METIS for graph manipulation. I tried with both 32 bit and 64 bit config for METIS. The error remains same.

[v-dobrev]

Hi Babai,

I was not able to reproduce the errors you get. Here's what I did:

1. I installed OpenSuse leap 42.1 64-bit in a VM and updated to the latest packages.

2. I installed the following packages:

   gcc-c++
   openmpi-devel
   make
   cmake (for METIS 5)
   

3. Then I configured OpenMPI with

   ~> mpi-selector-menu

   and selected the openmpi-x.x.x option.

4. I logged out and logged back in, in order to get the proper environment setup for OpenMPI.

5. Then I was able to build hypre 2.10.0b, metis 5.1.0, and mfem 3.2 successfully, following the steps from the "Building MFEM" webpage.

6. I ran make check and it worked fine.

7. I ran the first sample run from ex13p.cpp:

   ~/mfem-3.2/examples> make ex13p
   ~/mfem-3.2/examples> mpirun -np 4 ./ex13p -m ../data/star.mesh

   and that worked fine too.

Maybe you can try these steps (excluding 1, of course) and re-build hypre, metis and mfem from scratch.

Hope this helps,
Veselin

[mlstowell]

Hello, Babai,

First I'd like to thank Veselin for confirming that MFEM has been tested with OpenMPI. That is good to know.

I have an ARPACK wrapper for MFEM which is not yet in the main branch of the code and I've spent some time this weekend updating and testing this wrapper. Unfortunately it does a rather poor job computing the electromagnetic eigenmodes. HYPRE's AME eigenvalue solver is considerably faster and more accurate.

As far as METIS is concerned, I typically use METIS version 4.0.1. I also have version 5.1.0 which is part of ParMETIS version 4.0.3. I just tested the most recent MFEM release with METIS version 5.1.0 and it performed just fine. To do this I had to set MFEM_USE_METIS_5 in both config/config.hpp and config/config.mk and I also had to adjust the MFEM_LIBS variable in config.mk to link with the METIS library. It sounds like you have done the same but your build fails.

In situations like this (inexplicable memory related crashes) the answer is often incompatible object files due to header file changes. In this case changing the value of MFEM_USE_METIS_5 The easiest way to fix this is to rebuild everything. I suggest you type "make clean" in the MFEM root directory and then rebuild MFEM. This may fix the problem.

I hope this helps.

Mark

[haryegechi]

Hi Veselin,
I am grateful to you for kindly checking the issue in VM. In my case, I have HYPRE-2.10.1 and METIS-5.1.0 in stead of hypre 2.10.0b. Is it a grave concern?

Hi Mark!
I understand that you are all trying. I am grateful. Now, I have one issue. I have METIS-5.1.0 while I do not have ParMETIS. I did not install it as it was not mentioned. Do I need to install it? In Example ex13p.cc, you have used ParMesh. Does it require ParMETIS to partititon from serial mesh?

Thanks a lot.
Best Regards
Babai

[v-dobrev]

Hypre 2.10.1 works fine too.

[mlstowell]

Hello, Babai,

ParMETIS is not required for the eigenvalue solvers. MFEM itself does not use ParMETIS even in parallel. I happen to have it installed because I was testing another third party library that requires it.

Mark

[haryegechi]

Dear *,

Sorry that the issue is lingering. Please excuse.

I think the issue is not related to openMP or MPICH. I have installed mpich 3.2 and compiled hypre using this mpich and mfem was compiled with MPI and METIS 5 option. Then ex13p was compiled and tested to run while I get
mpirun -np 4 ./ex13p -m ../data/square-disc.mesh

Options used:1
--mesh ../data/square-disc.mesh
--refine-serial 2
--refine-parallel 1
--order 1
--num-eigs 5
--visualization
Current memory used: 392 bytes
Maximum memory used: 392 bytes

***Memory allocation failed for SetupCtrl: ctrl->tpwgts. Requested size: 42331197669392 bytes Mesh::GeneratePartitioning: error in METIS_PartGraphKway! application called MPI_Abort(MPI_COMM_WORLD, 1) - process 3 Current memory used: 392 bytes Maximum memory used: 392 bytes

***Memory allocation failed for SetupCtrl: ctrl->tpwgts. Requested size: 42331197669392 bytes Current memory used: 392 bytes Mesh::GeneratePartitioning: error in METIS_PartGraphKway! Maximum memory used: 392 bytes

***Memory allocation failed for SetupCtrl: ctrl->tpwgts. Requested size: 42331197669392 bytes application called MPI_Abort(MPI_COMM_WORLD, 1) - process 1 Mesh::GeneratePartitioning: error in METIS_PartGraphKway! application called MPI_Abort(MPI_COMM_WORLD, 1) - process 2 Current memory used: 392 bytes Maximum memory used: 392 bytes

***Memory allocation failed for SetupCtrl: ctrl->tpwgts. Requested size: 42331197669392 bytes Mesh::GeneratePartitioning: error in METIS_PartGraphKway! application called MPI_Abort(MPI_COMM_WORLD, 1) - process 0

I checked "Mesh and Graph Partitioning" topic in METIS forum and found that many people have faced this ctrl->tpwgts and METIS_PartGraphKway related issues. The error arises from the call to METIS_PartGraphKway(gp%ncv_ib, ncon, nbocv_i, nbocv_v, i, j, &k, npart, tpwgts, ubvec, options, edgecut, part) where tpwgts ~ 1/npart type.

In one case I saw one reply to user to use "METIS 5.0".

Since it is midnight, I shall try tomorrow morning.

Best,
Babai

[mlstowell]

Hello, Babai,

We just discussed your issue at our weekly meeting. One of our colleagues has seen this same issue with METIS when building with 64 bit integers. Please double check that you rebuild METIS with 32 bit integers. I hope this will help.

Thanks,
Mark

[haryegechi]

Dear Mark,

Thanks for kindly discussing the issue at your meeting. I used 32 bit. Here is the result with MPICH 3.2

mpirun -np 4 ./ex13p -m ../data/square-disc.mesh
Options used:
--mesh ../data/square-disc.mesh
--refine-serial 2
--refine-parallel 1
--order 1
--num-eigs 5
--visualization
Current memory used: 392 bytes Current memory used: 392 bytes Maximum memory used: 392 bytes ***Memory allocation failed for SetupCtrl: ctrl->tpwgts. Requested size: 42331197669392 bytes Current memory used: 392 bytes Maximum memory used: 392 bytes ***Memory allocation failed for SetupCtrl: ctrl->tpwgts. Requested size: 42331197669392 bytes Current memory used: 392 bytes

Maximum memory used: 392 bytes ***Memory allocation failed for SetupCtrl: ctrl->tpwgts. Requested size: 42331197669392 bytes Mesh::GeneratePartitioning: error in METIS_PartGraphKway!

`Mesh::GeneratePartitioning: error in METIS_PartGraphKway!
application called MPI_Abort(MPI_COMM_WORLD, 1) - process 0

Mesh::GeneratePartitioning: error in METIS_PartGraphKway!
application called MPI_Abort(MPI_COMM_WORLD, 1) - process 2
application called MPI_Abort(MPI_COMM_WORLD, 1) - process 1 `

The problem is with caller to METIS function. I wonder how 392 bytes is fixed and cannot be changed.
Best Regards
Babai

[v-dobrev]

Hi Babai,

At this point, I'll just suggest you download Metis 4.0.3 and try building MFEM with it, as described on the Building MFEM page.

Veselin

[haryegechi]

Hi Veselin,

Sorry to reply you at late as I was on experiment. Thanks a lot for your suggestion as it worked. I used mpich 3.2 and metis 4.0.3 while example 13p compiled and ran. If I use metis 5 then it creates all sorts of problem.

By the by, Do you have any example on simulation of RF cavity?

Thanks again.
Best Regards
Babai

[mlstowell]

Hello, Babai,

We have been preparing some electromagnetic miniapps. There are currently two in MFEM and two more in development. However, the closest thing to an RF cavity simulator is the eigenmode solver in ex13p.

I am always interested to hear what new features and examples users would like to see in MFEM. Specifically what type of RF simulation are you planning to study? Time domain or Frequency domain? What types of sources? What types of boundary conditions? Will they involve lossy materials?

I would like to add this type of mini app to MFEM although we will need to finish some other projects first so it may take some time.

Thanks in advance for your input,
Mark

[TangLaoya]

Dear Mark,
I am exciting to get to know that you are working on electromagnetic issues. I tried to use the MFEM before, but I still can't find ways to let it work for my problems. I have submitted issues before (the topic is not about this, but you can find them in the replies):

#11

Could you please also develop some examples about this kind of problems?
Thanks,
Tang Laoya