Meltpool 2D simulation

[KamalnathOSU]

@GiudGiud @cticenhour @jiangwen84

Dear MALAMUTE team,

Let me be the first to initiate a discussion on this repo 🥇 !

As Guillaume suggested, I tried running the simplest case 2D first ( no MPI, no threads, just plain old serial mode). I got almost the same answer for the first two non-linear runs and then code got stuck.

The following is the output.

M     M      A      L            A      M     M  U     U  T T T T  E E E E
MM   MM     A A     L           A A     MM   MM  U     U     T     E
M M M M    A   A    L          A   A    M M M M  U     U     T     E E E E
M  M  M   A A A A   L         A A A A   M  M  M  U     U     T     E
M     M  A       A  L L L L  A       A  M     M   U U U      T     E E E E


MALAMUTE: MOOSE Application Library for Advanced Manufacturing UTilitiEs


Copyright 2021 - 2023, Battelle Energy Alliance, LLC
ALL RIGHTS RESERVED


NOTICE: These data were produced by BATTELLE ENERGY ALLIANCE, LLC under Contract
No. DE-AC07-05ID14517 with the Department of Energy. For ten(10) years from
July 8, 2021, the Government is granted for itself and others acting on its behalf
a nonexclusive, paid-up, irrevocable worldwide license in this data to reproduce,
prepare derivative works, and perform publicly and display publicly, by or on
behalf of the Government. There is provision for the possible extension of the
term of this license. Subsequent to that period or any extension granted, the
Government is granted for itself and others acting on its behalf a nonexclusive,
paid-up, irrevocable worldwide license in this data to reproduce, prepare
derivative works, distribute copies to the public, perform publicly and display
publicly, and to permit others to do so. The specific term of the license can be
identified by inquiry made to Contractor or DOE. NEITHER THE UNITED STATES NOR
THE UNITED STATES DEPARTMENT OF ENERGY, NOR ANY OF THEIR EMPLOYEES, MAKES ANY
WARRANTY, EXPRESS OR IMPLIED, OR ASSUMES ANY LEGAL LIABILITY OR RESPONSIBILITY
FOR THE ACCURACY, COMPLETENESS, OR USEFULNESS OF ANY DATA, APPARATUS, PRODUCT,
OR PROCESS DISCLOSED, OR REPRESENTS THAT ITS USE WOULD NOT INFRINGE PRIVATELY
OWNED RIGHTS.



reinit0:
reinit0:
reinit0: M     M      A      L            A      M     M  U     U  T T T T  E E E E
reinit0: MM   MM     A A     L           A A     MM   MM  U     U     T     E
reinit0: M M M M    A   A    L          A   A    M M M M  U     U     T     E E E E
reinit0: M  M  M   A A A A   L         A A A A   M  M  M  U     U     T     E
reinit0: M     M  A       A  L L L L  A       A  M     M   U U U      T     E E E E
reinit0:
reinit0:
reinit0: MALAMUTE: MOOSE Application Library for Advanced Manufacturing UTilitiEs
reinit0:
reinit0:
reinit0: Copyright 2021 - 2023, Battelle Energy Alliance, LLC
reinit0: ALL RIGHTS RESERVED
reinit0:
reinit0:
reinit0: NOTICE: These data were produced by BATTELLE ENERGY ALLIANCE, LLC under Contract
reinit0: No. DE-AC07-05ID14517 with the Department of Energy. For ten(10) years from
reinit0: July 8, 2021, the Government is granted for itself and others acting on its behalf
reinit0: a nonexclusive, paid-up, irrevocable worldwide license in this data to reproduce,
reinit0: prepare derivative works, and perform publicly and display publicly, by or on
reinit0: behalf of the Government. There is provision for the possible extension of the
reinit0: term of this license. Subsequent to that period or any extension granted, the
reinit0: Government is granted for itself and others acting on its behalf a nonexclusive,
reinit0: paid-up, irrevocable worldwide license in this data to reproduce, prepare
reinit0: derivative works, distribute copies to the public, perform publicly and display
reinit0: publicly, and to permit others to do so. The specific term of the license can be
reinit0: identified by inquiry made to Contractor or DOE. NEITHER THE UNITED STATES NOR
reinit0: THE UNITED STATES DEPARTMENT OF ENERGY, NOR ANY OF THEIR EMPLOYEES, MAKES ANY
reinit0: WARRANTY, EXPRESS OR IMPLIED, OR ASSUMES ANY LEGAL LIABILITY OR RESPONSIBILITY
reinit0: FOR THE ACCURACY, COMPLETENESS, OR USEFULNESS OF ANY DATA, APPARATUS, PRODUCT,
reinit0: OR PROCESS DISCLOSED, OR REPRESENTS THAT ITS USE WOULD NOT INFRINGE PRIVATELY
reinit0: OWNED RIGHTS.
reinit0:
reinit0:

The following total 13 aux variables:
  delta_function
  enthalpy
  heaviside_function
  liquid_mass_fraction
  liquid_volume_fraction
  melt_pool_mass_rate
  mu
  permeability
  rho
  solid_mass_fraction
  solid_volume_fraction
  specific_heat
  thermal_conductivity
are added for automatic output by MaterialOutputAction.
Framework Information:
MOOSE Version:           git commit 459b699a8d on 2023-04-06
LibMesh Version:         80494219d2293e467392189814d30330c4f87a25
PETSc Version:           3.16.5
SLEPc Version:           3.16.2
Current Time:            Mon Apr 24 14:57:43 2023
Executable Timestamp:    Mon Apr 24 01:30:16 2023

Parallelism:
  Num Processors:          1
  Num Threads:             1

Mesh:
  Parallel Type:           replicated
  Mesh Dimension:          2
  Spatial Dimension:       2
  Nodes:                   2601
  Elems:                   2500
  Num Subdomains:          1

Nonlinear System:
  Num DOFs:                20808
  Num Local DOFs:          20808
  Variables:               { "ls" "temp" } { "grad_ls" "velocity" } { "p" "curvature" }
  Finite Element Types:    "LAGRANGE" "LAGRANGE_VEC" "LAGRANGE"
  Approximation Orders:    "FIRST" "FIRST" "FIRST"

Auxiliary System:
  Num DOFs:                35000
  Num Local DOFs:          35000
  Variables:               { "box" "delta_function" "enthalpy" "heaviside_function" "liquid_mass_fraction"
                             ... "rho" "solid_mass_fraction" "solid_volume_fraction" "specific_heat"
                             "thermal_conductivity" }
  Finite Element Types:    "MONOMIAL"
  Approximation Orders:    "CONSTANT"

Execution Information:
  Executioner:             Transient
  TimeStepper:             ConstantDT
  TimeIntegrator:          ImplicitEuler
  Solver Mode:             NEWTON
  MOOSE Preconditioner:    SMP

Currently Executing
  Performing Initial Setup
    Performing Initial Adaptivity
      Finished Handling Mesh Changes                                                     [  1.02 s] [  265 MB]
      Finished Handling Mesh Changes                                                     [  3.84 s] [  666 MB]
    Finished Performing Initial Adaptivity                                               [ 11.32 s] [  666 MB]
reinit0: Parallelism:
reinit0:   Num Processors:          1
reinit0:   Num Threads:             1
reinit0:
reinit0: Mesh:
reinit0:   Parallel Type:           replicated
reinit0:   Mesh Dimension:          2
reinit0:   Spatial Dimension:       2
reinit0:   Nodes:                   2601
reinit0:   Elems:                   2500
reinit0:   Num Subdomains:          1
reinit0:
reinit0: Nonlinear System:
reinit0:   Num DOFs:                7803
reinit0:   Num Local DOFs:          7803
reinit0:   Variables:               "ls" "grad_ls"
reinit0:   Finite Element Types:    "LAGRANGE" "LAGRANGE_VEC"
reinit0:   Approximation Orders:    "FIRST" "FIRST"
reinit0:
reinit0: Auxiliary System:
reinit0:   Num DOFs:                5101
reinit0:   Num Local DOFs:          5101
reinit0:   Variables:               "ls_0" "box"
reinit0:   Finite Element Types:    "LAGRANGE" "MONOMIAL"
reinit0:   Approximation Orders:    "FIRST" "CONSTANT"
reinit0:
reinit0: Execution Information:
reinit0:   Executioner:             Transient
reinit0:   TimeStepper:             ConstantDT
reinit0:   TimeIntegrator:          ImplicitEuler
reinit0:   Solver Mode:             NEWTON
reinit0:   MOOSE Preconditioner:    SMP (auto)
reinit0:
  Finished Performing Initial Setup                                                      [ 20.36 s] [  664 MB]

Time Step 0, time = 0

Time Step 1, time = 0.001, dt = 0.001

Performing automatic scaling calculation


    Computing Automatic Scaling
      Computing Jacobian..                                                               [ 15.24 s] [  667 MB]
    Finished Computing Automatic Scaling                                                 [ 15.34 s] [  667 MB]
 0 Nonlinear |R| = 7.961791e-05
    Computing Jacobian..                                                                 [ 18.77 s] [  829 MB]
      0 Linear |R| = 7.961791e-05
      1 Linear |R| = 7.104785e-06
 1 Nonlinear |R| = 8.609550e+05

    Computing Jacobian..                                                                 [ 18.05 s] [ 2792 MB]
      0 Linear |R| = 8.609550e+05
      1 Linear |R| = 7.767379e-06
 2 Nonlinear |R| = 1.410181e+01

    Computing Jacobian..                                                                 [ 18.05 s] [ 6370 MB]

Based on the earlier discussion on the MOOSE repo, I also checked the memory requirement. I had requested 32GB, 8-core node on the job scheduler and only about 10 GB were used for the calculation as shown in the image .

Does any of you have the expect simulation result as video ? I am curious to see the meltpool evolution.

[KamalnathOSU]

I also want to mention that in "devel" branch , the moose sub-module folder is empty and hence I am using the moose repo which I got from the latest docker image, to compile malamute.

[jiangwen84]

@KamalnathOSU Did you try @GiudGiud's suggestion to use only MPI without threads? I only tested with MPI before. It could be the solver is not robust with multi threads for this problem. I will take a closer look.

[lindsayad]

He said he's not running with mpi nor threads

[jiangwen84]

My bad. I just read his previous post on MOOSE discussion.

[KamalnathOSU]

@lindsayad Thank you.

[KamalnathOSU]

I didn't try that case. I will try that and update you. -Kamal Get Outlook for iOS<https://aka.ms/o0ukef>

…

________________________________ From: Wen Jiang ***@***.***> Sent: Monday, April 24, 2023 4:56:58 PM To: idaholab/malamute ***@***.***> Cc: Kadirvel, Kamal ***@***.***>; Mention ***@***.***> Subject: Re: [idaholab/malamute] Meltpool 2D simulation (Discussion #67) @KamalnathOSU Did you try @GiudGiud's suggestion to use only MPI without threads? I only tested with MPI before. It could be the solver is not robust with multi threads for this problem. I will take a closer look. — Reply to this email @KamalnathOSU<https://urldefense.com/v3/__https://github.com/KamalnathOSU__;!!KGKeukY!0-8KvZGXAy9PcNqgrFoo5LDLKprusszGH7Y_IOE_Aerf9JY9aFMcB0qpl3uT3EEc36ig2gT9ibvJ3DzM892r7FK4S1of1jENv-Ou$> Did you try @GiudGiud<https://urldefense.com/v3/__https://github.com/GiudGiud__;!!KGKeukY!0-8KvZGXAy9PcNqgrFoo5LDLKprusszGH7Y_IOE_Aerf9JY9aFMcB0qpl3uT3EEc36ig2gT9ibvJ3DzM892r7FK4S1of1k_RK5In$>'s suggestion to use only MPI without threads? I only tested with MPI before. It could be the solver is not robust with multi threads for this problem. I will take a closer look. — Reply to this email directly, view it on GitHub<https://urldefense.com/v3/__https://github.com/idaholab/malamute/discussions/67*discussioncomment-5713192__;Iw!!KGKeukY!0-8KvZGXAy9PcNqgrFoo5LDLKprusszGH7Y_IOE_Aerf9JY9aFMcB0qpl3uT3EEc36ig2gT9ibvJ3DzM892r7FK4S1of1ss3ju0h$>, or unsubscribe<https://urldefense.com/v3/__https://github.com/notifications/unsubscribe-auth/AIUJ4LK67P5XPQYAKUCNA3TXC3ZKVANCNFSM6AAAAAAXKA4ZOQ__;!!KGKeukY!0-8KvZGXAy9PcNqgrFoo5LDLKprusszGH7Y_IOE_Aerf9JY9aFMcB0qpl3uT3EEc36ig2gT9ibvJ3DzM892r7FK4S1of1t_v8rAi$>. You are receiving this because you were mentioned.Message ID: ***@***.***>

[jiangwen84]

@lindsayad I was a bit struggling with the DIVERGED_FNORM_NAN error. I tried both options you pointed out, and I did not see any Exception/Error caught by PETSc/MOOSE.

Actually I ran into this problem a while ago. I did search online, and only useful information is actually your post on PETSc couple of years ago. Looks like MUMPS helped you at that time. We did not have any luck with MUMPS for our problem.

Since it has been a few years now, I wonder if you have any better ideas what is causing this issue. Maybe invert of some zero value? The problem can eventually converge with smaller dt. If we can get the bottom of this problem, we can potentially solve this problem faster by taking larger dt.

[lindsayad]

Can you link me to the PETSc post?

[jiangwen84]

here you go: https://lists.mcs.anl.gov/pipermail/petsc-users/2015-September/027075.html

[lindsayad]

Haha wow that's way back there. I'll try running this input

[jiangwen84]

@lindsayad thanks. I made a new PR to improve code robustness. see #70

The new PR makes the solve much better, but still you will get DIVERGED_FNORM_NAN at later stages. In addition, I just cannot make any iterative solver work for this problem. If you can help look at these two issues, that would be greatly helpful.

[KamalnathOSU]

@lindsayad @jiangwen84
Hi guys,
Does the meltpool 2D simulation work now ?

[KamalnathOSU]

I think there is still something wrong with the inputfile. After about 7.5hrs on 8 MPI processess, the simulation only reached t=0.196s. The "ls" variable only showed negligible change but the liquid volume fraction showed a significant change. Is this something you saw on your end ?

[jiangwen84]

We got the same results with the parameters provided by this input file. You can consider modifying some parameters.

[KamalnathOSU]

Hi,
As the simulation was quite slow on my end, I was worried that the solver is not working properly. I just wanted to confirm whether my compilation of Malamute is proper. If you got the same result then I malamute is set up properly in my end. As the simulation is quite slow on my end, I want to use more MPI processes.

I am running the calculations in Pitzer (Ohio Supercomputer cluster) where each computational nodes has 40 cores ( Intel Xeon 6148s). At the beginning of the simulation "Number of Local DOFs" is 2840 which is quite small. In the Summer workshop 2020, Derek recommended around (10,000 DOFs)/MPI. I guess due to adaptive grid refinement, many more DOFs would be added during the simulation. My questions is, whether these additional DOFs (due to more nodes) will be split equally across all the MPI processes ?

The following is my current setup for running the 2D meltpool simulations.

Parallelism:
  Num Processors:          8
  Num Threads:             1

Mesh:
  Parallel Type:           replicated
  Mesh Dimension:          2
  Spatial Dimension:       2
  Nodes:
    Total:                 2601
    Local:                 355
    Min/Max/Avg:           299/355/325
  Elems:
    Total:                 2500
    Local:                 313
    Min/Max/Avg:           312/313/312
  Num Subdomains:          1
  Num Partitions:          8
  Partitioner:             metis

Nonlinear System:
  Num DOFs:                20808
  Num Local DOFs:          2840
  Variables:               { "ls" "temp" } { "grad_ls" "velocity" } { "p" "curvature" }
  Finite Element Types:    "LAGRANGE" "LAGRANGE_VEC" "LAGRANGE"
  Approximation Orders:    "FIRST" "FIRST" "FIRST"

Auxiliary System:
  Num DOFs:                37500
  Num Local DOFs:          4695
  Variables:               { "box" "delta_function" "enthalpy" "heaviside_function" "liquid_mass_fraction"
                             ... "saturated_vapor_pressure" "solid_mass_fraction" "solid_volume_fraction"
                             "specific_heat" "thermal_conductivity" }
  Finite Element Types:    "MONOMIAL"
  Approximation Orders:    "CONSTANT"

Execution Information:
  Executioner:             Transient
  TimeStepper:             ConstantDT
  TimeIntegrator:          ImplicitEuler
  Solver Mode:             NEWTON
  MOOSE Preconditioner:    SMP

[lindsayad]

We try to do a reasonable load balancing but it's definitely not perfect. But I would profile and see what is slow. That's always better than guessing

[lindsayad]

See https://mooseframework.inl.gov/application_development/profiling.html

[KamalnathOSU]

Great. Will run it today. -Kamal From: Wen Jiang ***@***.***> Sent: Wednesday, May 17, 2023 1:50 PM To: idaholab/malamute ***@***.***> Cc: Kadirvel, Kamal ***@***.***>; Mention ***@***.***> Subject: Re: [idaholab/malamute] Meltpool 2D simulation (Discussion #67) It should work now. Let us know if it does not. Thanks. — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you were mentioned. Message ID: <idaholab/malamute/repo-discussions/67/comments/5931073@ github. com> It should work now. Let us know if it does not. Thanks. — Reply to this email directly, view it on GitHub<https://urldefense.com/v3/__https:/github.com/idaholab/malamute/discussions/67*discussioncomment-5931073__;Iw!!KGKeukY!ysCyD5zaWe0lg4lZRk1ATPJodaXpa4-1FRi2oTMIW4XvzxuFPKOFZrdS2CC_sPszVflv3pP2KEjHqIM1picTSRyHrQr_42W2hlRQ$>, or unsubscribe<https://urldefense.com/v3/__https:/github.com/notifications/unsubscribe-auth/AIUJ4LJOHTCB4UE44OQPTJLXGUMVJANCNFSM6AAAAAAXKA4ZOQ__;!!KGKeukY!ysCyD5zaWe0lg4lZRk1ATPJodaXpa4-1FRi2oTMIW4XvzxuFPKOFZrdS2CC_sPszVflv3pP2KEjHqIM1picTSRyHrQr_43qdVL0e$>. You are receiving this because you were mentioned.Message ID: ***@***.******@***.***>>

[KamalnathOSU]

Hi all,
I need some help in understanding the input file and the configuration setup.

From meltpool.i this is the simulation geometry (shown below) I gather

As the laser moves towards right, it creates meltpool which solidifies after the laser moves away. As per my understanding, the parameter "ls" tracks the boundary between gas phase and condensed phase (solid+liquid). ls=0 is condensed phase and ls=1 is the gas phase.

But during the simulation I observe that even when ls=1 (gas phase), the liquid volume fraction in that location can become 1. This does not make sense to me. Can you tell me why this is happening ? Also, is there any output variable in the exodus file which can differentiate between solid, liquid and gas simultaneously ? Thermal conductivity showed some contrast, but a non-dimensional order parameter style variable would be better.

The following are the results that I got at t=0.221s.

[jiangwen84]

Those are great questions.

The liquid volume fraction is determined by the temperature, so you would see the it can be 1 even in the gas phase. However, the volume fraction in the gas phase is not typically used by any physics. Many of the PDE terms will be multiplied by a Heaviside function, which makes their contribution to the gas phase zero.

For visualization, you can 1) write a new material to identify the different regions by using some calculated quantities, such as level set value, volume fraction, etc. Or 2) apply filter (clip, threshold) in Paraview to get the desirable output.

[KamalnathOSU]

Understood. Thank you.

Just one more thing to confirm. The model parameters used here are from Table 2 of Xie, Z., Jiang, W., Wang, C., & Wu, X. (2022). Bayesian inverse uncertainty quantification of a MOOSE-based melt pool model for additive manufacturing using experimental data. Annals of Nuclear Energy, 165, 108782. right ? The 2D simulation here is the simplified version of the 3D simulation presented in this work ?

[jiangwen84]

Most of the parameters should be the same. In that paper, we did not evolve the level set, so the interface remains flat. By doing that, it is easier to "measure" the depth of the melt pool.