Coagulation #120

pressel · 2023-01-27T20:43:20Z

This pull request ports the MAM4 representation of Coagulation from Fortran to C++. The refactored Fortran source upon which this port is based can be found here.

A few comments on this PR that mostly arise due to the fact that we have maintained a direct port of the F90 code:

Note that 2D arrays with dimensions of num_aer and num_mode are transposed from the order assumed by haero. For example in Coagulation qaer arrays are dimensioned with as [num_aer][num_mode], where as within haero qaer arrays would be dimensioned as [num_mode][num_aer]. The transposition of indices is consistent with Aging and Gas Aerosol Exchange processes.
Like Aging and Gas Aerosol Exchange, Coagulation accesses array elements in qaer_cur that strictly speaking shouldn't exist as prognostic variables within MAM4. It is my understanding that these array elements are being used by MAM4 temporary storage for values that are used later by other processes. We must thus be careful account for this at the level of the source code where these processes are called.

pressel · 2023-01-27T23:40:37Z

@cameronrutherford @overfelt Just letting you both know about this Draft PR so you can keep your eyes on it.

src/validation/coagulation/coagulation_driver.cpp

src/tests/mam4_coagulation_unit_tests.cpp

jeff-cohere

This looks pretty good to me. I left a few minor comments. Also: Do we need an empty wet_density.hpp file?

@pressel , can you update the description for this PR with any relevant information, including anything that you'd like reviews to give more focused attention?

src/mam4xx/coagulation.hpp

src/tests/mam4_coagulation_unit_tests.cpp

odiazib · 2023-02-20T18:08:23Z

src/tests/mam4_coagulation_unit_tests.cpp

+  const auto tend_qgas0 = tends.q_gas[0];
+  auto h_prog_qgas0 = Kokkos::create_mirror_view(prog_qgas0);
+  auto h_tend_qgas0 = Kokkos::create_mirror_view(tend_qgas0);
+  Kokkos::deep_copy(h_prog_qgas0, prog_qgas0);


Check this deep_copy:
data from device is copied to host, but I believe you need to copy from host to device. Also, values in device are initialize to zeros.

@jeff-cohere Do you want to take a look at this, I essentially copied this bit of code from mam4_nucleation_unit_tests.cpp.

Maybe it would be more instructive to copy test code from calcsize or gasaerexch or another process that uses several aerosol species. The nucleation parameterization was the first one we attacked and it only really adds sulfate to the aitken mode. And the testing on the other parameterizations is definitely more thorough at this point--I'm going to have to revisit nucleation and raise it to the standard set by others!

In consultation with @jeff-cohere I'm going to opt for improving the testing beyond a GPU smoke tests in another PR.

odiazib · 2023-02-20T18:11:49Z

src/validation/coagulation/coag_1subarea.cpp

+    const int max_agepair = Coagulation::max_agepair;
+    Real qaer_cur_c[num_aero][num_modes];
+    int n = 0;
+    for (int imode = 0; imode < num_modes; ++imode) {


check validation::convert_vector_to_mass_mixing_ratios

I directly ported https://github.com/eagles-project/e3sm_mam4_refactor/blob/refactor-maint-2.0/components/eam/src/chemistry/modal_aero/modal_aero_coag.F90

There the various qaer arrays are transposed from what is assumed in validation::convert_vector_mass_mixing ratios.

I switched q_mmr[ispec][imode] to q_mmr[imode] [ispec] to be consistent with the prognostic variables in aero_config.hpp

Doing this change will require to modified the order of imode and ispec in the code.

I'm going to add a validation convenience function similar to validation::convert_mass_mixing_ratios_to_vector but that properly accounts of the transposition of mode and aerosol dimensions that is used in coagulation, aging, and gas aerosol exchange processes (I'd imagine similar transpositions occur in other parts of the code as well). I will add that function as part of another PR.

src/validation/coagulation/coag_1subarea.cpp

odiazib · 2023-02-20T18:13:05Z

src/validation/coagulation/coag_1subarea.cpp

+        qaer_del_coag_out_c);
+
+    n = 0;
+    for (int imode = 0; imode < num_modes; ++imode) {


check validation::convert_mass_mixing_ratios_to_vector

I directly ported https://github.com/eagles-project/e3sm_mam4_refactor/blob/refactor-maint-2.0/components/eam/src/chemistry/modal_aero/modal_aero_coag.F90

There the various qaer arrays are transposed from what is assumed in validation::convert_vector_mass_mixing ratios.

src/mam4xx/coagulation.hpp

odiazib · 2023-02-20T18:30:33Z

src/mam4xx/coagulation.hpp

+  const Real bijqnumj2 = haero::max(0.0, ybetaij3[2] * qnum_tavg[npca]);
+  Real decay_const = bijqnumj1 + bijqnumj2;
+
+  constexpr float epsilonx2 = std::numeric_limits<float>::epsilon() * 2.0;


Are we allow to use std functions in CUDA?

@jeff-cohere and I talked about handling this numerical limit, and I think because it is being used as a constexper it is ok. @jeff-cohere correct me if I am wrong.

Well, I don't know that this is supposed to work--the std namespace is not available to CUDA C++ as far as I know. Can someone try to build this on a GPU workstation available to them?

The std::numeric_limits() compiles and the tests pass with Cuda 11.2 and GCC 10.3 on Linux.

Well, that's interesting, though it doesn't give me a very warm and fuzzy feeling! I guess we'll find out how intentional this is as we build on more exotic platforms.

Thanks for testing, James.

I believe there is an equivalent version of std::numeric_limits::epsilon() in kokkos.

src/mam4xx/coagulation.hpp

pressel · 2023-02-20T19:11:26Z

@jeff-cohere I removed wet_density.hpp.

codecov-commenter · 2023-02-21T19:14:17Z

Codecov Report

Merging #120 (9b83463) into main (95bd06e) will decrease coverage by 16.63%.
The diff coverage is 100.00%.

@@             Coverage Diff             @@
##             main     #120       +/-   ##
===========================================
- Coverage   91.54%   74.92%   -16.63%     
===========================================
  Files          54       27       -27     
  Lines        4529     2823     -1706     
===========================================
- Hits         4146     2115     -2031     
- Misses        383      708      +325

Impacted Files	Coverage Δ
src/mam4xx/aging.hpp	`98.85% <ø> (-1.15%)`	⬇️
src/mam4xx/aero_config.hpp	`100.00% <100.00%> (ø)`
src/mam4xx/coagulation.hpp	`100.00% <100.00%> (ø)`
src/tests/mam4_coagulation_unit_tests.cpp	`100.00% <100.00%> (ø)`
src/mam4xx/merikanto2007.hpp	`0.00% <0.00%> (-100.00%)`	⬇️
src/mam4xx/vehkamaki2002.hpp	`0.00% <0.00%> (-100.00%)`	⬇️
src/mam4xx/nucleation.hpp	`25.00% <0.00%> (-41.91%)`	⬇️
src/mam4xx/calcsize.hpp	`52.27% <0.00%> (-31.82%)`	⬇️
src/mam4xx/gasaerexch_soaexch.hpp	`96.10% <0.00%> (-1.30%)`	⬇️
... and 29 more

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

Adding data tables.

c75d4d9

pressel added the microphysics label Jan 27, 2023

pressel self-assigned this Jan 27, 2023

pressel added 4 commits January 27, 2023 13:26

moving data to functions

a4502ae

Merge branch 'main' into kylepres/Coag_1

d0f50b0

Adding get getcoags.

a9f6908

getcoags_wrapper_f

e8423c6

pressel requested a review from cameronrutherford January 27, 2023 23:39

pressel added 15 commits January 27, 2023 16:35

mam_coag_aer_update

a6b48dc

clang-format

928404b

update_qnum_for_intra_and_intermodal_coag

3d0be9f

update_qnum_for_intramodal_coag

b4d6b3a

update_qnum_for_intramodal_coag

ecde42e

mam_coag_1subarea

20f1c2a

compute_tendencies

1c8dd81

broken compile

887f9b3

validation

70fc4ce

Validation running.

89a7565

Tracking branch of mam_x_validation.

4c0a6fd

Tests passing.

b955e0d

Merge branch 'main' into kylepres/Coag_1

a4384e1

Fix merge issue.

2aeb2b4

Updating validation data.

1b99b34

pressel changed the title ~~Draft - First Part of Coagulation~~ Coagulation Feb 17, 2023

pressel added 5 commits February 16, 2023 16:44

Backing up cmake file.

fde50da

Updating validation scripts.

292ac09

Clean up.

5cd926f

Clean-up.

f285861

Moving getcoags test to device.

4a9c39b

pressel requested review from jeff-cohere, mjs271, odiazib and overfelt and removed request for odiazib February 18, 2023 00:06

jeff-cohere reviewed Feb 20, 2023

View reviewed changes

src/validation/coagulation/coagulation_driver.cpp Outdated Show resolved Hide resolved

jeff-cohere reviewed Feb 20, 2023

View reviewed changes

src/tests/mam4_coagulation_unit_tests.cpp Outdated Show resolved Hide resolved

jeff-cohere reviewed Feb 20, 2023

View reviewed changes

src/tests/mam4_coagulation_unit_tests.cpp Outdated Show resolved Hide resolved

jeff-cohere reviewed Feb 20, 2023

View reviewed changes

src/tests/mam4_coagulation_unit_tests.cpp Outdated Show resolved Hide resolved

jeff-cohere approved these changes Feb 20, 2023

View reviewed changes

overfelt approved these changes Feb 20, 2023

View reviewed changes

src/mam4xx/coagulation.hpp Outdated Show resolved Hide resolved

src/mam4xx/coagulation.hpp Outdated Show resolved Hide resolved

src/mam4xx/coagulation.hpp Outdated Show resolved Hide resolved

odiazib reviewed Feb 20, 2023

View reviewed changes

pressel added 5 commits February 20, 2023 10:57

Merge branch 'main' into kylepres/Coag_1

8f4dfe8

Remove extra print statement.

864ff2a

Removing place holder unit tests.

8bc74a2

Removed extra call to compute_tendencies.

bba5b56

Remove empty file.

c8d8569

pressel added 4 commits February 20, 2023 15:01

Use haero round.

4c7a8f6

haero sqrt

ffbfe50

Updating x-validation submod.

9692464

clang format

365b908

odiazib approved these changes Feb 21, 2023

View reviewed changes

jeff-cohere mentioned this pull request Feb 21, 2023

PNNL CI Ubuntu image needs upgrade #131

Closed

pressel added 2 commits February 21, 2023 12:00

Merge branch 'main' into kylepres/Coag_1

6c82b9b

Adding license header.

9b83463

pressel merged commit 8cc0a3f into main Feb 21, 2023

pressel deleted the kylepres/Coag_1 branch February 21, 2023 20:30

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Coagulation #120

Coagulation #120

pressel commented Jan 27, 2023 •

edited

Loading

pressel commented Jan 27, 2023

jeff-cohere left a comment •

edited

Loading

odiazib Feb 20, 2023

pressel Feb 21, 2023

jeff-cohere Feb 21, 2023

pressel Feb 21, 2023

odiazib Feb 20, 2023

pressel Feb 21, 2023

odiazib Feb 21, 2023

pressel Feb 21, 2023

odiazib Feb 20, 2023

pressel Feb 21, 2023

odiazib Feb 20, 2023

pressel Feb 20, 2023

jeff-cohere Feb 20, 2023

overfelt Feb 20, 2023

jeff-cohere Feb 20, 2023

odiazib Feb 20, 2023

pressel commented Feb 20, 2023 •

edited

Loading

codecov-commenter commented Feb 21, 2023 •

edited

Loading

Coagulation #120

Coagulation #120

Conversation

pressel commented Jan 27, 2023 • edited Loading

pressel commented Jan 27, 2023

jeff-cohere left a comment • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

pressel commented Feb 20, 2023 • edited Loading

codecov-commenter commented Feb 21, 2023 • edited Loading

Codecov Report

pressel commented Jan 27, 2023 •

edited

Loading

jeff-cohere left a comment •

edited

Loading

pressel commented Feb 20, 2023 •

edited

Loading

codecov-commenter commented Feb 21, 2023 •

edited

Loading