Refactor: use less memory to calculate stress in pw base #4047

dyzheng · 2024-04-23T09:52:07Z

I have refactored stress code structure in this PR.
In case Mg16Al16, the memory cost of stress calculation from 16752 MB to 194 MB.

Linked Issue

Close #3714
Close #4158
Close #3710
Close #4026
Close #3931
Close #4031

Unit Tests and/or Case Tests for my changes

A unit test is added for each new feature or bug fix.

What's changed?

Example: My changes might affect the performance of the application under certain conditions, and I have tested the impact on various scenarios...

Any changes of core modules? (ignore if not applicable)

Example: I have added a new virtual function in the esolver base class in order to ...

… pw_stress

Qianruipku · 2024-04-24T02:12:18Z

Has the efficiency of the new algorithm been tested? Are there any test data available?

dyzheng · 2024-04-24T02:40:45Z

Has the efficiency of the new algorithm been tested? Are there any test data available?

I have not tested many cases, in Mg16Al16 case , time of stress_nl change from 94 s to 124 s, I think the performance of new method still can be improved.

Qianruipku · 2024-04-24T03:24:56Z

Perhaps the QE code can be used as a reference.

mohanchen · 2024-04-26T12:37:27Z

source/module_hamilt_pw/hamilt_pwdft/stress_func_nl.cpp

-		if (gx[ig].norm2() > 1e-9) {
+	for(int ig = 0;ig< ngy;ig++)
+    {
+        FPTYPE norm2 = gx[ig * 3] * gx[ig * 3] + gx[ig * 3 + 1] * gx[ig * 3 + 1] + gx[ig * 3 + 2] * gx[ig * 3 + 2];


ig * 3 can be calculated and stored first

mohanchen · 2024-04-26T12:37:45Z

source/module_hamilt_pw/hamilt_pwdft/stress_func_nl.cpp

+    {
+        FPTYPE norm2 = gx[ig * 3] * gx[ig * 3] + gx[ig * 3 + 1] * gx[ig * 3 + 1] + gx[ig * 3 + 2] * gx[ig * 3 + 2];
+		dg [ig] = delta * sqrt(norm2) ;
+		if (norm2 > 1e-9) {


why set this value to 1e-9?

mohanchen · 2024-04-26T12:38:22Z

source/module_hamilt_pw/hamilt_pwdft/stress_func_us.cpp

@@ -49,7 +49,8 @@ void Stress_PW<FPTYPE, Device>::stress_us(ModuleBase::matrix& sigma,
    ModuleBase::matrix dylmk0(ppcell_in->lmaxq * ppcell_in->lmaxq, npw);
    for (int ipol = 0; ipol < 3; ipol++)
    {
-        this->dylmr2(ppcell_in->lmaxq * ppcell_in->lmaxq, npw, rho_basis->gcar, dylmk0, ipol);
+        double* gcar_ptr = reinterpret_cast<double*>(rho_basis->gcar);


what's the type of gcar? do you have to use cast?

mohanchen · 2024-04-26T12:39:06Z

source/module_hamilt_pw/hamilt_pwdft/stress_func_nl.cpp

@@ -634,30 +815,32 @@ void Stress_Func<FPTYPE, Device>::dylmr2 (
 	// gx = g +/- dg


-	ModuleBase::Vector3<FPTYPE> *gx = new ModuleBase::Vector3<FPTYPE> [ngy];
+	FPTYPE *gx = new FPTYPE[3 * ngy];


can you use instead of new?

mohanchen · 2024-04-26T12:39:48Z

source/module_hamilt_pw/hamilt_pwdft/stress_func_nl.cpp

+            // loop over all G-vectors
+            for(int ig=0;ig<npw;ig++)
+            {
+                vkb_ptr[ig] -= 2.0 * ylm_ptr[ig] * vq_deri_ptr[ig] * sk_in[ig] * pref_in[ih]


ig * 3 can be calculated first? multiple of 2.0 can be done after this for loop

mohanchen · 2024-04-26T12:43:15Z

source/module_hamilt_pw/hamilt_pwdft/stress_func_nl.cpp

+        for (int ig = 0; ig < npw; ig++)
+        {
+            vq_ptr[ig] = this->Polynomial_Interpolation_nl(
+						GlobalC::ppcell.tab, 


do not use GlobalC

mohanchen · 2024-04-26T12:43:24Z

source/module_hamilt_pw/hamilt_pwdft/stress_func_nl.cpp

+std::vector<FPTYPE> Stress_Func<FPTYPE, Device>::cal_vq_deri(int it, const FPTYPE* gk, int npw)
+{
+    // calculate beta in G-space using an interpolation table
+    const int nbeta = GlobalC::ucell.atoms[it].ncpp.nbeta;


do not use GlobalC

mohanchen · 2024-04-26T12:43:35Z

source/module_hamilt_pw/hamilt_pwdft/stress_func_nl.cpp

+std::vector<FPTYPE> Stress_Func<FPTYPE, Device>::cal_vq(int it, const FPTYPE* gk, int npw)
+{
+    // calculate beta in G-space using an interpolation table
+    const int nbeta = GlobalC::ucell.atoms[it].ncpp.nbeta;


do not use GlobalC

mohanchen · 2024-04-26T12:43:46Z

source/module_hamilt_pw/hamilt_pwdft/stress_func_nl.cpp

+        gk[ig*3+2] = tmp.z;
+        FPTYPE norm = sqrt(tmp.norm2());
+        gk[3 * npw + ig] = norm * GlobalC::ucell.tpiba;
+        gk[4 * npw + ig] = norm<1e-8?0.0:1.0/norm*GlobalC::ucell.tpiba;


GlobalC likes a cancer

mohanchen · 2024-04-26T12:44:00Z

source/module_hamilt_pw/hamilt_pwdft/stress_func_nl.cpp

+template <typename FPTYPE, typename Device>
+std::vector<FPTYPE> Stress_Func<FPTYPE, Device>::cal_gk(int ik, ModulePW::PW_Basis_K* wfc_basis)
+{
+    int npw = wfc_basis->npwk[ik];


this is a const

hongriTianqi · 2024-04-28T01:35:14Z

source/module_hamilt_pw/hamilt_pwdft/stress_func_nl.cpp

 	ModuleBase::timer::tick("Stress_Func","stress_nl");
 }
+// cal_gk
+template <typename FPTYPE, typename Device>
+std::vector<FPTYPE> Stress_Func<FPTYPE, Device>::cal_gk(int ik, ModulePW::PW_Basis_K* wfc_basis)


Does this function need unit test?

hongriTianqi · 2024-04-28T01:35:34Z

source/module_hamilt_pw/hamilt_pwdft/stress_func_nl.cpp

+}
+// cal_vq
+template <typename FPTYPE, typename Device>
+std::vector<FPTYPE> Stress_Func<FPTYPE, Device>::cal_vq(int it, const FPTYPE* gk, int npw)


Does this function need unit test?

hongriTianqi · 2024-04-28T01:35:51Z

source/module_hamilt_pw/hamilt_pwdft/stress_func_nl.cpp

+
+// cal_vq_deri
+template <typename FPTYPE, typename Device>
+std::vector<FPTYPE> Stress_Func<FPTYPE, Device>::cal_vq_deri(int it, const FPTYPE* gk, int npw)


Does this function need unit test?

hongriTianqi · 2024-04-28T01:36:00Z

source/module_hamilt_pw/hamilt_pwdft/stress_func_nl.cpp

+}
+// cal_ylm
+template <typename FPTYPE, typename Device>
+std::vector<FPTYPE> Stress_Func<FPTYPE, Device>::cal_ylm(int lmax, int npw, const FPTYPE* gk_in)


Does this function need unit test?

hongriTianqi · 2024-04-28T01:36:19Z

source/module_hamilt_pw/hamilt_pwdft/stress_func_nl.cpp

+// cal_vkb
+// cpu version first, gpu version later
+template <typename FPTYPE, typename Device>
+void Stress_Func<FPTYPE, Device>::cal_vkb(


Does this function need unit test?

hongriTianqi · 2024-04-28T01:36:26Z

source/module_hamilt_pw/hamilt_pwdft/stress_func_nl.cpp

+// cal_vkb
+// cpu version first, gpu version later
+template <typename FPTYPE, typename Device>
+void Stress_Func<FPTYPE, Device>::cal_vkb_deri(


Does this function need unit test?

… pw_stress

…pw_stress

dyzheng · 2024-05-06T05:02:35Z

I will work with @grysgreat to accelerate performance in GPU/DCU, change this PR to draft.

denghuilu · 2024-05-11T03:28:53Z

source/module_hamilt_pw/hamilt_pwdft/kernels/stress_op.h

@@ -162,6 +181,23 @@ struct cal_stress_nl_op<FPTYPE, psi::DEVICE_GPU> {
                    FPTYPE* stress);
 };

+// cpu version first, gpu version later
+template <typename FPTYPE>
+struct cal_vkb_op<FPTYPE, psi::DEVICE_GPU>{


struct cal_vkb_op<FPTYPE, psi::DEVICE_GPU>{

Did this function called by any other functions ?

… pw_stress

…init func (GPU)

… pw_stress

…o pw_stress

…nto pw_stress

…init func (GPU)

…o pw_stress

dyzheng · 2024-06-07T04:18:29Z

This PR has merged changes from @grysgreat and @Religious-J , not ready for review now, I will refactor it later.

… pw_stress

dyzheng added 2 commits April 23, 2024 17:48

Refactor: use less memory to calculate stress in pw base

64ee98c

Merge branch 'develop' of github.com:deepmodeling/abacus-develop into…

faac19f

… pw_stress

dyzheng requested review from WHUweiqingzhou and Qianruipku April 23, 2024 10:07

dyzheng added 2 commits April 23, 2024 22:18

Fix: nondiagonal matrix element in PW-Stress calculation

dfc9b0f

Fix: uspp stress calculation

12b407f

mohanchen reviewed Apr 27, 2024

View reviewed changes

Merge branch 'develop' into pw_stress

260bc30

hongriTianqi reviewed Apr 28, 2024

View reviewed changes

hongriTianqi suggested changes Apr 28, 2024

View reviewed changes

mohanchen and others added 9 commits May 4, 2024 17:23

Merge branch 'develop' into pw_stress

75f01cc

Merge branch 'develop' into pw_stress

ccfd2f1

Merge branch 'develop' into pw_stress

4114e85

add gemm_op in gpu and add vkb_op

2eecc2b

delete comment

89a9ce9

Merge branch 'develop' of github.com:deepmodeling/abacus-develop into…

bb7ea83

… pw_stress

delete comment2

05d59b4

Merge branch 'pw_stress' of github.com:grysgreat/abacus-develop into …

d7d2782

…pw_stress

Fix: error in merging

cb63316

dyzheng marked this pull request as draft May 6, 2024 05:02

update ops.

95eef5f

dyzheng mentioned this pull request May 8, 2024

Out of memory bug in DCU calculation (Stress Memory) #3710

Open

16 tasks

grysgreat added 2 commits May 9, 2024 15:33

add correct vq.

158cfa2

finish stress ops!

2072611

WHUweiqingzhou requested a review from denghuilu May 11, 2024 02:46

denghuilu reviewed May 11, 2024

View reviewed changes

dyzheng and others added 22 commits May 13, 2024 11:23

Refactor: delete GlobalC used in stress_nl code

971f539

Merge branch 'develop' of github.com:deepmodeling/abacus-develop into…

144496d

… pw_stress

Fix: compiling error in CUDA

c2be080

Fix: DCU error in synchronize_ptrs

1db1b36

Fix: GlobalC in Sto_stress_PW

9e9d748

Merge branch 'develop' into pw_stress

7e848b4

add new stress and force in high speed.

2f658f2

delete time prints.

929b9f8

Merge branch 'develop' of github.com:deepmodeling/abacus-develop into…

c0757cb

… pw_stress

Fix: compiling error from merge

ea2c84e

Fix: compiling error from merge on GPU

1e9938f

remove some if(GPU)

719945a

finish force remove gpus.

49b0c9a

Memory: reduce memory allocation for hpsi and spsi in diagH_subspace_…

25e7302

…init func (GPU)

Merge branch 'develop' of github.com:deepmodeling/abacus-develop into…

97e49d0

… pw_stress

fix: compiler error

a22c4c3

Merge branch 'diagH_GPU' of github.com:Religious-J/abacus-develop int…

d0e8b7b

…o pw_stress

Merge branch 'pw_stress_dev' of github.com:grysgreat/abacus-develop i…

b33a790

…nto pw_stress

Memory: reduce memory allocation for hpsi and spsi in diagH_subspace_…

6d19e65

…init func (GPU)

Memory: reduce memory allocation for hpsi and spsi in diagH_subspace_…

d0d0f8c

…init func (GPU)

Fix: force calculation error

324c7c7

Merge branch 'diagH_GPU' of github.com:Religious-J/abacus-develop int…

7bef0cd

…o pw_stress

dyzheng added 7 commits June 7, 2024 17:02

Fix: error of Lcao_in_PW

830de64

Fix: error in GPU

6b1a778

Fix: force error

37b33b6

Merge branch 'develop' of github.com:deepmodeling/abacus-develop into…

73f0772

… pw_stress

Try: use one temp psi for cg

b15143a

fix: error in cg

00518b3

Fix: sdft error

cd03b76

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Refactor: use less memory to calculate stress in pw base #4047

Refactor: use less memory to calculate stress in pw base #4047

dyzheng commented Apr 23, 2024 •

edited by WHUweiqingzhou

Qianruipku commented Apr 24, 2024

dyzheng commented Apr 24, 2024

Qianruipku commented Apr 24, 2024

mohanchen Apr 26, 2024

mohanchen Apr 26, 2024

mohanchen Apr 26, 2024

mohanchen Apr 26, 2024

mohanchen Apr 26, 2024

mohanchen Apr 26, 2024

mohanchen Apr 26, 2024

mohanchen Apr 26, 2024

mohanchen Apr 26, 2024

mohanchen Apr 26, 2024

hongriTianqi Apr 28, 2024

hongriTianqi Apr 28, 2024

hongriTianqi Apr 28, 2024

hongriTianqi Apr 28, 2024

hongriTianqi Apr 28, 2024

hongriTianqi Apr 28, 2024

dyzheng commented May 6, 2024

denghuilu May 11, 2024

dyzheng commented Jun 7, 2024

Refactor: use less memory to calculate stress in pw base #4047

Are you sure you want to change the base?

Refactor: use less memory to calculate stress in pw base #4047

Conversation

dyzheng commented Apr 23, 2024 • edited by WHUweiqingzhou

Linked Issue

Unit Tests and/or Case Tests for my changes

What's changed?

Any changes of core modules? (ignore if not applicable)

Qianruipku commented Apr 24, 2024

dyzheng commented Apr 24, 2024

Qianruipku commented Apr 24, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dyzheng commented May 6, 2024

Choose a reason for hiding this comment

dyzheng commented Jun 7, 2024

dyzheng commented Apr 23, 2024 •

edited by WHUweiqingzhou