Nexus: enable improved generation of QMCPACK inputs #1471

jtkrogel · 2019-03-22T15:08:01Z

This PR pulls in QMCPACK input generation methods from qmcpack_worfklows into the main code (now made accessible via generate_qmcpack). This update is fully backwards compatible with existing Nexus scripts, as confirmed by passing tests for unchanged example inputs.

The main advantages of the changes are to enable more compact user input to request sensible defaults for initial Jastrow factors and OPT/VMC/DMC calculation sections.

For example, good default Jastrow factors are generated by setting, e.g. J2=True or J3=True in generate_qmcpack rather than providing more detailed inputs via the jastrows keyword:

Input:
generate_qmcpack(...,J3=True)

Output:
         <jastrow type="One-Body" name="J1" function="bspline" source="ion0" print="yes">
            <correlation elementType="C" size="10" rcut="4.7703702464" cusp="0.0">
               <coefficients id="eC" type="Array">                  
0 0 0 0 0 0 0 0 0 0
               </coefficients>
            </correlation>
         </jastrow>
         <jastrow type="Two-Body" name="J2" function="bspline" print="yes">
            <correlation speciesA="u" speciesB="u" size="10" rcut="4.7703702464">
               <coefficients id="uu" type="Array">                  
0 0 0 0 0 0 0 0 0 0
               </coefficients>
            </correlation>
            <correlation speciesA="u" speciesB="d" size="10" rcut="4.7703702464">
               <coefficients id="ud" type="Array">                  
0 0 0 0 0 0 0 0 0 0
               </coefficients>
            </correlation>
         </jastrow>
         <jastrow type="eeI" name="J3" function="polynomial" print="yes" source="ion0">
            <correlation ispecies="C" especies1="u" especies2="u" isize="3" esize="3" rcut="4.7703702464">
               <coefficients id="uuC" type="Array" optimize="yes"/>
            </correlation>
            <correlation ispecies="C" especies1="u" especies2="d" isize="3" esize="3" rcut="4.7703702464">
               <coefficients id="udC" type="Array" optimize="yes"/>
            </correlation>
         </jastrow>

The default cutoffs for J1/J2 are the n.n. wigner radius and the number of B-spline coeffecients is one per 0.5 Bohr.

QMC sections are autogenerated by setting qmc='XXX' instead of providing structured input via calculations. Examples for optimization (quartic/oneshiftonly), VMC, and DMC (direct and timestep extrapolation) are shown below.

Optimization:

Input:
generate_qmcpack(...,qmc='opt',minmethod='quartic',cost='energy')

Output:
   <loop max="4">
      <qmc method="linear" move="pbyp" checkpoint="-1">
         <cost name="energy"              >    0.0                </cost>
         <cost name="unreweightedvariance">    1.0                </cost>
         <cost name="reweightedvariance"  >    0.0                </cost>
         <parameter name="warmupSteps"         >    300                </parameter>
         <parameter name="blocks"              >    100                </parameter>
         <parameter name="steps"               >    1                  </parameter>
         <parameter name="subSteps"            >    10                 </parameter>
         <parameter name="timestep"            >    0.3                </parameter>
         <parameter name="useDrift"            >    no                 </parameter>
         <parameter name="samples"             >    204800             </parameter>
         <parameter name="MinMethod"           >    quartic            </parameter>
         <parameter name="minwalkers"          >    0.3                </parameter>
         <parameter name="nonlocalpp"          >    yes                </parameter>
         <parameter name="useBuffer"           >    yes                </parameter>
         <parameter name="alloweddifference"   >    0.0001             </parameter>
         <parameter name="exp0"                >    -6                 </parameter>
         <parameter name="bigchange"           >    10.0               </parameter>
         <parameter name="stepsize"            >    0.15               </parameter>
         <parameter name="nstabilizers"        >    1                  </parameter>
         <estimator name="LocalEnergy" hdf5="yes"/>
      </qmc>
   </loop>
   <loop max="12">
      <qmc method="linear" move="pbyp" checkpoint="-1">
         <cost name="energy"              >    1.0                </cost>
         <cost name="unreweightedvariance">    0.0                </cost>
         <cost name="reweightedvariance"  >    0.0                </cost>
         <parameter name="warmupSteps"         >    300                </parameter>
         <parameter name="blocks"              >    100                </parameter>
         <parameter name="steps"               >    1                  </parameter>
         <parameter name="subSteps"            >    10                 </parameter>
         <parameter name="timestep"            >    0.3                </parameter>
         <parameter name="useDrift"            >    no                 </parameter>
         <parameter name="samples"             >    204800             </parameter>
         <parameter name="MinMethod"           >    quartic            </parameter>
         <parameter name="minwalkers"          >    0.3                </parameter>
         <parameter name="nonlocalpp"          >    yes                </parameter>
         <parameter name="useBuffer"           >    yes                </parameter>
         <parameter name="alloweddifference"   >    0.0001             </parameter>
         <parameter name="exp0"                >    -6                 </parameter>
         <parameter name="bigchange"           >    10.0               </parameter>
         <parameter name="stepsize"            >    0.15               </parameter>
         <parameter name="nstabilizers"        >    1                  </parameter>
         <estimator name="LocalEnergy" hdf5="yes"/>
      </qmc>
   </loop>

Input:
generate_qmcpack(...,qmc='opt',minmethod='oneshiftonly')

Output:
   <loop max="12">
      <qmc method="linear" move="pbyp" checkpoint="-1">
         <cost name="energy"              >    0.0                </cost>
         <cost name="unreweightedvariance">    1.0                </cost>
         <cost name="reweightedvariance"  >    0.0                </cost>
         <parameter name="warmupSteps"         >    300                </parameter>
         <parameter name="blocks"              >    100                </parameter>
         <parameter name="steps"               >    1                  </parameter>
         <parameter name="subSteps"            >    10                 </parameter>
         <parameter name="timestep"            >    0.3                </parameter>
         <parameter name="useDrift"            >    no                 </parameter>
         <parameter name="samples"             >    204800             </parameter>
         <parameter name="MinMethod"           >    OneShiftOnly         </parameter>
         <parameter name="minwalkers"          >    0.3                </parameter>
         <parameter name="nonlocalpp"          >    yes                </parameter>
         <parameter name="shift_i"             >    0.01               </parameter>
         <parameter name="shift_s"             >    1.0                </parameter>
         <estimator name="LocalEnergy" hdf5="yes"/>
      </qmc>
   </loop>

VMC:

Input:
generate_qmcpack(...,qmc='vmc')

Output:
   <qmc method="vmc" move="pbyp" checkpoint="-1">
      <parameter name="walkers"             >    1               </parameter>
      <parameter name="warmupSteps"         >    50              </parameter>
      <parameter name="blocks"              >    800             </parameter>
      <parameter name="steps"               >    10              </parameter>
      <parameter name="subSteps"            >    3               </parameter>
      <parameter name="timestep"            >    0.3             </parameter>
      <estimator name="LocalEnergy" hdf5="yes"/>
   </qmc>

DMC:

Input:
generate_qmcpack(...,qmc='dmc')

Output:
   <qmc method="vmc" move="pbyp" checkpoint="-1">
      <parameter name="walkers"             >    1               </parameter>
      <parameter name="warmupSteps"         >    30              </parameter>
      <parameter name="blocks"              >    40              </parameter>
      <parameter name="steps"               >    10              </parameter>
      <parameter name="subSteps"            >    3               </parameter>
      <parameter name="timestep"            >    0.3             </parameter>
      <parameter name="samples"             >    2048            </parameter>
      <estimator name="LocalEnergy" hdf5="yes"/>
   </qmc>
   <qmc method="dmc" move="pbyp" checkpoint="-1">
      <parameter name="warmupSteps"         >    20              </parameter>
      <parameter name="blocks"              >    200             </parameter>
      <parameter name="steps"               >    10              </parameter>
      <parameter name="timestep"            >    0.01            </parameter>
      <estimator name="LocalEnergy" hdf5="yes"/>
   </qmc>

DMC timestep extrapolation:

Input:
generate_qmcpack(...,qmc='dmc',timestep=0.02,ntimesteps = 4)

Output:
   <qmc method="vmc" move="pbyp" checkpoint="-1">
      <parameter name="walkers"             >    1               </parameter>
      <parameter name="warmupSteps"         >    30              </parameter>
      <parameter name="blocks"              >    40              </parameter>
      <parameter name="steps"               >    10              </parameter>
      <parameter name="subSteps"            >    3               </parameter>
      <parameter name="timestep"            >    0.3             </parameter>
      <parameter name="samples"             >    2048            </parameter>
      <estimator name="LocalEnergy" hdf5="yes"/>
   </qmc>
   <qmc method="dmc" move="pbyp" checkpoint="-1">
      <parameter name="warmupSteps"         >    20              </parameter>
      <parameter name="blocks"              >    200             </parameter>
      <parameter name="steps"               >    10              </parameter>
      <parameter name="timestep"            >    0.02            </parameter>
      <estimator name="LocalEnergy" hdf5="yes"/>
   </qmc>
   <qmc method="dmc" move="pbyp" checkpoint="-1">
      <parameter name="warmupSteps"         >    40              </parameter>
      <parameter name="blocks"              >    200             </parameter>
      <parameter name="steps"               >    20              </parameter>
      <parameter name="timestep"            >    0.01            </parameter>
      <estimator name="LocalEnergy" hdf5="yes"/>
   </qmc>
   <qmc method="dmc" move="pbyp" checkpoint="-1">
      <parameter name="warmupSteps"         >    80              </parameter>
      <parameter name="blocks"              >    200             </parameter>
      <parameter name="steps"               >    40              </parameter>
      <parameter name="timestep"            >    0.005           </parameter>
      <estimator name="LocalEnergy" hdf5="yes"/>
   </qmc>
   <qmc method="dmc" move="pbyp" checkpoint="-1">
      <parameter name="warmupSteps"         >    160             </parameter>
      <parameter name="blocks"              >    200             </parameter>
      <parameter name="steps"               >    80              </parameter>
      <parameter name="timestep"            >    0.0025          </parameter>
      <estimator name="LocalEnergy" hdf5="yes"/>
   </qmc>

markdewing · 2019-03-22T15:09:18Z

okay to test

qmc-robot · 2019-03-22T15:11:04Z

Can one of the maintainers verify this patch?

ye-luo · 2019-03-22T20:31:42Z

What is the main reason for yes <estimator name="LocalEnergy" hdf5="yes"/>

ye-luo · 2019-03-22T20:36:08Z

Please also remove in OneShiftOnly

         <cost name="energy"              >    0.0                </cost>
         <cost name="unreweightedvariance">    1.0                </cost>
         <cost name="reweightedvariance"  >    0.0                </cost>
         <parameter name="shift_i"             >    0.01               </parameter>
         <parameter name="shift_s"             >    1.0                </parameter>

Please follow the example in the manual for one shift only. Two optimization sections are recommended.

jtkrogel · 2019-03-22T20:47:45Z

With hdf5=yes the scalar data is written to the stat.h5.

Why should the cost function be removed? It is simply user choice. The shift values are just the defaults, so I don't see why this is harmful.

I can enable a way to use separate minwalkers if desired.

ye-luo · 2019-03-22T21:04:12Z

Minimal I/O is preferred.
Changing cost function fraction doesn't affect the OneShiftOnly optimizer. It doesn't affect its decision. The default values are good enough.
If paramters are the same as the default, they should not list. If the default is changed, it should be propagated instead of being blocked by the workflow setting.

jtkrogel · 2019-03-27T11:53:17Z

I don't personally see why it's such a big issue, but I am willing to change (1) and (3) according to @ye-luo's requests.

I'm confused about your statements regarding the cost function. Is oneshift only ever optimizing according to a fixed cost function regardless of user input? If so, which is it (energy?)?

ye-luo · 2019-03-27T14:57:09Z

Quartic optimizer makes the decision whether a new set of parameter is accepted or ignored based on fitting the CostFunction values to a quartic function. OneShiftOnly is pure energy minimization and has a different strategy. It almost always accept new parameter set unless miniwalkers screens crazy bad parameters. CostFunction is calculated and printed only for user to have a feeling of the optimization but it doesn't get involved in any decision making of OSO.
It is a user choice in general but unnecessary for OSO. Users are not able to influence the optimizer by changing the input parameters. Thus, it is better not to put it in the input example. The default value is 0.9 energy + 0.1 reweightedvariance. Your input example has reweightedvariance = 1.0 just confuses users that it looks like a variance minimization.

jtkrogel · 2019-03-27T15:18:59Z

OK, so OSO cost function is fixed internally regardless of user input.

Is it fixed to pure energy (1.0energy) or 0.9energy+0.1*reweightedvariance? You mention both above.

(NB: recent commits address (1) above)

ye-luo · 2019-03-27T15:31:54Z

The optimization has two stages.

In the first stage, the optimizer just compute WF and LocalEnergy derivatives and propose a new set of parameters. All choices of optimizer are doing linear method energy minimization rescale/quartic/OSO.
There is no cost function involved.

In the second stage,
quartic:
Compute the cost function values and fit them to a quartic function and make a decision.
OSO:
Compute the cost function value and print it. The decision making is only based on minwalkers.

ye-luo · 2019-03-27T15:34:21Z

@jtkrogel It is not fixed because the energy minimization doesn't even know the costfunction. user input of cost function is respected but used differently.

jtkrogel · 2019-03-27T15:36:37Z

OK, perhaps we are conflating two things: cost function as input, and the actual target of optimization.

Does OSO find the lowest energy solution? If not energy, what is it minimizing by default? Is there (and what is) a way to influence what it is minimizing via user input?

ye-luo · 2019-03-27T15:44:45Z

In the first stage, OSO is targetting energy. In the second stage, OSO doesn't pursuit the absolute lowest CostFunction because each VMC section has limited samples and the energy fluctuates. The parameter set gives the lowest CostFunction is not necessarily the best parameter set. For this reason, OSO let the optimization continue while the quartic struggles with the `optimal' CostFunction. As long as stage 1 is correct, OSO should converge statistically.

jtkrogel · 2019-03-27T16:52:31Z

See #1494 (the final result of this line of discussion via phone).

Next, I will make commits addressing points 2 and 3 above.

…nwalkers loop

jtkrogel · 2019-03-27T20:46:37Z

Points 2 and 3 have been addressed. Cost function and defaults are now suppressed for OneShiftOnly. A simple means is added to request a split loop w/ smaller and larger minwalkers as follows:

Input:
generate_qmcpack(...,qmc='opt',minmethod='oneshiftonly',init_cycles=6)

Output:
   <loop max="6">
      <qmc method="linear" move="pbyp" checkpoint="-1">
         <parameter name="warmupSteps"         >    300                </parameter>
         <parameter name="blocks"              >    100                </parameter>
         <parameter name="steps"               >    1                  </parameter>
         <parameter name="subSteps"            >    10                 </parameter>
         <parameter name="timestep"            >    0.3                </parameter>
         <parameter name="useDrift"            >    no                 </parameter>
         <parameter name="samples"             >    204800             </parameter>
         <parameter name="MinMethod"           >    OneShiftOnly         </parameter>
         <parameter name="minwalkers"          >    0.0001             </parameter>
         <parameter name="nonlocalpp"          >    yes                </parameter>
      </qmc>
   </loop>
   <loop max="12">
      <qmc method="linear" move="pbyp" checkpoint="-1">
         <parameter name="warmupSteps"         >    300                </parameter>
         <parameter name="blocks"              >    100                </parameter>
         <parameter name="steps"               >    1                  </parameter>
         <parameter name="subSteps"            >    10                 </parameter>
         <parameter name="timestep"            >    0.3                </parameter>
         <parameter name="useDrift"            >    no                 </parameter>
         <parameter name="samples"             >    204800             </parameter>
         <parameter name="MinMethod"           >    OneShiftOnly         </parameter>
         <parameter name="minwalkers"          >    0.5                </parameter>
         <parameter name="nonlocalpp"          >    yes                </parameter>
      </qmc>
   </loop>

jtkrogel · 2019-03-27T20:47:00Z

@ye-luo OK to merge now?

ye-luo · 2019-03-27T23:50:25Z

I feel there is some inconsistency between blocks, steps and samples. At least one parameter will be dropped. But it is a overall parameter design problem.

jtkrogel added 9 commits March 21, 2019 06:37

nexus: enable flat jastrow inputs and improve defaults

49dc4f3

nexus: better control of qmcpack defaults

e837c71

nexus: improve logic

968eeb5

nexus: start method defaults

9e93f73

nexus: small update

a233212

nexus: fixes for default jastrow generation

e5ec4e1

nexus: add vmc section generation, step toward dmc

a885c3f

nexus: finish qmc method defaults

5325233

nexus: do not require input_type for generate_qmcpack

c05a329

ghost assigned jtkrogel Mar 22, 2019

ghost added the in progress label Mar 22, 2019

nexus: missed one key

e4c60d4

jtkrogel added 4 commits March 27, 2019 07:58

nexus: remove local energy from qmc methods by default

4b98723

nexus: add test logging functions

ab8f92b

nexus: simplify reference file paths

f7e55b1

nexus: improve ability to update reference files

067089c

nexus: update reference qmcpack input files, no localenergy

376ab67

nexus: suppres defaults and cost function for oneshift, enable low mi…

48904aa

…nwalkers loop

ye-luo approved these changes Mar 27, 2019

View reviewed changes

Merge branch 'develop' into nx_qmc_defaults

5b35272

ghost assigned ye-luo Mar 27, 2019

ye-luo merged commit 55ab1af into QMCPACK:develop Mar 28, 2019

ghost removed the in progress label Mar 28, 2019

jtkrogel deleted the nx_qmc_defaults branch September 16, 2019 12:12

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Nexus: enable improved generation of QMCPACK inputs #1471

Nexus: enable improved generation of QMCPACK inputs #1471

jtkrogel commented Mar 22, 2019

markdewing commented Mar 22, 2019

qmc-robot commented Mar 22, 2019

ye-luo commented Mar 22, 2019 •

edited

Loading

ye-luo commented Mar 22, 2019 •

edited

Loading

jtkrogel commented Mar 22, 2019

ye-luo commented Mar 22, 2019 •

edited

Loading

jtkrogel commented Mar 27, 2019

ye-luo commented Mar 27, 2019 •

edited

Loading

jtkrogel commented Mar 27, 2019

ye-luo commented Mar 27, 2019 •

edited

Loading

ye-luo commented Mar 27, 2019

jtkrogel commented Mar 27, 2019

ye-luo commented Mar 27, 2019

jtkrogel commented Mar 27, 2019

jtkrogel commented Mar 27, 2019

jtkrogel commented Mar 27, 2019 •

edited

Loading

ye-luo commented Mar 27, 2019

Nexus: enable improved generation of QMCPACK inputs #1471

Nexus: enable improved generation of QMCPACK inputs #1471

Conversation

jtkrogel commented Mar 22, 2019

markdewing commented Mar 22, 2019

qmc-robot commented Mar 22, 2019

ye-luo commented Mar 22, 2019 • edited Loading

ye-luo commented Mar 22, 2019 • edited Loading

jtkrogel commented Mar 22, 2019

ye-luo commented Mar 22, 2019 • edited Loading

jtkrogel commented Mar 27, 2019

ye-luo commented Mar 27, 2019 • edited Loading

jtkrogel commented Mar 27, 2019

ye-luo commented Mar 27, 2019 • edited Loading

ye-luo commented Mar 27, 2019

jtkrogel commented Mar 27, 2019

ye-luo commented Mar 27, 2019

jtkrogel commented Mar 27, 2019

jtkrogel commented Mar 27, 2019

jtkrogel commented Mar 27, 2019 • edited Loading

ye-luo commented Mar 27, 2019

ye-luo commented Mar 22, 2019 •

edited

Loading

ye-luo commented Mar 22, 2019 •

edited

Loading

ye-luo commented Mar 22, 2019 •

edited

Loading

ye-luo commented Mar 27, 2019 •

edited

Loading

ye-luo commented Mar 27, 2019 •

edited

Loading

jtkrogel commented Mar 27, 2019 •

edited

Loading