New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Nexus: Quantum Package interface #1093

Merged
merged 15 commits into from Oct 15, 2018

Conversation

Projects
None yet
4 participants
@jtkrogel
Copy link
Contributor

jtkrogel commented Oct 3, 2018

The purpose of this PR is to create a Nexus interface to Quantum Package. This interface will enable routine selected CI calculations with QMCPACK.

  • Lay down basic simulation classes
  • General input specification extraction
  • General input read
  • General input generation
  • General input write
  • HF workflow (QP only)
  • CIS workflow (QP only)
  • FCI workflow (QP only)

Saved for later PR when hdf5 route fully available

  • HF workflow (QP -> QMCPACK)
  • FCI workflow (QP -> QMCPACK)
@qmc-robot

This comment has been minimized.

Copy link
Collaborator

qmc-robot commented Oct 3, 2018

Can one of the maintainers verify this patch?

@wafflebot wafflebot bot added the in progress label Oct 3, 2018

@qmc-robot

This comment has been minimized.

Copy link
Collaborator

qmc-robot commented Oct 3, 2018

Can one of the maintainers verify this patch?

@jtkrogel jtkrogel changed the title [WIP] Nexus: Quantum Package interface Nexus: Quantum Package interface Oct 11, 2018

@jtkrogel

This comment has been minimized.

Copy link
Contributor

jtkrogel commented Oct 11, 2018

Ready for review and merge.

@markdewing

This comment has been minimized.

Copy link
Contributor

markdewing commented Oct 11, 2018

okay to test

@markdewing

This comment has been minimized.

Copy link
Contributor

markdewing commented Oct 11, 2018

It would have been made the review smaller and easier to understand if the changes to machines.py (and possibly related changes in ntest) were in a separate PR.
Do you have an example nexus file for running quantum package?

@jtkrogel

This comment has been minimized.

Copy link
Contributor

jtkrogel commented Oct 11, 2018

I do, but it is a work in progress. These developments are for a project with Anouar, so I would like to see if there are adjustments needed in that context before providing a user facing example.

@jtkrogel

This comment has been minimized.

Copy link
Contributor

jtkrogel commented Oct 12, 2018

Do you want to try running an example to verify the interface works (if so I can provide you one), or is this more in the interest of having a documented example for general use (see prior comment)?

@markdewing

This comment has been minimized.

Copy link
Contributor

markdewing commented Oct 12, 2018

Both reasons.
I lack a general mental model on how to map between the description of input files in the Nexus code, and an actual Nexus input file. That's why I'm often asking for example files - to try understand the Nexus library code.

@jtkrogel

This comment has been minimized.

Copy link
Contributor

jtkrogel commented Oct 12, 2018

The following is working example for an H2O molecule:

#! /usr/bin/env python

from nexus import settings,job,run_project
from nexus import generate_physical_system
from nexus import generate_quantum_package

settings(
    results       = '',
    status_only   = 0,
    generate_only = 0,
    sleep         = 3,
    machine       = 'cooley',
    account       = 'PSFMat',
    qprc          = '/soft/applications/quantum_package/quantum_package.rc',
    )

scf_job = job(nodes=4,threads=12,hours=1,queue='debug')

system = generate_physical_system(
    structure = 'H2O.xyz',
    )

qp = generate_quantum_package(
    path       = 'qp_test',
    identifier = 'h2o_scf',
    job        = scf_job,
    system     = system,
    prefix     = 'h2o',
    run_type   = 'SCF',
    ao_basis   = 'cc-pvtz',
    )

run_project()

Contents of H2O.xyz:

3

O  0.000000  0.000000  0.000000 
H  0.000000  0.757160  0.586260
H  0.000000  0.757160 -0.586260

I can run this on Cooley or Theta at ALCF as these are the two machines I have access to with a QM Package install.

The Quantum Package configuration file needs to be sourced prior to running the script:

source /soft/applications/quantum_package/quantum_package.rc
@jtkrogel

This comment has been minimized.

Copy link
Contributor

jtkrogel commented Oct 12, 2018

The Nexus script performs steps equivalent to the following to create the input for Quantum Package and run the HF in a split node configuration:

# create basic ezfio input with a cc-pvtz basis
qp_create_ezfio_from_xyz -b cc-pvtz h2o.xyz

# force all QP variables to be created and initialized
qp_edit -c h2o.ezfio

# request one and two body integrals be saved
echo "Write" > h2o.ezfio/integrals_monoelec/disk_access_ao_one_integrals
echo "Write" > h2o.ezfio/integrals_monoelec/disk_access_mo_one_integrals
echo "Write" > h2o.ezfio/integrals_bielec/disk_access_ao_integrals
echo "Write" > h2o.ezfio/integrals_bielec/disk_access_mo_integrals

# run HF job
export OMP_NUM_THREADS=12
mpirun -np 1 qp_run SCF h2o.ezfio >h2o_scf.output 2>h2o_scf.error&
sleep 30
mpirun -np 3 qp_run -slave qp_ao_ints h2o.ezfio >h2o_scf_slave.output 2>h2o_scf_slave.error
@jtkrogel

This comment has been minimized.

Copy link
Contributor

jtkrogel commented Oct 12, 2018

Quantum Package's approach to input is somewhat unique. The "input file" is actually a directory tree with each input variable cached in a separate ASCII file.

This directory/file structure also represents the internal state of the program, and so subsequent runs, e.g. to perform CI singles or selected CI, require updating input variables stored in these files. Following a run, some of these files/variables change value.

Below is the result of running "tree" on an ezfio input directory:

h2o.ezfio/
├── ao_basis
│   ├── ao_basis
│   ├── ao_cartesian
│   ├── ao_md5
│   └── ao_num
├── davidson
│   ├── davidson_sze_max
│   ├── disk_based_davidson
│   ├── distributed_davidson
│   ├── n_states_diag
│   ├── state_following
│   └── threshold_davidson
├── determinants
│   ├── n_det
│   ├── n_det_max
│   ├── n_det_max_jacobi
│   ├── n_det_max_property
│   ├── n_det_max_stored
│   ├── n_states
│   ├── only_single_double_dm
│   ├── read_wf
│   ├── s2_eig
│   ├── store_full_h_mat
│   ├── target_energy
│   ├── threshold_generators
│   └── threshold_selectors
├── electrons
│   ├── elec_alpha_num
│   └── elec_beta_num
├── ezfio
│   ├── creation
│   ├── library
│   └── user
├── full_ci_zmq
│   └── iterative_save
├── hartree_fock
│   ├── energy
│   ├── level_shift
│   ├── max_dim_diis
│   ├── mo_guess_type
│   ├── n_it_scf_max
│   ├── no_oa_or_av_opt
│   ├── scf_algorithm
│   ├── threshold_diis
│   └── thresh_scf
├── integrals_bielec
│   ├── direct
│   ├── disk_access_ao_integrals
│   ├── disk_access_mo_integrals
│   ├── no_ivvv_integrals
│   ├── no_vvv_integrals
│   ├── no_vvvv_integrals
│   ├── threshold_ao
│   └── threshold_mo
├── integrals_monoelec
│   ├── disk_access_ao_one_integrals
│   └── disk_access_mo_one_integrals
├── mo_basis
│   ├── ao_md5
│   ├── mo_label
│   ├── mo_occ.gz
│   └── mo_tot_num
├── mrpt_utils
│   └── do_third_order_1h1p
├── nuclei
│   ├── disk_access_nuclear_repulsion
│   └── nucl_num
├── perturbation
│   ├── correlation_energy_ratio_max
│   ├── do_pt2
│   ├── pt2_absolute_error
│   ├── pt2_max
│   ├── pt2_relative_error
│   ├── threshold_generators_pt2
│   └── threshold_selectors_pt2
├── properties
│   ├── threshld_two_bod_dm
│   └── z_one_point
├── pseudo
│   ├── do_pseudo
│   ├── pseudo_grid_rmax
│   ├── pseudo_grid_size
│   ├── pseudo_klocmax
│   ├── pseudo_kmax
│   └── pseudo_lmax
└── qmc
    └── ci_threshold

The nesting corresponds roughly to what Fortran namelists looks like, and so in practice the fundamental input structure is not more complicated than e.g. Quantum Espresso's (but working with it is).

The Nexus spec corresponding to this allowed input structure, including types, is found in quantum_package_input.py assigned to the variable "input_specification".

The actual variables (the terminal files in the tree above) are actually unique names, so in Nexus one is allowed to generate the input by specifying nothing more than a single list of input variable names and values supplied to "generate_quantum_package". Nexus' internal represenation of the input follows the directory tree however, with directories being labeled as "Section"'s and each section containing the keyword/value pairs.

Input writing is done by first calling "qp_create_ezfio_from_xyz" and "qp_edit -c" if the ezfio directory does not exist, then if/once it exists all variables are written to the corresponding files.

In practice, the Nexus inputs contain the delta's for the input required at each step, so for HF, for example, the Nexus input file is (print qp.input):

  ao_basis
    ao_basis        = cc-pvtz
  end ao_basis
  determinants
    n_det_max       = 5000
  end determinants
  electrons
    elec_alpha_num  = 5
    elec_beta_num   = 5
  end electrons
  integrals_bielec
    disk_access_ao_integrals = Write
    disk_access_mo_integrals = Write
  end integrals_bielec
  integrals_monoelec
    disk_access_ao_one_integrals = Write
    disk_access_mo_one_integrals = Write
  end integrals_monoelec
  run_control
    postprocess     = []
    prefix          = h2o
    run_type        = SCF
    sleep           = 30
  end run_control
  structure
    axes            = []
    background_charge = 0
    bconds          = []
    center          = [ 0.  0.  0.]
    dim             = 3
    elem            = ['O' 'H' 'H']
    folded_structure = None
    frozen          = None
    kaxes           = []
    kpoints         = []
    kweights        = []
    mag             = []
    pos             = [[ 0.       0.       0.     ]
                       [ 0.       0.75716  0.58626]
                       [ 0.       0.75716 -0.58626]]
    scale           = 1.0
    units           = A
  end structure

The named sections "ao_basis", "determinants", "electrons", "integrals_bielec", and "integrals_monoelec" correspond directly to ezfio directories and variables. Additionally, "structure" is present to represent the internal state of the ezfio file regarding the atomic structure and "run_control" is added to represent the action-style inputs required to run the code.

@jtkrogel

This comment has been minimized.

Copy link
Contributor

jtkrogel commented Oct 12, 2018

Input generation and input representation and read/write are handled by generate_quantum_package_input() and QuantumPackageInput, repectively, in quantum_package_input.py. Any action required to update the input state (calling qp_create_ezfio/qp_edit/qp_set_frozen_core.py and writing variables to files) is handled by QuantumPackageInput.

Actions relating to major simulation functions, like SCF/CIS/FCI (running "qp_run ...") are handled by the QuantumPackage class in quantum_package.py.

@jtkrogel

This comment has been minimized.

Copy link
Contributor

jtkrogel commented Oct 12, 2018

The need to run commands in a split fashion (master/slave) when using multiple nodes

mpirun -np 1 qp_run SCF h2o.ezfio >h2o_scf.output 2>h2o_scf.error&
sleep 30
mpirun -np 3 qp_run -slave qp_ao_ints h2o.ezfio >h2o_scf_slave.output 2>h2o_scf_slave.error

or more traditionally when run on a single node

mpirun -np 1 qp_run SCF h2o.ezfio >h2o_scf.output 2>h2o_scf.error

required extensions to Nexus' job framework represented by the Job class in machines.py. This is handled primarily by the "split_nodes" function in the Job class. This function is used to create unified or split application run commands in the "app_command" function of the QuantumPackage class. Additional extensions are also present there to run CIS in a loop to iteratively generate natural orbitals to form a better starting point for selected CI.

No tests existed for Machine and Job classes prior to this point and the extensions required for Job splitting presented the possibility that some functionality would not work as expected, so I added tests for Machine and Job classes (including an extensive check performed by the "check_job_idempotency" function previously resident in machines.py) to the ntest script. Tests there now include an explicit check that simple job launch commands (including for various node and thread counts) are created correctly for all currently supported machines. Tests were then added for the new job splitting facility.

#end def write_qp_value



This comment has been minimized.

@markdewing

markdewing Oct 12, 2018

Contributor

This looks like it comes from extract_input_specification? If so, there should a comment about where this piece of code came from.

This comment has been minimized.

@markdewing

markdewing Oct 12, 2018

Contributor

I mean the code following this comment. The email message sent a bit of code from before this comment.

This comment has been minimized.

@jtkrogel

jtkrogel Oct 15, 2018

Contributor

Comment added.

@markdewing

This comment has been minimized.

Copy link
Contributor

markdewing commented Oct 12, 2018

The descriptions were helpful - they should make it into the documentation.

@jtkrogel

This comment has been minimized.

Copy link
Contributor

jtkrogel commented Oct 15, 2018

Ready to merge now?

@ye-luo

This comment has been minimized.

Copy link
Contributor

ye-luo commented Oct 15, 2018

Okay to test

@markdewing markdewing merged commit a37e1a5 into QMCPACK:develop Oct 15, 2018

2 checks passed

qmcpack rhea Build finished.
Details
qmcpack rhea (gpu) Build finished.
Details

@wafflebot wafflebot bot removed the in progress label Oct 15, 2018

@jtkrogel jtkrogel deleted the jtkrogel:nx_qm_package branch Jan 8, 2019

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment