Merge pull request #523 from deepmodeling/devel

Merge devel into master

.github/workflows/mirror_gitee.yml

@@ -15,5 +15,5 @@ jobs:
           ORGANIZATION: deepmodeling
           SSH_PRIVATE_KEY: ${{ secrets.SYNC_GITEE_PRIVATE_KEY }}
         with:
-          source-repo: "git@github.com:deepmodeling/dpgen.git"
+          source-repo: "https://github.com/deepmodeling/dpgen.git"
           destination-repo: "git@gitee.com:deepmodeling/dpgen.git"

.github/workflows/release.yml

@@ -0,0 +1,34 @@
+name: release_conda_and_github
+
+on:
+  push:
+    tags:
+      - 'v*'
+
+jobs:
+  publish-conda:
+    runs-on: ubuntu-latest
+    steps:
+    - uses: actions/checkout@v2
+    - name: publish-to-conda
+      uses: felix5572/conda-publish-action@v1.9
+      with:
+        subdir: 'conda'
+        anacondatoken: ${{ secrets.ANACONDA_TOKEN }}
+        platforms: 'noarch'
+  construct-and-publish:
+    runs-on: ubuntu-latest
+    needs: conda-publish
+    steps:
+    - uses: actions/checkout@master
+    - uses: s-weigand/setup-conda@v1
+    - run: conda install constructor jinja2 -y
+    - run: constructor ./conda
+    - name: Upload to release
+      uses: softprops/action-gh-release@master
+      if: startsWith(github.ref, 'refs/tags/')
+      with:
+        files: dpgen-*.sh
+        repository: ${{ env.GITHUB_REPOSITORY }}
+      env:
+        GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}

CITATION.cff

@@ -0,0 +1,25 @@
+preferred-citation:
+  type: article
+  authors:
+  - family-names: "Zhang"
+    given-names: "Yuzhi"
+  - family-names: "Wang"
+    given-names: "Haidi"
+  - family-names: "Chen"
+    given-names: "Weijie"
+  - family-names: "Zeng"
+    given-names: "Jinzhe"
+  - family-names: "Zhang"
+    given-names: "Linfeng"
+  - family-names: "Wang"
+    given-names: "Han"
+  - family-names: "E"
+    given-names: "Weinan"
+  doi: "10.1016/j.cpc.2020.107206"
+  journal: "Computer Physics Communications"
+  month: 8
+  start: 107206 # First page number
+  end: 107206 # Last page number
+  title: "DP-GEN: A concurrent learning platform for the generation of reliable deep learning based potential energy models"
+  volume: 253
+  year: 2020

README.md

@@ -25,7 +25,7 @@
 [![conda install](https://img.shields.io/conda/dn/conda-forge/dpgen?label=conda%20install)](https://anaconda.org/conda-forge/dpgen)
 [![pip install](https://img.shields.io/pypi/dm/dpgen?label=pip%20install)](https://pypi.org/project/dpgen)
 
-DP-GEN (Deep Generator)  is a software written in Python, delicately designed to generate a deep learning based model of interatomic potential energy and force field. DP-GEN is depedent on DeepMD-kit (https://github.com/deepmodeling/deepmd-kit/blob/master/README.md). With highly scalable interface with common softwares for molecular simulation, DP-GEN is capable to  automatically prepare scripts and maintain job queues on HPC machines (High Performance Cluster) and analyze results.
+DP-GEN (Deep Generator)  is a software written in Python, delicately designed to generate a deep learning based model of interatomic potential energy and force field. DP-GEN is dependent on [DeepMD-kit](https://github.com/deepmodeling/deepmd-kit/blob/master/README.md). With highly scalable interface with common softwares for molecular simulation, DP-GEN is capable to  automatically prepare scripts and maintain job queues on HPC machines (High Performance Cluster) and analyze results.
 
 If you use this software in any publication, please cite:
 
@@ -34,7 +34,7 @@ Yuzhi Zhang, Haidi Wang, Weijie Chen, Jinzhe Zeng, Linfeng Zhang, Han Wang, and
 ### Highlighted features
 + **Accurate and efficient**: DP-GEN is capable to sample more than tens of million structures and select only a few for first principles calculation. DP-GEN will finally obtain a uniformly accurate model.
 + **User-friendly and automatic**: Users may install and run DP-GEN easily. Once succusefully running, DP-GEN can dispatch and handle all jobs on HPCs, and thus there's no need for any personal effort.
-+ **Highly scalable**: With modularized code structures, users and developers can easily extend DP-GEN for their most relevant needs. DP-GEN currently supports for HPC systems (Slurm, PBS, LSF and cloud machines ), Deep Potential interface with DeePMD-kit, MD interface with LAMMPS  and *ab-initio* calculation interface with VASP, PWSCF,SIESTA and Gaussian. We're sincerely welcome and embraced to users' contributions, with more possibilities and cases to use DP-GEN.
++ **Highly scalable**: With modularized code structures, users and developers can easily extend DP-GEN for their most relevant needs. DP-GEN currently supports for HPC systems (Slurm, PBS, LSF and cloud machines ), Deep Potential interface with DeePMD-kit, MD interface with [LAMMPS](https://www.lammps.org/), [Gromacs](http://www.gromacs.org/)   and *ab-initio* calculation interface with VASP, PWSCF, CP2K, SIESTA and Gaussian, Abacus, PWMAT, etc . We're sincerely welcome and embraced to users' contributions, with more possibilities and cases to use DP-GEN.
 
 ### Code structure and interface
 + dpgen:
@@ -43,7 +43,8 @@ Yuzhi Zhang, Haidi Wang, Weijie Chen, Jinzhe Zeng, Linfeng Zhang, Han Wang, and
     * generator: source codes for main process of deep generator.
 
     * auto_test : source code for undertaking materials property analysis.
-    * remote : source code for automatically submiting scripts,maintaining job queues and collecting results.
+    * remote and dispatcher : source code for automatically submiting scripts,maintaining job queues and collecting results.
+    Notice this part hase been integrated into [dpdispatcher](https://github.com/deepmodeling/dpdispatcher) 
     * database : source code for collecting data generated by DP-GEN and interface with database.
 + examples : providing example JSON files.
 
@@ -63,6 +64,15 @@ Options for TASK:
 * `test`: Auto-test for Deep Potential.
 * `db`: Collecting data from DP-GEN.
 
+
+[Here](examples) are examples you can refer to. You should make sure that provide a correct [JSON](https://docs.python.org/3/library/json.html) file.
+You can use following command to check your JSON file.
+```python
+import json
+#Specify machine parameters in machine.json
+json.load(open("machine.json"))
+```
+
 ## Download and Install
 One can download the source code of dpgen by
 ```bash
@@ -539,8 +549,14 @@ The bold notation of key (such aas **type_map**) means that it's a necessary key
 | **model_devi_skip** | Integer | 0 | Number of structures skipped for fp in each MD
 | **model_devi_f_trust_lo** | Float | 0.05 | Lower bound of forces for the selection.
  | **model_devi_f_trust_hi** | Float | 0.15 | Upper bound of forces for the selection
-| **model_devi_e_trust_lo**  | Float | 1e10                                                         | Lower bound of energies for the selection. Recommend to set them a high number, since forces provide more precise information. Special cases such as energy minimization may need this. |
-| **model_devi_e_trust_hi**  | Float | 1e10                                                         | Upper bound of energies for the selection. |
+| **model_devi_v_trust_lo**  | Float | 1e10                                                         | Lower bound of virial for the selection. Should be used with DeePMD-kit v2.x |
+| **model_devi_v_trust_hi**  | Float | 1e10                                                         | Upper bound of virial for the selection. Should be used with DeePMD-kit v2.x |
+| model_devi_adapt_trust_lo  | Boolean | False | Adaptively determines the lower trust levels of force and virial. This option should be used together with `model_devi_numb_candi_f`,  `model_devi_numb_candi_v` and optionally with `model_devi_perc_candi_f` and `model_devi_perc_candi_v`. `dpgen` will make two sets: 1. From the frames with force model deviation lower than `model_devi_f_trust_hi`, select `max(model_devi_numb_candi_f, model_devi_perc_candi_f*n_frames)` frames with largest force model deviation. 2. From the frames with virial model deviation lower than `model_devi_v_trust_hi`, select `max(model_devi_numb_candi_v, model_devi_perc_candi_v*n_frames)` frames with largest virial model deviation. The union of the two sets is made as candidate dataset|
+| model_devi_numb_candi_f  | Int | 10 | See `model_devi_adapt_trust_lo`.|
+| model_devi_numb_candi_v  | Int | 0  | See `model_devi_adapt_trust_lo`.|
+| model_devi_perc_candi_f  | Float | 0.0 | See `model_devi_adapt_trust_lo`.|
+| model_devi_perc_candi_v  | Float | 0.0 | See `model_devi_adapt_trust_lo`.|
+| model_devi_f_avg_relative | Boolean | False | Normalized the force model deviations by the RMS force magnitude along the trajectory. This key should not be used with `use_relative`. |
 | **model_devi_clean_traj**  | Boolean | true                                                         | Deciding whether to clean traj folders in MD since they are too large. |
 | **model_devi_nopbc**  | Boolean | False                                                         | Assume open boundary condition in MD simulations. |
 | model_devi_activation_func | List of list of string | [["tanh","tanh"],["tanh","gelu"],["gelu","tanh"],["gelu","gelu"]]	| Set activation functions for models, length of the List should be the same as `numb_models`, and two elements in the list of string respectively assign activation functions to the embedding and fitting nets within each model. *Backward compatibility*: the orginal "List of String" format is still supported, where embedding and fitting nets of one model use the same activation function, and the length of the List should be the same as `numb_models`|
@@ -1322,7 +1338,9 @@ mem_limit | Interger | 16 | Maximal memory permitted to apply for the job.
 | # End of resources
 | command | String | "lmp_serial" | Executable path of software, such as `lmp_serial`, `lmp_mpi` and `vasp_gpu`, `vasp_std`, etc.
 | group_size | Integer | 5 | DP-GEN will put these jobs together in one submitting script.
-
+| user_forward_files | List of str | ["/path_to/vdw_kernel.bindat"] | These files will be uploaded in each calculation task. You should make sure provide the path exists. 
+| user_backward_files | List of str | ["HILLS"] | Besides DP-GEN's normal output, these files will be downloaded after each calculation. You should make sure these files can be generated.
+
 ## Troubleshooting
 1. The most common problem is whether two settings correspond with each other, including:
     - The order of elements in `type_map` and `mass_map` and **`fp_pp_files`**.

conda/conda_build_config.yaml

@@ -0,0 +1,5 @@
+channel_sources: 
+  - defaults
+  - conda-forge
+channel_targets: 
+  - deepmodeling

conda/construct.yaml

@@ -0,0 +1,18 @@
+{% set version = os.popen('git describe --tags --abbrev=0').read().strip('\n').lstrip('v').replace('-', '_') %}
+
+name: dpgen
+version: {{ version }}
+
+channels:
+  - defaults
+  - conda-forge
+  - deepmodeling
+
+specs:
+  - python 3.8
+  - pip
+  - dpgen {{ version }} 
+
+ignore_duplicate_files: True
+
+license_file: ../LICENSE

conda/meta.yaml

@@ -0,0 +1,52 @@
+{% set name = "dpgen" %}
+{% set version = os.popen('git describe --tags --abbrev=0').read().strip('\n').lstrip('v').replace('-', '_') %}
+
+package:
+  name: {{ name|lower }}
+  version: {{ version }}
+
+source:
+  path: ..
+
+build:
+  number: 0
+  noarch: python
+  script: python -m pip install --no-deps --ignore-installed .
+
+requirements:
+  build:
+    - git
+  host:
+    - python >=3.6
+    - pip
+    - setuptools_scm
+    - dargs
+    - paramiko
+    - requests
+    - dpdata
+    - dpdispatcher
+
+  run:
+    - python >=3.6
+    - dargs
+    - paramiko
+    - requests
+    - dpdata
+    - dpdispatcher
+    - pymatgen
+
+test:
+  imports:
+    - dpgen
+
+about:
+  home: https://github.com/deepmodeling/dpgen
+  license: LGPL-3.0
+  license_family: LGPL
+  license_file: LICENSE
+  doc_url: https://github.com/deepmodeling/dpgen
+  dev_url: https://github.com/deepmodeling/dpgen
+
+extra:
+  recipe-maintainers:
+    - felix5572

doc/CONTRIBUTING.md

@@ -0,0 +1,10 @@
+# DP-GEN Contributing Guide
+Welcome to [DP-GEN](https://github.com/deepmodeling/dpgen/tree/master/dpgen) !
+
+
+## How to contribute
+DP-GEN adopts the same convention as other softwares in DeepModeling Community. 
+You can first refer to DeePMD-kit's
+[Contributing guide](https://github.com/deepmodeling/deepmd-kit/edit/devel/CONTRIBUTING.md)
+and [Developer guide](https://github.com/deepmodeling/deepmd-kit/edit/devel/doc/development/index.md).
+

dpgen/auto_test/common_equi.py

@@ -9,10 +9,9 @@
 from dpgen.auto_test.calculator import make_calculator
 from dpgen.auto_test.mpdb import get_structure
 from dpgen.dispatcher.Dispatcher import make_dispatcher
-from dpgen.remote.decide_machine import decide_fp_machine, decide_model_devi_machine
 from distutils.version import LooseVersion
 from dpgen.dispatcher.Dispatcher import make_submission
-
+from dpgen.remote.decide_machine import convert_mdata
 lammps_task_type = ['deepmd', 'meam', 'eam_fs', 'eam_alloy']
 
 
@@ -133,9 +132,9 @@ def run_equi(confs,
     inter_type = inter_param['type']
     # vasp
     if inter_type == "vasp":
-        mdata = decide_fp_machine(mdata)
+        mdata = convert_mdata(mdata, ["fp"])
     elif inter_type in lammps_task_type:
-        mdata = decide_model_devi_machine(mdata)
+        mdata = convert_mdata(mdata, ["model_devi"])
     else:
         raise RuntimeError("unknown task %s, something wrong" % inter_type)
 

dpgen/auto_test/common_prop.py

@@ -13,9 +13,8 @@
 from dpgen.auto_test.Vacancy import Vacancy
 from dpgen.auto_test.calculator import make_calculator
 from dpgen.dispatcher.Dispatcher import make_dispatcher
-from dpgen.remote.decide_machine import decide_fp_machine, decide_model_devi_machine
 from dpgen.dispatcher.Dispatcher import make_submission
-
+from dpgen.remote.decide_machine import convert_mdata
 lammps_task_type = ['deepmd', 'meam', 'eam_fs', 'eam_alloy']
 
 
@@ -150,9 +149,9 @@ def run_property(confs,
             inter_type = inter_param_prop['type']
             # vasp
             if inter_type == "vasp":
-                mdata = decide_fp_machine(mdata)
+                mdata = convert_mdata(mdata, ["fp"])
             elif inter_type in lammps_task_type:
-                mdata = decide_model_devi_machine(mdata)
+                mdata = convert_mdata(mdata, ["model_devi"])
             else:
                 raise RuntimeError("unknown task %s, something wrong" % inter_type)
 

dpgen/auto_test/lib/util.py

@@ -77,11 +77,11 @@ def get_machine_info(mdata,task_type):
         command = vasp_exec
         command = cmd_append_log(command, "log")
     elif task_type in lammps_task_type:
-        lmp_exec = mdata['lmp_command']
+        model_devi_exec = mdata['model_devi_command']
         group_size = mdata['model_devi_group_size']
         resources = mdata['model_devi_resources']
         machine=mdata['model_devi_machine']
-        command = lmp_exec + " -i in.lammps"
+        command = model_devi_exec + " -i in.lammps"
         command = cmd_append_log(command, "model_devi.log")
     return machine, resources, command, group_size