Merged in dev (pull request #76)

Dev Approved-by: Matthew Jones <mattyl.jones@dunelm.org.uk>
matty-jones · Dec 10, 2018 · 60c616e · 60c616e
2 parents 0f3ec10 + 463230e
commit 60c616e
Show file tree

Hide file tree

Showing 761 changed files with 751,717 additions and 2,047 deletions.
diff --git a/.coveragerc b/.coveragerc
@@ -1,6 +1,17 @@
 [run]
 branch = True
+source = MorphCT
 
 [report]
 exclude_lines =
+    if self.debug:
     pragma: no cover
+    raise AssertionError
+    raise NotImplementedError
+    if __name__ == .__main__.:
+ignore_errors = True
+omit =
+    tests/*
+
+[html]
+directory = codecoverage
diff --git a/.coveralls.yml b/.coveralls.yml
diff --git a/README.md b/README.md
@@ -2,7 +2,7 @@
 
 [![DOI](https://zenodo.org/badge/100152281.svg)](https://zenodo.org/badge/latestdoi/100152281)
 [![Run Status](https://api.shippable.com/projects/5ae1ee243e7d2b0700153be4/badge?branch=dev)](https://app.shippable.com/bitbucket/cmelab/morphct) 
-[![Coverage Status](https://coveralls.io/repos/bitbucket/cmelab/morphct/badge.svg?branch=dev)](https://coveralls.io/bitbucket/cmelab/morphct?branch=dev)
+[![Coverage Status](https://codecov.io/bb/cmelab/morphct/branch/dev/graph/badge.svg)](https://codecov.io/bb/cmelab/morphct)
 
 # Intellectual Property #
 
@@ -18,15 +18,16 @@ MorphCT accomplishes this by:
 
 * Converting any coarse-grained organic morphology to the atomistic representation for submission to quantum chemical calculations through a process called fine-graining.
 * Splitting the morphology into electronically-active chromophores that charge carriers (holes in the case of donor materials, electrons for acceptors) are likely to be delocalised along, and can perform quantized charge hops between.
-* Performing high-throughput, fast quantum chemical calculations to obtain the energetic landscape caused by conformational disorder, as well as electronic transfer integrals between chromophore pairs.
-* Using these calculated electronic properties as inputs into a kinetic Monte Carlo algorithm to simulate the motion of charge carriers throughout the device, allowing carrier mobilities to be obtained (a good proxy for device performance)
+* Performing high-throughput, fast quantum chemical calculations (QCCs) to obtain the energetic landscape caused by conformational disorder, as well as electronic transfer integrals between chromophore pairs.
+* Using these calculated electronic properties as inputs into a kinetic Monte Carlo (KMC) algorithm to simulate the motion of charge carriers throughout the device, allowing carrier mobilities to be obtained (a good proxy for device performance)
 * Using the charge transport properties of previous morphological moieties to simulate the various mechanisms as play in an organic photovoltaic device (applied voltage bias, photoinjection, dark-current injection, exciton transport, exciton dissociation, carrier transport, extraction, geminate recombination, bimolecular recombination) in order to generate J-V curves and calculate OPV device efficiencies to guide manufacturing.
 
 ---
 
 # Tagged Releases #
 
-* v3.0: MorphCT codebase brought up to PEP8 standard (with line limit of 120 characters), refactored to work as a package, added extensive unit tests to the pipeline (pytest) and added continuous integration support using Shipable and Coveralls.
+* v3.1: Many quality-of-life improvements, issue resolutions, and bug fixes including but not limited to: the ability to write ORCA jobs to a RAM disk for 30x faster calculations, significantly more unit tests to increase code coverage to 75%, full testing the device simulations, better python formatting, and migration from Coveralls to Codecov.io.
+* v3.0: MorphCT codebase brought up to PEP8 standard (with line limit of 120 characters), refactored to work as a package, added extensive unit tests to the pipeline (pytest) and added continuous integration support using Shippable and Coveralls.
 * v2.2: Additional funcionality added for blend morphologies, multiple donor/acceptor species, variable-range hopping, Voronoi neighbourhood analysis and other performance tweaks. Results in preparation to be submitted in Q2 2018.
 * v2.1: MorphCT updated from python 2.7 to python 3.5
 * v2.0: Hardcode removed, utility of MorphCT expanded to any organic molecule with more customizable features (support for small molecules included)
@@ -69,7 +70,7 @@ MorphCT accomplishes this by:
 
 ## Running Tests ##
 
-* More than 100 unit tests are provided to ensure that MorphCT has been set up correctly.
+* 300 unit tests are provided to ensure that MorphCT has been set up correctly.
 * These can be executed by running `pytest` in the MorphCT root directory, or in `morphct/tests`.
 
 ---
@@ -84,11 +85,13 @@ MorphCT accomplishes this by:
     * Atomistic Template: If fine-graining is desired, this HOOMD xml file describes the atomstic template to be mapped onto the input coarse-grained system. Examples are given in the `templates` directory for a variety of organic molecule types: `P3HT.xml`, `C60.xml`, `BDTTPD.xml`, `perylene.xml`, `perylothiophene.xml` and `PCBM.xml`.
     * Atomistic Forcefields: If fine-graining is desired, this xml file contains the forcefield parameters describing the atomistic interactions used to relax the morphology to a realistic conformation after fine-graining. Examples are given in the `templates` directory for several of the example molecules `FFP3HT.xml`, `FFC60.xml`, `FFPerylene.xml`, `FFPerylothiophene.xml`.
 * An example is provided for the user in `morphct/templates/par_template.py`. This file is heavily documented to describe the functions and features of MorphCT, and should be the first port-of-call for seeing what is possible with the program.
+* For device simulations, a set of molecular systems already run through the pipeline (up to the mobility KMC phase - i.e. morphology fine-grained and relaxed, chromophores detected, and energy levels and transfer integrals calculated) for every component moiety of the input device must be present and have its location identified in the parameter file. An additional device input file identifying the arrangement of the morphology moieties in the device must also be present and have its location identified in the parameter file.
 
 ## Job Execution ##
 
 * Jobs can be invoked by using the HOOMD v1.3 python wrapper around the relevant parameter file for the job: `hoomd par_template.py`.
-* MorphCT has also been heavily tested on several High Performance Computing Clusters, and found to scale well. Child jobs are spawned for parallelisable modules in the pipeline, so simply submitting the `hoomd par_template.py` command to the HPC resource management system should be sufficient to parallelise the jobs where possible.
+* MorphCT has also been heavily tested on several High Performance Computing Clusters (Kestrel, Fry, R1, R2, Comet, Bridges, XStream), and found to scale well. Child jobs are spawned for parallelisable modules (quantum chemical calculations, mobility KMC and device KMC) in the pipeline, so simply submitting the `hoomd par_template.py` command to the HPC resource management system should be sufficient to parallelise the jobs where possible.
+* For large systems (> 1000 chromophores), it is recommended to perform the QCC calculations in memory, or at least on a fast file system with high file i/o bandwidth (Orca reads and writes a large number of temporary files to the disk for every calculation, which can quickly clog up a slower file system), by setting the `orca_output_directory` parameter.
 
 ---
 
@@ -99,6 +102,7 @@ MorphCT accomplishes this by:
 * Alternatively, raising issues and voting for the issues most important to you is highly recommended and appreciated.
 * All contributions should be PEP8 compliant (comment lines limited to 80 characters, code lines limited to 120 characters).
 * All pull requests require approval from at least one reviewer, a successful build on the latest commit to the fork, and no failed builds on the latest commit to the fork to be accepted.
+* Please note that if the pull request of your fork includes changes to the shippable.yml, the CI will still use the yml on the destination branch (usually dev). To resolve this, please activate CI for your fork first to ensure build stability, and link the CI results in your pull request.
 
 ---
 

diff --git a/environment.yml b/environment.yml
@@ -13,5 +13,5 @@ dependencies:
 - pytest
 - pytest-cov
 - PyYAML
-- coveralls
+- codecov
 - imagemagick
diff --git a/morphct/code/device_KMC.py b/morphct/code/device_KMC.py
@@ -9,13 +9,11 @@
 
 class morphology_moiety:
     def __init__(self, mol_morph_name, parameter_dict):
-        chromophore_list_location = (
-            parameter_dict["output_morph_dir"]
-            + "/"
-            + mol_morph_name
-            + "/code/"
-            + mol_morph_name
-            + ".pickle"
+        chromophore_list_location = os.path.join(
+            parameter_dict["output_morph_dir"],
+            mol_morph_name,
+            "code",
+            "".join([mol_morph_name, ".pickle"]),
         )
         pickle_data = hf.load_pickle(chromophore_list_location)
         self.AA_morphology_dict = pickle_data[0]
@@ -127,15 +125,15 @@ def return_device_moiety_type(self, device_position):
 
 
 def load_device_morphology(parameter_dict):
-    device_dir = (
-        parameter_dict["input_device_dir"] + "/" + parameter_dict["device_morphology"]
+    device_dir = os.path.join(
+        parameter_dict["input_device_dir"], parameter_dict["device_morphology"]
     )
     y_slices = os.listdir(device_dir)
     # Initialize the array of the correct size (assumes cubic morphology)
     device_array = np.zeros([len(y_slices)] * 3, dtype=int)
     for y_val, file_name in enumerate(y_slices):
         # Load the ySlice as-presented in the input files
-        y_slice = np.loadtxt(device_dir + "/" + file_name, dtype=int)
+        y_slice = np.loadtxt(os.path.join(device_dir, file_name), dtype=int)
         if len(y_slice.shape) > 0:
             # The z-origin is at the top, and we need it at the bottom, so turn
             # the array upside down
@@ -158,7 +156,8 @@ def load_device_morphology(parameter_dict):
 
 def main(parameter_dict):
     # Get the random seed now for all the child processes
-    np.random.seed(hf.obtain_random_seed())
+    if parameter_dict["random_seed_override"] is not None:
+        np.random.seed(parameter_dict["random_seed_override"])
     # First job will be to load in the device morphology, when I work out what
     # format I want it to be.
     device_array, moiety_dictionary = load_device_morphology(parameter_dict)
@@ -171,14 +170,13 @@ def main(parameter_dict):
     # Write these classes out to a pickle file so that they can be loaded by the
     # child processes later
     to_pickle = [device_array, chromophore_data, morphology_data, parameter_dict]
-    save_directory = (
-        parameter_dict["output_device_dir"]
-        + "/"
-        + parameter_dict["device_morphology"]
-        + "/code"
+    save_directory = os.path.join(
+        parameter_dict["output_device_dir"], parameter_dict["device_morphology"], "code"
     )
     if parameter_dict["overwrite_current_data"] is True:
-        with open(save_directory + "/device_data.pickle", "wb+") as pickle_file:
+        with open(
+            os.path.join(save_directory, "device_data.pickle"), "wb+"
+        ) as pickle_file:
             pickle.dump(to_pickle, pickle_file)
     voltages = []
     for V in parameter_dict["voltage_sweep"]:
@@ -196,13 +194,13 @@ def main(parameter_dict):
     )
     print("Writing job pickles for each CPU...")
     for proc_ID, jobs in enumerate(jobs_list):
-        pickle_name = os.path.join(output_dir, "KMC_data_%02d.pickle" % (proc_ID))
+        pickle_name = os.path.join(output_dir, "KMC_data_{:02d}.pickle".format(proc_ID))
         with open(pickle_name, "wb+") as pickle_file:
             pickle.dump(jobs, pickle_file)
         print(
             "KMC jobs for proc_ID",
             proc_ID,
-            "written to KMC_data_%02d.pickle" % (proc_ID),
+            "written to KMC_data_{:02d}.pickle".format(proc_ID),
         )
         # Open the required processes to execute the KMC jobs
         # Random seeding is a little weird here. If we don't generate a random
@@ -215,7 +213,7 @@ def main(parameter_dict):
         child_seed = np.random.randint(0, 2 ** 32)
         # Previous run command:
         run_command = [
-            "python ",
+            "python",
             SINGLE_RUN_DEVICE_KMC_FILE,
             output_dir,
             str(proc_ID),

diff --git a/morphct/code/execute_ZINDO.py b/morphct/code/execute_ZINDO.py
@@ -29,10 +29,7 @@ def create_input_files(chromophore_list, AA_morphology_dict, parameter_dict):
             [chromophore.image] * len(chromophore.AAIDs),
             terminating_group_positions,
             terminating_group_images,
-            parameter_dict["output_morph_dir"]
-            + "/"
-            + parameter_dict["morphology"][:-4]
-            + chromophore.orca_input,
+            "".join([parameter_dict["output_orca_directory"], chromophore.orca_input]),
         )
     print("")
     # Determine how many pairs there are first:
@@ -58,7 +55,7 @@ def create_input_files(chromophore_list, AA_morphology_dict, parameter_dict):
                 continue
             # Update the orca input name
             input_name = chromophore1.orca_input.replace(
-                ".inp", "-%05d.inp" % (chromophore2.ID)
+                ".inp", "-{:05d}.inp".format(chromophore2.ID)
             ).replace("single", "pair")
             # Find the correct relative image for the neighbour chromophore
             chromophore2_relative_image = neighbours_image[
@@ -103,10 +100,7 @@ def create_input_files(chromophore_list, AA_morphology_dict, parameter_dict):
                     images,
                     term_group_posns1 + term_group_posns2,
                     terminating_group_images1 + terminating_group_images2,
-                    parameter_dict["output_morph_dir"]
-                    + "/"
-                    + parameter_dict["morphology"][:-4]
-                    + input_name,
+                    "".join([parameter_dict["output_orca_directory"], input_name]),
                 )
             else:
                 # Write the dimer input file
@@ -116,10 +110,7 @@ def create_input_files(chromophore_list, AA_morphology_dict, parameter_dict):
                     images,
                     None,
                     None,
-                    parameter_dict["output_morph_dir"]
-                    + "/"
-                    + parameter_dict["morphology"][:-4]
-                    + input_name,
+                    "".join([parameter_dict["output_orca_directory"], input_name]),
                 )
     print("")
 
@@ -219,8 +210,7 @@ def write_orca_inp(
     # Create the lines to be written in the input file
     for index, position in enumerate(all_positions):
         lines_to_write.append(
-            " %s  %.5f  %.5f  %.5f\n"
-            % (
+            " {0:s}  {1:.5f}  {2:.5f}  {3:.5f}\n".format(
                 all_atom_types[index],
                 position[0] - central_position[0],
                 position[1] - central_position[1],
@@ -229,7 +219,7 @@ def write_orca_inp(
         )
     # Load the orca input template
     orca_temp_dir = os.path.join(PROJECT_ROOT, "templates")
-    orca_temp_test_dir = os.path.join(PROJECT_ROOT, "code/unit_testing/assets")
+    orca_temp_test_dir = os.path.join(PROJECT_ROOT, "code", "unit_testing", "assets")
     try:
         with open(os.path.join(orca_temp_dir, "template.inp"), "r") as template_file:
             inp_file_lines = template_file.readlines()
@@ -244,11 +234,7 @@ def write_orca_inp(
     # Write the orca input file
     with open(input_name, "w+") as orca_file:
         orca_file.writelines(inp_file_lines)
-    print(
-        "\rOrca Input File written as",
-        input_name[hf.find_index(input_name, "/")[-1] + 1 :],
-        end=" ",
-    )
+    print("\rOrca Input File written as", os.path.split(input_name[1]), end=" ")
 
 
 def terminate_monomers(chromophore, parameter_dict, AA_morphology_dict):
@@ -295,15 +281,15 @@ def get_orca_jobs(input_dir, parameter_dict, proc_IDs):
     except OSError:
         pass
     # Obtain a list of files to run
-    single_orca_file_list = os.listdir(input_dir + "/single")
-    pair_orca_file_list = os.listdir(input_dir + "/pair")
+    single_orca_file_list = os.listdir(os.path.join(input_dir, "single"))
+    pair_orca_file_list = os.listdir(os.path.join(input_dir, "pair"))
     orca_files_to_run = []
     for file_name in single_orca_file_list:
         if file_name[-4:] == ".inp":
-            orca_files_to_run.append(input_dir + "/single/" + file_name)
+            orca_files_to_run.append(os.path.join(input_dir, "single", file_name))
     for file_name in pair_orca_file_list:
         if file_name[-4:] == ".inp":
-            orca_files_to_run.append(input_dir + "/pair/" + file_name)
+            orca_files_to_run.append(os.path.join(input_dir, "pair", file_name))
     orca_files_to_run.sort()
     if parameter_dict["overwrite_current_data"] is False:
         # Do not run any jobs that have already have an output file (and so have
@@ -345,15 +331,12 @@ def main(
     parameter_dict,
     chromophore_list,
 ):
-    print(parameter_dict["output_morph_dir"])
     # Get the random seed now for all the child processes
-    np.random.seed(hf.obtain_random_seed())
+    if parameter_dict["random_seed_override"] is not None:
+        np.random.seed(parameter_dict["random_seed_override"])
     create_input_files(chromophore_list, AA_morphology_dict, parameter_dict)
-    input_dir = (
-        parameter_dict["output_morph_dir"]
-        + "/"
-        + parameter_dict["morphology"][:-4]
-        + "/chromophores/input_orca"
+    input_dir = os.path.join(
+        parameter_dict["output_orca_directory"], "chromophores", "input_orca"
     )
     proc_IDs = parameter_dict["proc_IDs"]
     jobs_list = get_orca_jobs(input_dir, parameter_dict, proc_IDs)
@@ -378,11 +361,11 @@ def main(
                     [
                         "python",
                         SINGLE_ORCA_RUN_FILE,
-                        parameter_dict["output_morph_dir"]
-                        + "/"
-                        + parameter_dict["morphology"][:-4],
+                        parameter_dict["output_orca_directory"],
+                        parameter_dict["output_morphology_directory"],
                         str(CPU_rank),
                         str(int(parameter_dict["overwrite_current_data"])),
+                        str(int(parameter_dict["remove_orca_inputs"])),
                     ]
                 )
             )