Running ePTM using bundled scripts
As specific users will want to build their own pipelines for running the model, we have provided a bare-bones set of Python and R scripts that will allow you to set up and run a single instance of ePTM v2. Please follow the documentation on how to compile the source code provided here link or, use the precompiled jar file here link with the scripts provided. For a description of model inputs, outputs and configuration files, please refer to the document here Link.
ePTM v2 is path agnostic. The scripts and configuration files we have provided need to be suitably modified to reflect your own directory structure, but once set up, will run wherever you have placed your configuration and model files.
Just please make sure there are no white spaces in any paths.
Using the HDF viewer, one can see the structure of the DSM2 hydrodynamic model results, as well as the ePTM behavior assignments and the ePTM model outputs.
Open the DSM2-generated H5 file and navigate to the Scalars path to get the start and end dates of the hydrodynamic model simulation. These will be required in the DSM2 configuration file.
The DSM2 modeling suite, including DSM2-HYDRO and DSM2-PTM, on which ePTM v2 is based, compress the H5 outputs using a proprietary szip compression algorithm. This makes it impossible to read these files in scripting languages such as R, Python and Matlab. If you downloaded the HDF5 suite of prebuilt binaries, you can use the script h5rpkBatch.bat to unpack these files and import them into your scripting language of choice.
First make copies of the configFiles and scripts folders. These contain template files which you will routinely modify, so having an original to reference back to is a good idea.
The R script ePTMCreateInputs.R is a heavily commented script that provided guides to where the user must modify the code. Please follow in the instructions in that script to make any necessary changes.
You will have to provide a CSV file containing a release regime or a timeseries of release dates at each release location you wish to simulate. In essence, you are telling ePTM v2 that you would like to release a number of virtual fish at certain locations in a specific order.
If you are using the model for the California Central Valley, we strongly recommend using the searchable DSM2 grid map bundled in the modelFiles folder to get DSM2 external and internal nodes and channel numbers. In populating the ePTMCreateInputs.R script, you will use DSM2 external node numbers to populate the list of checkpoints where you would like to record particle histories, and external nodes where particle releases will occur.
Next, you will need to modify ALL CAPS entries in the DSM2ConfigTemplate.inp. The single largest time sink in setting up an ePTM v2 model run is getting the directory structure represented correctly, so please make sure every path is referenced correctly.
It does not matter to ePTM v2 if you choose to work with absolute paths or relative paths. But you should be careful if you are using relative paths.
Using the start and end times you grabbed from the DSM2-generated hydrodynamics HDF5 output file, modify these entries appropriately.
Make sure to move the QUAL_START_DATE by one day forward.
If you have made sure your directory structure is correct, and if you have carefully constructed your release regime and checkpoints list, you should get a behaviors HDF5 and a ePTM configuration .inp file created. The behavior files assigns values to parameters in the behavior models which change the functional response of simulated fish spatially within the domain, while the configuration file tells ePTM v2 where it needs to release fish, in what order, and how to save the output.
Once you have created the DSM2 and ePTM configuration files, and the ePTM behaviors HDF5 file, you can use the two Python scripts to run the model. Here too, comments guide where you will have to make modifications. You will need to modify ALL CAPS entries in the DSM2ConfigTemplate.inp.
The single largest time sink in setting up an ePTM v2 model run is getting the directory structure represented correctly, so please make sure every path is referenced correctly.
If you have a single version of Python installed, please consider adding it to the system path, so that you can simply invoke python at the command prompt. If you have multiple environments and are using an environment manager such as Conda, you should activate a Python 3.3 or above environment to run the ePTM v2 batch and run scripts.
Open a command prompt and run the following command:
python ePTMBatch.py
This is where you will get creative with your pipeline depending on your needs. For example, you may wish to run just a single release of particles at a single node: this script will let you do that. However, if you want to run large processes, such as recalibrating the model, or running multi-decadal water management scenario evaluations, you might want to parallelize and have multiple instances of the model running. This is where you will make such decisions.
Since there are no inter-particle dependencies in the model, you can make an already embarrassingly parallelizable problem even faster by decoupling release locations (DSM2 nodes) and run multiple instances of the model where each instance releases particles in different places. You can also simulate different types of fish by having different behavior HDF5 files and running multiple instances of the model, each with its own behavior parameters to compare how different species use the water at the same time.
ePTMBatch.bat will invoke ePTMRun.py, which actually runs the model. If you have ensured that all directory structures have been mapped correctly, and that there are no bugs in your DSM2 setup, the model will start running and update progress daily. An output HDF5 file will be created where you specified. This file will contain:
For each particle a history of where and when it was released, its arrival times at the checkpoints specified, and where and when it died.
It will also contain the same information but from the point of view of the checkpoints. Imagine these as stationary observers who count fish that pass them. These contain records of each fish that passed the checkpoint with its passage time and unique id.
There are only three reasons the code will break at this point:
- There is something wrong with the version of Java you are using. Make sure you have version 8 installed.
- There is something wrong with the directory structure, and its specification in the configuration file and scripts. Make sure everything is correct. Wear your Sherlock Holmes cap here and be thorough and meticulous.
- There is something inconsistent between your hydrodynamic simulation, model grid and your particle release regime. Ensure that particles are released only within the domain and only during the times the model simulation was requested to run.
Congragulations! You have successfully run ePTM v2.
While beyond the scope of this document, you can use the h5rpkBatch.bat to unpack the HDF5 output files generated by DSM2 and ePTM v2. Then you can import these files into your favorite post-processing language and work away at your custom analysis and visualizations.
Enjoy.