Copy from SEIMS-UserManual

doc/en/get_started/data_preparation.md

@@ -27,15 +27,15 @@ SUBSCENARIO,NAME,LANDUSE,PARAMETERS
 ``` 
 can be parsed as a Python data structure of dictionary like:
 ```
-01 demo_dict = {
-02     'SUBSCENARIO': 1,
-03     'NAME':'Closing measures'
-04     'LANDUSE': [7, 16],
-05     'PARAMETERS': [
-06         ['Interc_max', 'Maximum Interception Capacity', 'AC', 1],
-07         ['Conductivity', 'Soil hydraulic conductivity', 'RC', 3.5]        
-08     ]
-09 }
+demo_dict = {
+    'SUBSCENARIO': 1,
+    'NAME':'Closing measures'
+    'LANDUSE': [7, 16],
+    'PARAMETERS': [
+        ['Interc_max', 'Maximum Interception Capacity', 'AC', 1],
+        ['Conductivity', 'Soil hydraulic conductivity', 'RC', 3.5]        
+    ]
+}
 ```
 # Spatial data
 The demo data named Youwuzhen watershed (~5.39 km<sup>2</sup>) is located in Changting County of Fujian province, China (Figure 1). It belongs to the typical red-soil hilly region in southeastern China and suffers from severe soil erosion. The study area has hills with steep slopes (up to 52.9° and with an average slope of 16.8°) and broad alluvial valleys. The elevation ranges from 295.0 m to 556.5 m. The landuse types are mainly forest (59.8%), paddy field (20.6%), and orchard (12.8%) (Figure 2). Soil types in the study area are red soil (78.4%) and paddy soil (21.6%) which can be classified as *Ultisols* and *Inceptisols* in US Soil Taxonomy, respectively (Figure 3). 
@@ -396,27 +396,27 @@ SUBSCENARIO,NAME,DESC,REFERENCE,LANDUSE,SLPPOS,PARAMETERS,EFFECTIVENESS,CAPEX,OP
 1,fengjin, CM (closing measures),Qin et al (2018),8-6,1-4,OM:Organic matter:RC:1.22-Density:bulk density:RC:0.98-Porosity:Total porosity:RC:1.02-USLE_K:USLE soil erodibility:RC:1.01-Conductivity:Soil hydraulic conductivity:RC:0.81-FieldCap:Soil field capacity:RC:1.02-Wiltingpoint:Wiltingpoint:RC:1.02-SOL_AWC:Soil available water:RC:1.02-SOL_UL:Soil saturated water:RC:1.02-SOL_CBN:Soil carbon content:RC:1.22-USLE_P:USLE practice factor:RC:0.9,3,15.5,1.5,2.0
 ```
 which can be parsed as a Python data structure of dictionary like:
-```json
-01 demo_dict = {'NAME': 'fengjin',
-02               'DESC': 'CM (closing measures)',
-03               'REFERENCE': 'Qin et al (2018)',
-04               'LANDUSE': [8, 6],
-05               'SLPPOS': [1, 4],
-06               'PARAMETERS': [
-07                               {'NAME': 'OM',
-08                                'DESC': 'ORGANIC MATTER',
-09                                'CHANGE': 'RC',
-10                                'IMPACT': 1.22
-11                                },
-12                               {'NAME': 'DENSITY',
-13                                'DESC': 'BULK DENSITY',
-14                                'CHANGE': 'RC',
-15                                'IMPACT': 0.98
-16                                }
-17                               ],
-18               'EFFECTIVENESS': 3,
-19               'CAPEX': 15.5,
-20               'OPEX': 1.5,
-21               'INCOME': 2,
-22               }
+```
+demo_dict = {'NAME': 'fengjin',
+            'DESC': 'CM (closing measures)',
+            'REFERENCE': 'Qin et al (2018)',
+            'LANDUSE': [8, 6],
+            'SLPPOS': [1, 4],
+            'PARAMETERS': [
+                            {'NAME': 'OM',
+                            'DESC': 'ORGANIC MATTER',
+                            'CHANGE': 'RC',
+                            'IMPACT': 1.22
+                            },
+                            {'NAME': 'DENSITY',
+                            'DESC': 'BULK DENSITY',
+                            'CHANGE': 'RC',
+                            'IMPACT': 0.98
+                            }
+                            ],
+            'EFFECTIVENESS': 3,
+            'CAPEX': 15.5,
+            'OPEX': 1.5,
+            'INCOME': 2,
+            }
 ```

doc/en/get_started/data_preprocessing.md

@@ -37,50 +37,50 @@ The simple usage of the data preprocessing includes two steps such as generating
 
 SEIMS takes the `INI` file as the format of configuration files. `INI` files are plain text files with a basic structure composed of sections and options (an option is a pair of property and value). Semicolons (`;`) or number sign (`#`) at the beginning of the line indicate a comment which will be ignored. The data preprocessing configuration content of Youwuzhen watershed is as follows:
 ```ini
-01 [PATH]
-02 PREPROC_SCRIPT_DIR = d:\demo\SEIMS\seims\preprocess
-03 CPP_PROGRAM_DIR = d:\demo\SEIMS\bin
-04 MPIEXEC_DIR = None
-05 BASE_DATA_DIR = d:\demo\SEIMS\data\youwuzhen
-06 CLIMATE_DATA_DIR = d:\demo\SEIMS\data\youwuzhen\data_prepare\climate
-07 SPATIAL_DATA_DIR = d:\demo\SEIMS\data\youwuzhen\data_prepare\spatial
-08 MEASUREMENT_DATA_DIR = d:\demo\SEIMS\data\youwuzhen\data_prepare\observed
-09 BMP_DATA_DIR = d:\demo\SEIMS\data\youwuzhen\data_prepare\scenario
-10 MODEL_DIR = d:\demo\SEIMS\data\youwuzhen\demo_youwuzhen30m_longterm_model
-11 TXT_DB_DIR = d:\demo\SEIMS\data\youwuzhen\data_prepare\lookup
-12 WORKING_DIR = d:\demo\SEIMS\data\youwuzhen\workspace
-13 [MONGODB]
-14 HOSTNAME = 127.0.0.1
-15 PORT = 27017
-16 ClimateDBName = demo_youwuzhen30m_HydroClimate
-17 BMPScenarioDBName = demo_youwuzhen30m_Scenario
-18 SpatialDBName = demo_youwuzhen30m_longterm_model
-19 [CLIMATE]
-20 HydroClimateVarFile = Variables.csv
-21 MeteoSiteFile = Sites_M.csv
-22 PrecSiteFile = Sites_P.csv
-23 MeteoDataFile = meteo_daily.csv
-24 PrecDataFile = pcp_daily.csv
-25 thiessenIdField = ID
-26 [SPATIAL]
-27 dem = ywzdem30m.tif
-28 outlet_file = outlet_beijing1954.shp
-29 PrecSitesThiessen = thiessen_pcp.shp
-30 MeteoSitesThiessen = thiessen_meteo.shp
-31 landuseFile = ywzlanduse30m.tif
-32 landcoverInitFile = landcover_initial_parameters.csv
-33 soilSEQNFile = ywzsoil30m.tif
-34 soilSEQNText = soil_properties_lookup.csv
-35 field_partition_thresh = 15
-36 [OPTIONAL_PARAMETERS]
-37 D8AccThreshold = 35
-38 np = 4
-39 D8DownMethod = Surface
-40 dorm_hr = -1.
-41 T_base = 0.
-42 imperviousPercInUrbanCell = 0.3
-43 defaultLanduse = 33
-44 defaultSoil = 201
+[PATH]
+PREPROC_SCRIPT_DIR = d:\demo\SEIMS\seims\preprocess
+CPP_PROGRAM_DIR = d:\demo\SEIMS\bin
+MPIEXEC_DIR = None
+BASE_DATA_DIR = d:\demo\SEIMS\data\youwuzhen
+CLIMATE_DATA_DIR = d:\demo\SEIMS\data\youwuzhen\data_prepare\climate
+SPATIAL_DATA_DIR = d:\demo\SEIMS\data\youwuzhen\data_prepare\spatial
+MEASUREMENT_DATA_DIR = d:\demo\SEIMS\data\youwuzhen\data_prepare\observed
+BMP_DATA_DIR = d:\demo\SEIMS\data\youwuzhen\data_prepare\scenario
+MODEL_DIR = d:\demo\SEIMS\data\youwuzhen\demo_youwuzhen30m_longterm_model
+TXT_DB_DIR = d:\demo\SEIMS\data\youwuzhen\data_prepare\lookup
+WORKING_DIR = d:\demo\SEIMS\data\youwuzhen\workspace
+[MONGODB]
+HOSTNAME = 127.0.0.1
+PORT = 27017
+ClimateDBName = demo_youwuzhen30m_HydroClimate
+BMPScenarioDBName = demo_youwuzhen30m_Scenario
+SpatialDBName = demo_youwuzhen30m_longterm_model
+[CLIMATE]
+HydroClimateVarFile = Variables.csv
+MeteoSiteFile = Sites_M.csv
+PrecSiteFile = Sites_P.csv
+MeteoDataFile = meteo_daily.csv
+PrecDataFile = pcp_daily.csv
+thiessenIdField = ID
+[SPATIAL]
+dem = ywzdem30m.tif
+outlet_file = outlet_beijing1954.shp
+PrecSitesThiessen = thiessen_pcp.shp
+MeteoSitesThiessen = thiessen_meteo.shp
+landuseFile = ywzlanduse30m.tif
+landcoverInitFile = landcover_initial_parameters.csv
+soilSEQNFile = ywzsoil30m.tif
+soilSEQNText = soil_properties_lookup.csv
+field_partition_thresh = 15
+[OPTIONAL_PARAMETERS]
+D8AccThreshold = 35
+np = 4
+D8DownMethod = Surface
+dorm_hr = -1.
+T_base = 0.
+imperviousPercInUrbanCell = 0.3
+defaultLanduse = 33
+defaultSoil = 201
 ```
 
 The configuration file for data preprocessing, such as that of the Youwuzhen watershed, includes five sections, i.e., `PATH`, `MONGODB`, `CLIMATE`, `SPATIAL`, and `OPTIONAL_PARAMETERS`. 

doc/en/get_started/parameters_sensitivity.md

@@ -3,6 +3,88 @@
 
 [TOC]
 
-Coming soon...
+Parameter sensitivity analysis is useful for identifying the most important or influential parameters for a specified simulation target (e.g., streamflow at the watershed outlet) (Zhan et al., 2013). A typical sensitivity analysis procedure includes sampling from the value ranges of determined model inputs, evaluating the model with the generated samples and saving the interested outputs, and calculating sensitivity indices by various methods such as the Morris screening method (Morris, 1991) and FAST (Fourier Amplitude Sensitivity Test, Cukier et al., 1978). The sensitivity analysis tool in SEIMS is organized as separated functions according to these steps, which means it is easy to incorporate any sampling method and sensitivity analysis method. The most time-consuming step is repetitive model evaluations, which is parallelized at model level by parallel job management based on SCOOP (Hold-Geoffroy et al., 2014).
+
+# Simple usage
+For simple usage, open a CMD window, enter the following commands to execute the predefined parameter sensitivity analysis of the Youwuzhen watershed model.
+
+```python
+cd D:\demo\SEIMS\seims\test
+D:
+python –m scoop –n 2 demo_parameters_sensitivity.py -name youwuzhen
+```
+
+The runtime logs of the parameter sensitivity analysis including the commands of each model run and the results of sensitivity analysis were showed in Figure 2:6 1.
+
+Firstly, the sampled calibration values of selected parameters were organized as float arrays and imported as `CALI_VALUES` field into the `PARAMETERS` collection of the main spatial database. The SEIMS-based watershed model under a combination of calibrated parameters can be evaluated with the `calibrationID` argument set to the corresponding array index (start from 0) (Figure 2:6 1a). The calibration values can be queried by the following command in the console window of Robo 3T (Figure 2:6 2).
+
+```
+db.getCollection('PARAMETERS').find({'CALI_VALUES':{$exists:true}})
+```
+
+Then, all combinations of calibrated parameters were evaluated in parallel according to the workers’ number of SCOOP, e.g., two workers were allocated in this simple usage (-m scoop –n 2). After the evaluation of each calibrated model, the interested model performance indicators will be calculated and saved. In current version of SEIMS, NSE (Nash-Sutcliffe efficiency), RSR (root mean square error-standard deviation ratio), and PBIAS (percent bias) (Moriasi et al., 2007) were used as model performance indicators.
+Finally, the sensitivity analysis method (Morris screening method in this demo) will be executed according to the above introduced model outputs. For example, Figure 2:6 3 showed the parameter sensitivity analysis result with respect to the NSE of the simulated streamflow (m3s-1) at the watershed outlet. Each point in the screening plot represents one parameter of the Youwuzhen watershed model. The larger the modified means μ* (Campolongo et al., 2007), the more sensitive the watershed response is to variation in the parameter (Yang, 2011; Zhan et al., 2013). More results can be found in the output directory, e.g., `SEIMS\data\youwuzhen\demo_youwuzhen30m_longterm_model\PSA_Morris_N4L2`.
+
+# Configuration file of parameter sensitivity analysis
 
 Also see introduction of the Python package @subpage intro_parameters_sensitivity_pkg
+
+The configuration file of parameter sensitivity analysis, such as that of the Youwuzhen watershed model shown in Figure 2:6 4, includes four sections, i.e., SEIMS_Model, PARAMETERS, OPTIONAL_PARAMETERS, and OPTIONAL_MATPLOT_SETTING. Among them, OPTIONAL_PARAMETERS and OPTIONAL_MATPLOT_SETTING are optional. The names of sections and options should not be changed. 
+
++ SEIMS_Model: Basic settings of the SEIMS-based watershed model, see Section 2:4.3.2 for details. Note that the OpenMP version of SEIMS main program is used by default. However, the options of version, MPI_BIN, and processNum are supported if the MPI&OpenMP version is wanted.
++ PSA_PARAMETERS: Basic settings for parameter sensitivity analysis.
+    1.	evaluateParam: Simulation target(s) to be evaluated for sensitivity analysis, e.g., Q for streamflow, SED for sediment, and CH_TN for total nitrogen. This option is similar to PLOT_VARIABLES of postprocessing in Section 2:5.2. Multiple parameters should be separated by comma.
+    2.	paramRngDef: Filename of the definition of parameters’ range. The file follows the basic plain text file format of SEIMS. Each line indicates one parameter to be analyzed with the format of NAME, lower_bound, upper_bound. For example, K_pet,-0.3,0.3 means that the impact value of K_pet ranges from -0.3 to 0.3. The type of CHANGE is the same with that defined in PARAMETERS collection. Relative information please refer to Section 2:4.2.3.
+    3.	PSA_Time_start: Starting date time of the calculation of model performance for the sensitivity analysis with the format of YYYY-MM-DD HH:MM:SS.
+    4.	PSA_Time_end: Ending date time of the calculation of model performance for the sensitivity analysis with the format of YYYY-MM-DD HH:MM:SS.
++ Sections of specific sensitivity analysis methods supported by SALib. Currently, the Morris screening method including groups and optimal trajectories (Morris_Method) and Fourier Amplitude Sensitivity Test (FAST_Method) have been integrated and tested. In the future version of SEIMS, more methods and not limited to the methods of SALib should be integrated.
+    1.	Morris_Method:
+    N: The number of trajectories to generate.
+    num_levels: The number of grid levels. The resulting samples have a row number of (D+1)*N, and column number of D, where D is the number of parameters.
+    optimal_trajectories: The number of optimal trajectories to sample (between 2 and N) which can also be None.
+    local_optimization: Flag whether to use local optimization. Speeds up the process tremendously for bigger N and num_levels. If set to False brute force method is used.
+    2.	FAST_Method:
+    N: The number of samples to generate.
+    M: The interference parameter, i.e., the number of harmonics to sum in the Fourier series decomposition, the default is 4.
++ OPTIONAL_MATPLOT_SETTINGS: Plot settings for matplotlib, see Section 2:5.2.
+
+```
+[SEIMS_Model] 
+MODEL_DIR = D:\demo\SEIMS\data\youwuzhen\demo_youwuzhen30m_longterm_model
+BIN_DIR = D:\demo\SEIMS\bin
+HOSTNAME = 127.0.0.1
+PORT = 27017
+threadsNum = 2
+layeringMethod = 1
+scenarioID = 0
+# Simulation period (UTCTIME)
+Sim_Time_start = 2012-01-01 00:00:00
+Sim_Time_end = 2012-03-09 23:59:59
+
+[PSA_Settings]
+evaluateParam = Q
+paramRngDef = morris_param_rng-Q.def
+# Objective calculation period (UTCTIME)
+PSA_Time_start = 2012-02-27 00:00:00
+PSA_Time_end = 2012-03-09 23:59:59
+
+[Morris_Method]
+N = 4
+num_levels = 2
+optimal_trajectories = None
+local_optimization = True
+
+[OPTIONAL_MATPLOT_SETTINGS]
+FIGURE_FORMATS = PDF,PNG
+FONT_TITLE = Times New Roman
+DPI = 300
+```
+
+# Advanced usage
+The Python scripts of parameter sensitivity analysis are in `SEIMS/seims/parameters_sensitivity`. The main.py is the entrance which can be executed though the unified format of running SEIMS Python scripts, e.g.,
+
+```python
+cd D:\demo\SEIMS\seims\parameters_sensitivity
+python main.py -ini D:\demo\SEIMS\data\youwuzhen\workspace\sensitivity_analysis.ini
+```
+