mHM terminates without returning an error

[klekarkar]

Dear mHM community,

I am running mHM for the first time and I encountered a case where the model terminates quietly without returning an error.
The model loads the inputs but then stops after reading the number of outlets at L0 and L1. I have thoroughly checked the mhm.nml for any inconsistencies with the knowledge I have.
My model model resolutions are in EPSG 4326 as follows:
L0: 0.001935125
L1, L11: 0.0078125
L2: 0.25

The answer that was given to a similar case in the discussion did not appear to be clear (at least to me).

I would appreciate your assistance to get past this step. I have attached the mhm.nml for reference.

Thanks!

mhm_nml.zip

[ViniciusBMoreira]

It seems the problem is related to mRM.

As a first attempt, could you replace the "0" values in your fdir.asc file with -9999?

Vinicius B.

[StephanThober]

I also think so. Did you check that the river network is correct and contains no loops?

Stephan

[klekarkar]

Thanks for the suggestions, @StephanThober and Bryan
I followed the suggestion of Bryan and then investigated the files to find any loops but it appears there are none.
Even after implementing the suggestions, the model still terminates at the same point of the simulation.

[pallavhydro]

Hi @klekarkar

I have the following suggestions for debugging:

1. You are running the model at less than a km. Could you try running it at 0.25 degree (both $L_1$ and $L_{11}$)? That should clear out whether the issue is with the fdir.
2. Could you also make a run with current setting (fine resolution) but with the routing turned off i.e., processCase(8) = 0 in mhm.nml. And then share the output ConfigFile.log here.

Cheers
Pallav

[klekarkar]

Thanks a lot for the suggestions, @pallavhydro . I ran the model at 0.25 degrees and it worked alright, I have even run it at higher resolution. It was a matter of computation time because the domain was relatively large so I compiled it on a HPC and waited for it to finish.
One issue I ran into is that if I do not provide observed flow, the model still tries to calculate the performance but would (obviously) run into an arithmetic error. The error the model spits out is quite vague (attached screenshot) so it might be useful if something more explicit could be added into the model for this step. I had observations for my domain so I managed to fix it.

Is there a switch to turn off this model evaluation step, e.g., when one does not have any observations?
I will mark your reply an answer but I would appreciate your feedback on my last question.

Best wishes,
Katoria.

[pallavhydro]

Hi Katoria

Great to know you are progressing.

The error you sent is a bug. Sorry about that. There were some recent developments which is the source of the no gauge issue. We are on it!

For the time being, a hack could be, for setup where there are not gauge observations, the user can create a dummy observation file (say with -9999 values). That should get help to circumvent the bug for now.

Cheers
Pallav

[diegodinamarca]

@pallavhydro Sorry for reopening this thread but I'm having the same trouble running the model even at low resolution (0.25 degrees). It could be an fdir problem? (fdir attached below)
fdir.asc.zip

Tillage layers: the tillage horizons are modelled
                upto mHM layers:          3
    Reading data for domain: 1 ...
      Reading slope ...
      Reading aspect ...
      Reading soil ids ...
      Reading LAI ...
      Reading latitude/logitude ...
    in     0.145 seconds.
  in     0.180 seconds.
    in     0.180 seconds.
  Initialize domains ...
  in     0.153 seconds.
      Reading data for domain: 1 ...

    Domain:   1
      Number of outlets found at Level 0:..     346
      Number of outlets found at Level 11:.       0

If I run it with routing turned off it works (configFile attached)
ConfigFile.log

[pallavhydro]

1. In the screenshot I do not see the rest of the runtime output, or is it that it gets stuck at the last line shown?

2. Since the setup is not able to produce any outlet at L11, this could be a problem. Your L0 has outlets which is fine. Have you been able to run the setup yet, any resolutions? If not, better check the DEM which was the basis for the facc, fdir. e.g. did you fill the DEM before generating facc and fdir out of it?

P

[diegodinamarca]

1. it gets stuck at that last line
2. I was able to run it without routing, and yes I filled the DEM and burned the stream network before generating the facc and fdir. The facc is correctly generating the stream lines

[klekarkar]

Dear @diegodinamarca,
On top of Pallav's great answer, I have some suggestions to check if your model actually stopped running, or if it's quietly running but it's taking long to get to the next step (I realized my case was because my L11 resolution was too high so it was taking time for the model to run).

1. First, run the model at a reasonably coarse L11 resolution without turning off the routing. For instance, you can set L11=L2. If the model runs successfully and produces output at a coarser scale, then any issues at high resolution are likely related to computational limits or runtime.

2. If you're on an HPC system, avoid launching ./mhm directly in the terminal. Instead, queue it as a job using a scheduler like SLURM. This keeps it running in the background and gives you better control over memory usage and also gives you error notifications when your model runs into errors. If you're running mHM on your local machine (e.g., with WSL on Windows), you can’t use SLURM, but you can still queue jobs using tools like Task Spooler ( I have not run mHM locally on my PC because my domain is quite big so I havent used Task Spooler before).

Here is an example of how I queue mHM to run on a HPC

#!/bin/bash
#SBATCH --job-name=run_mhm__simulation              # Name of the job
#SBATCH --output=output_%j.log                  # Log file for output (%j is the job ID)
#SBATCH --error=error_%j.log                    # Log file for errors
#SBATCH --ntasks=1                                 # Number of tasks (usually 1 for a single program execution)
#SBATCH --cpus-per-task=8                       # Number of CPU cores per task (adjust based on model requirements)
#SBATCH --mem=128G                               # Memory required (adjust as needed)
#SBATCH --time=120:00:00                         # Maximum runtime (adjust as needed)
#SBATCH --partition= <name of your partition >                        # Partition or queue name (you can check this with your system admin)
#SBATCH --mail-user=<your email address> 
#SBATCH --mail-type=END,FAIL
 

# Change to the directory where the model is located
cd <directory where mHM executable is located>

# Run the model
./mhm

I put this code in a bash file and then i run this file on the terminal.
In this example you would get a notification when the model runs completes or fails and you also get an error log file about what caused the model to fail.

I hope this helps

[joshfaf]

Title: v5.13.2 stops at mRM initialization when testing routing with dummy gauge

Hello, I need help with this.

I am working with mHM v5.13.2 on Ubuntu and setting up a custom single-domain model. My hydrological and routing resolutions are both 0.05° (equal resolutions).

The model runs when routing is turned off, so the general domain setup, forcings, and hydrological configuration seem fine.

I then wanted to test routing before moving on to calibration, so I enabled routing and added a dummy gauge. I also switched several output variables to .FALSE. in the output namelists to keep the run light and focus only on whether routing initializes correctly. In particular, I reduced optional routed/output variables and kept only the minimum necessary outputs for debugging.

However, when routing is enabled, the model stops during mRM initialization with the following error:

***ERROR: mhm.nml: ID -9999 of evaluation gauge 1 for subdomain 1 is not defined!
Error occured in namelist: evaluation_gauges
STOP 1

This happens after the model reads the namelists successfully and confirms that hydrological and routing resolutions are equal.

So in summary:

• Version: mHM v5.13.2
• Hydrology resolution: 0.05°
• Routing resolution: 0.05°
• Model runs without routing
• Output flags were reduced to .FALSE. where possible to simplify the test run and reduce output load
• When routing is enabled for a test run using a dummy gauge, initialization fails with the gauge ID error above

I will attach a screenshot of the full output.

[klekarkar]

Hi @joshfaf,
Can you check that your evaluation gauge exists in the idgauges.asc file. Please also check if that you have your dummy station timeseries as a .txt file. Make sure the gauge number in your mhm.nml matches the name in the idgauges.asc.
Could you attach your mhm.nml file?

Good luck with your project.

[joshfaf]

Hi, thank you for your response.

I checked the gauge setup and corrected it. The evaluation gauge now exists in idgauges.asc, the gauge number in mhm.nml matches the raster ID, and the dummy station file 00001.txt is present in input/gauge/ using the required mHM timeseries format.
After fixing that, the model now proceeds much further: it reads all inputs, initializes routing, reads forcings, and starts the run. It reaches (attached)

but then crashes with:

double free or corruption (out)
Program received signal SIGABRT

I am currently using:

mHM version 5.13.2
Single domain test case over Ghana
Hydrology resolution = Routing resolution = 0.05°

Could this crash be related to the equal hydrology/routing resolution setup, or possibly another configuration issue during MPR/allocation?

I have attached my mhm.nml file as requested.
Thank you again for your help.

mhm.txt

[klekarkar]

I suspect the error is because the first value of your idgauges file is a float not an integer. Open the file and check that the first value is e.g. 0 and not 0.0 or -9999 not -9999.0.

Good luck

[joshfaf]

Hello @klekarkar ,

Thank you for your response.

I checked and corrected several possible causes that I suspected, but the crash still persists.

So far I verified:

1. gauge rasters and dummy gauge file are valid with no floats except the headers
2. headers/extents of main morpho ASCII files match
3. DEM and other morpho data masks are consistent
4. soil_class.asc IDs are valid
5. I even replaced the full soil database with a single soil class and minimal soil definition for testing and that didn't work either

The model now successfully reads config, morpho data, routing, gauges, forcing data, and starts MPR, then crashes (attached)

Could hidden issues in morpho inputs (landcover / LAI / geology / raster values / categorical ranges) be a possible cause, even though they are read successfully?

Would you recommend any specific checks for morpho files used during MPR?

Thank you.

[joshfaf]

Hello @klekarkar,

I wanted to share an update on the issue above. The problem was not related to the general model setup, but specifically to the routing configuration. The run failed with "processCase(8) = 1", but completed successfully when I changed it to "processCase(8)" to either "2" or "3".

Thank you for your input.