Stochastic Tools Batch Mode HPC Hang

[millerzac]

Check these boxes if you have followed the posting rules.

• Q&A General is the most appropriate section for my question
• I have consulted the posting Guidelines on the Discussions front page
• I have searched the Discussions forum and my question has not been asked before
• I have searched the MOOSE website and the documentation does not answer my question
• I have formatted my post following the posting guidelines (screenshots as a last resort, triple back quotes around pasted text)

Question

Hi!

I have been using/building various stochastic tools and have found that in different scenarios while using the batch mode and running a job on INL HPC (PBS or SLURM), my job will get hung up after completing the first iteration during a batch-reset job.

For example:
I am running a fairly complex parameter study with 16 parameters, LHS sampling, with a complex 2D bison fuel pin as my sub-app. When setting min_procs_per_app/row to 8 with 2400 rows and requesting 48 tasks on 1 node, in SLURM, it properly creates 400 batch iterations (as seen in my "INITIAL" output file) but when rank 0 gets done with its first iteration, it doesn't move onto the next step, instead it gets hung up and the solver continues in the background indefinitely.

I have found that on PBS (sawtooth), when requesting 1 node with 48 tasks, it will run fine with 8 min_proc_per_app/row however if I change it to 6, it gets hung up. When the batch system properly gets through the process, it will move through all 400 iterations (I will see 400 rank 0 processes in the output file), once complete it informs me that y/X samples complete until I get X/X complete and it will finish and output the results.

I have tried to ensure there is sufficient memory, various different combinations of nodes and tasks and min_procs_per_app/row but have not found a solution for SLURM. Is this a known issue/problem or is there guidance on how to properly utilize batch mode and HPC resources?

Thanks!

   Finished Executing MultiApps                                                     
    Linear solve converged due to CONVERGED_RTOL iterations 1
    Linear solve converged due to CONVERGED_RTOL iterations 1
    Linear solve converged due to CONVERGED_RTOL iterations 1
    Linear solve converged due to CONVERGED_RTOL iterations 1
    Linear solve converged due to CONVERGED_RTOL iterations 1
    Linear solve converged due to CONVERGED_RTOL iterations 1
    Linear solve converged due to CONVERGED_RTOL iterations 1
    Linear solve converged due to CONVERGED_RTOL iterations 1
    Linear solve converged due to CONVERGED_RTOL iterations 1
    Linear solve converged due to CONVERGED_RTOL iterations 1
    Linear solve converged due to CONVERGED_RTOL iterations 1
    Linear solve converged due to CONVERGED_RTOL iterations 1
    Linear solve converged due to CONVERGED_RTOL iterations 1
    Linear solve converged due to CONVERGED_RTOL iterations 1
    Linear solve converged due to CONVERGED_RTOL iterations 1
    Linear solve converged due to CONVERGED_RTOL iterations 1
    Linear solve converged due to CONVERGED_RTOL iterations 1
    Linear solve converged due to CONVERGED_RTOL iterations 1
    Linear solve converged due to CONVERGED_RTOL iterations 1
  Nonlinear solve converged due to CONVERGED_FNORM_ABS iterations 5
    Linear solve converged due to CONVERGED_RTOL iterations 1
    Linear solve converged due to CONVERGED_RTOL iterations 1
  Nonlinear solve converged due to CONVERGED_FNORM_ABS iterations 10
    Linear solve converged due to CONVERGED_RTOL iterations 1
    Linear solve converged due to CONVERGED_RTOL iterations 1
    Linear solve converged due to CONVERGED_RTOL iterations 1
    Linear solve converged due to CONVERGED_RTOL iterations 1
    Linear solve converged due to CONVERGED_RTOL iterations 1
    Linear solve converged due to CONVERGED_RTOL iterations 1
  Nonlinear solve converged due to CONVERGED_FNORM_ABS iterations 7
    Linear solve converged due to CONVERGED_RTOL iterations 1
  Nonlinear solve converged due to CONVERGED_FNORM_ABS iterations 7
    Linear solve converged due to CONVERGED_RTOL iterations 1
    Linear solve converged due to CONVERGED_RTOL iterations 1
    Linear solve converged due to CONVERGED_RTOL iterations 1
    Linear solve converged due to CONVERGED_RTOL iterations 1
    Linear solve converged due to CONVERGED_RTOL iterations 1
    Linear solve converged due to CONVERGED_RTOL iterations 1
    Linear solve converged due to CONVERGED_RTOL iterations 1
    Linear solve converged due to CONVERGED_RTOL iterations 1
    Linear solve converged due to CONVERGED_RTOL iterations 1
  Nonlinear solve converged due to CONVERGED_FNORM_ABS iterations 5
      Still Solving Step Batch For SamplerFullSolveMultiApp    Linear solve converged due to CONVERGED_RTOL iterations 1
    Linear solve converged due to CONVERGED_RTOL iterations 1
    Linear solve converged due to CONVERGED_RTOL iterations 1
  Nonlinear solve converged due to CONVERGED_FNORM_ABS iterations 3

[GiudGiud]

Hello

I have found that on PBS (sawtooth), when requesting 1 node with 48 tasks, it will run fine with 8 min_proc_per_app/row however if I change it to 6, it gets hung up

what about just 2 nodes?

Did you use the HPC-provided MPI compiler? You cannot use the conda or locally installed MPI distributions when working on HPC
What does moose/scripts/diagnostics.sh return ?

[millerzac]

So I just tried it on PBS with 4 nodes and 48 tasks per node and also on SLURM with 1 node and 48 tasks. I found that in both cases, all of the groupings of tasks that are NOT rank 0 are progressing through properly to the next step however the first batch of tasks (rank 0), in my case stdout.processor.0-7, get to the end the of the first solve and then stalls at this:

        Finished Executing MultiApps                                                     [�[33m722.17 s�[39m] [�[33m  523 MB�[39m]
      Still Solving Step Batch For SamplerFullSolveMultiApp.........................................................................```

[millerzac]

So after some debugging of the sampler codes, I have found that while using batch-reset, the LHS sampler gets put in LOCAL mode and the shuffle function was using "SEMI-LOCAL" which was causing rank 0 to get stuck. I have switched this to "NONE" and now the system progresses properly. I need to do more investigation into the impacts of this but none are found as of yet. shuffle(local, col + getNumberOfCols(), CommMethod::SEMI_LOCAL); -> shuffle(local, col + getNumberOfCols(), CommMethod::NONE);

Additionally, while performing all of the debugging, I also noticed that if I don't use --keep-cout --redirect-stdout I get segment faults when all of the processors are trying to send their information to a single output instead of creating independent outputs for all of the processors.

[zachmprince]

Thank you for looking deep in the code! I will take a look to see if I can recreate the issue you're seeing with LHS. I don't think the change to NONE is quite the correct solution since it will affect the algorithm.

As for the race condition for the output, are you setting file_base in your sub-app input? If so, that would definitely cause conflict in writing the outputs. Batch-reset should be outputting to unique outputs if that is not set...

[zachmprince]

@millerzac I can't seem to recreate the issue with a minimal example. If you could post your stochastic tools input, I can maybe try with something closer with what you're running.

If the issue is really the LHS sampler. One work around that doesn't require modifying the code is to save the samples into a CSV then use a CSVSampler in your main input. Basically, you create an input that just outputs the LHS samples (which can be run a single processor):

[StochasticTools]
[]

[Distributions]
  [uniform]
    type = Uniform
    lower_bound = 0
    upper_bound = 1
  []
[]

[Samplers]
  [lhs]
    type = LatinHypercube
    distributions = uniform
    num_rows = 2400
  []
[]

[Reporters]
  [sampling_matrix]
    type = StochasticMatrix
    sampler = lhs
    parallel_type = ROOT
  []
[]

[Outputs]
  csv = true
  execute_on = timestep_end
[]

You can then replace the LHS sampler in your input with this:

[Samplers]
  [lhs]
    type = CSVSampler
    samples_file = <csv_out_input_name>_out_sampling_matrix_0001.csv
    min_procs_per_row = 6
    execute_on = PRE_MULTIAPP_SETUP
  []
[]

[millerzac]

Hi! The CSVSampler is a great idea. So far it is running well and got through the first iteration. It takes about 12-24 hours to finish all 2400 samples so I will let you know if this processes completely.

Looking at the sampler code that you are trying, since you are not using batch-reset, it calls the GLOBAL 'for loop' in the sampling code for the LHS which the shuffle is set to NONE. I believe this would follow the same shuffle process as what I had when I changed the LOCAL 'for loop' shuffle to NONE. It looks like in the LHS sampler, if batch-reset is called, it goes through the LOCAL 'for loop' and thats where the SEMI_LOCAL is located. I will try and put together a basic script that gets the same issues I have been having so you can investigate further.

Ultimately, the goal of all of this is to get the LHS sampler to run through all of my parameters so I can use the outputs to create multiple surrogate models that will be fed into my new multi variate MCMC sampling. Right now, the MCMC takes a week to gather only 300-400 samples so a surrogate will help gather more samples. I have manipulated the surrogate code so that I can feed CSV files of parameters and outputs to it, then train and test it. This way I can run 1000+ iterations in a fraction of the time.

[zachmprince]

Here is the input I was actually trying. It seems to work 48 procs with min_procs_per_row = 8, 48 procs with min_procs_per_row = 6, 96 procs with min_procs_per_row = 8, and 96 procs with min_procs_per_row = 6. The sub-app is a simple 1D diffusion, which probably isn't indicative of the model you are actually running.

[StochasticTools]
[]

[Distributions]
  [uniform]
    type = Uniform
    lower_bound = 0
    upper_bound = 1
  []
[]

[Samplers]
  [lhs]
    type = LatinHypercube
    distributions = uniform
    num_rows = 2400
    min_procs_per_row = 6
    execute_on = PRE_MULTIAPP_SETUP
  []
[]

[GlobalParams]
  sampler = lhs
[]

[MultiApps]
  [runner]
    type = SamplerFullSolveMultiApp
    input_files = sub.i
    min_procs_per_app = 6
    mode = batch-reset
  []
[]

[Controls]
  [cmdline]
    type = MultiAppSamplerControl
    multi_app = runner
    param_names = 'BCs/left/value'
  []
[]

[Transfers]
  [data]
    type = SamplerReporterTransfer
    from_multi_app = runner
    stochastic_reporter = results
    from_reporter = 'pp/value'
  []
[]

[Reporters]
  [results]
    type = StochasticMatrix
  []
[]

[millerzac]

So the CSV sampler has fixed the issue!
However, I will need to spend a little more time trying to replicate my issue with a simpler output. I will keep this open for now as I believe this is an issue within the samplers that still needs to be addressed.