About the restart time

[keithbennett]

In GitLab by Zhongwei on 19 Jan 2021, 12:51 UTC (GitLab issue GL#2300)

Dear developers,

I have a restart file (about 972G, generated on Jan 17 14:35, e.g., r0004.sdf). When I try to continue the run from the restart file r0004.sdf. 1 day past, the code is running, but the status is still reading...

Usually, there is a time-wall for jobs on HPC.

So may I ask did you do some tests on the loading time for the Epoch code? If you could provide a Table, e.g.,

the size of the restart file,cost time to restart

100MB,10s

1GB,5min

1TB,?

etc.

It will be really appreciated.

[keithbennett]

In GitLab by Zhongwei on 19 Jan 2021, 12:52 UTC

changed the description

[keithbennett]

In GitLab by Zhongwei on 19 Jan 2021, 12:52 UTC

changed the description

[keithbennett]

In GitLab by Thomas Goffrey (@TomGoffrey) on 19 Jan 2021, 13:43 UTC

Unfortunately such information is going to depend on the filesystem you're running on, the MPI implementation, and how many cores you are using. You'd have to do your own benchmarking for the system you are using.

If I was to provide information for a system I use, it isn't going to be necessarily useful for you.

My advice would be to benchmark your own system. See how loading times vary with file size, and numbers of cores. You could report those numbers to whoever manages the system you are using, and hopefully they could advise you further.

24 hours+ for the restart file to load sounds like a problem, but I think your best hope in diagnosing the problem is to do such a benchmark for the system you are using and investigate from there.

[keithbennett]

In GitLab by Zhongwei on 19 Jan 2021, 14:28 UTC

@tom
I see. Thank you very much!

[smallrainbow]

@keithbennett
Dear developer,

The new version seems only can read the restart file named as, e.g., 0001.sdf instead of r0001.sdf or something else.
Could you help check why?
Thank you!

[TomGoffrey]

@smallrainbow could you please avoid asking unrelated questions on existing issues.

You should be able to specify restart_snapshot = r0001.sdf in the input deck. If this doesn't work, please open a new issue and I will investigate further.