Simplify checkpointing, change defaults, fix legacy, implement in batched

[prckent]

Proposed changes

This PR aims to simplify and improve the checkpointing process.

As mentioned in #4633 the documented every N blocks checkpointing feature was not working as expected. In legacy, this required storeconfigs to also be specified to work (undocumented/unexpected). Checkpointing was also not implemented in the batched drivers.

Here:

• checkpoint={-1,0,N} driver parameter option sufficient to control checkpointing
• Default changed to checkpoint=0. Configs will be written at the end of each QMC section by default. -1 disables. N sets write period in addition to end of section write.
• Removed storeconfigs option completely.
• Implemented checkpointing in batched code.
• Updated docs appropriately

Have verified by eye that the checkpoints look sensible and are updated periodically. Open to testing suggestions.

Batched code does not create a info.xml as part of checkpoint.

Also remove unused FastGrad option.

What type(s) of changes does this code introduce?

• Bugfix
• New feature

Does this introduce a breaking change?

• No

What systems has this change been tested on?

sulfur gcc build only. PR for CI.

Checklist

Update the following with a yes where the items apply. If you're unsure about any of them, don't hesitate to ask. This is
simply a reminder of what we are going to look for before merging your code.

• Yes/ This PR is up to date with current the current state of 'develop'
• Yes/ Code added or changed in the PR has been clang-formatted
• (Existing restart tests use code paths). This PR adds tests to cover any new code, or to catch a bug that is being fixed
• Yes. Documentation has been added (if appropriate)

[ye-luo]

Based on the documentation the default value of checkpoint is -1, should it be better to have 0 as default, namely end of driver section?

[prckent]

I agree. Pushed change and will update PR text

[prckent]

Test this please

[ye-luo]

@prckent your fix brings me attention that we don't have the corresponding tests for batched drivers.

[prckent]

Agree we are missing some tests here. Which were you thinking of? batched dmc restart tests? or something else?

I don't think that we have any tests on checkpoints saved within a section (as opposed to at the end), whether legacy or batched. The restart tests use the "checkpoints" saved at the end of a section. It is not obvious how to reliably catch the checkpoints written within a section for testing.

[ye-luo]

Agree we are missing some tests here. Which were you thinking of? batched dmc restart tests? or something else?

I don't think that we have any tests on checkpoints saved within a section (as opposed to at the end), whether legacy or batched. The restart tests use the "checkpoints" saved at the end of a section. It is not obvious how to reliably catch the checkpoints written within a section for testing.

That types of tests can only be made on the unit test side. We can simply run a driver with coordinates dumping in the middle, then load it back without using a driver and check against fixed numbers.

[prckent]

Test this please