Checkpoints and restarts

[dbeurle]

During long simulations it is useful to write out the state of the simulation such that it can be restarted from this checkpoint. The checkpoint should contain:

1. History variables (accumulated scalars / tensors)
2. Current displacements or nodal coordinates
3. Current time step and step size.

Required changes

The classes which hold state will require new constructors or logic to populate accumulation fields from restart data. Whether the restart should handle that different data as input is debatable. If there is a checkpoint at which point a simulation diverges due to bad material properties or time expiry on a shared compute device, it would be sensible to allow a change however this would break the continuity of the analysis and should perhaps be avoided entirely.

Each constitutive model should specify the internal variables which are listed as the history variables and provide this set through a constant method. The fem_submesh that is responsible for the internal variables is also responsible for serialisation of the scalar and tensor variables. Due to the potentially large size of the internal variable set, the results should be in binary format with the option of an ASCII output.

Classes with restart information could provide this in json format so this could be written / appended to a restart file <simulation_name>.restart with a mapping to a binary object containing the history variables. This is best handled in the fem*matrix classes as the time / load incrementation occurs here.

Serialisation libraries

The cereal library is header only and allows for a binary representation of the matrices. It seems somewhat easy to implement:

#include <Eigen/Dense>

#include <cereal/archives/binary.hpp>
#include <cereal/cereal.hpp>

#include <fstream>
#include <type_traits>

namespace cereal
{
template <class Archive, typename MatrixType>
inline std::enable_if_t<traits::is_output_serializable<BinaryData<typename MatrixType::Scalar>, Archive>::value> save(
    Archive& archive, MatrixType const& m)
{
    typename MatrixType::Index rows = m.rows();
    typename MatrixType::Index cols = m.cols();
    archive(rows);
    archive(cols);
    archive(binary_data(m.data(), rows * cols * sizeof(typename MatrixType::Scalar)));
}

template <class Archive, typename MatrixType>
inline std::enable_if_t<traits::is_input_serializable<BinaryData<typename MatrixType::Scalar>, Archive>::value> load(
    Archive& archive, MatrixType& m)
{
    typename MatrixType::Index rows, cols;
    archive(rows);
    archive(cols);

    m.resize(rows, cols);

    archive(binary_data(m.data(),
                        static_cast<std::size_t>(rows * cols * sizeof(typename MatrixType::Scalar))));
}
}

int main()
{
    Eigen::MatrixXd test1 = Eigen::MatrixXd::Random(10, 3);
    Eigen::MatrixXd test2 = Eigen::MatrixXd::Random(3, 5);

    {
        std::ofstream out("eigen.cereal", std::ios::binary);
        cereal::BinaryOutputArchive archive_o(out);
        archive_o(test1);
        archive_o(test2);
    }

    std::cout << "test1:" << std::endl << test1 << std::endl;
    std::cout << "test2:" << std::endl << test2 << std::endl;

    Eigen::MatrixXd test1_loaded, test2_loaded;
    {
        std::ifstream in("eigen.cereal", std::ios::binary);
        cereal::BinaryInputArchive archive_i(in);
        archive_i(test1_loaded, test2_loaded);
    }
    std::cout << "Norm of test matrix 1: " << (test1 - test1_loaded).norm() << "\n";
    std::cout << "Norm of test matrix 2: " << (test2 - test2_loaded).norm() << "\n";
}

and also supports containers in the standard library and json output (https://uscilab.github.io/cereal/index.html).

Implementation steps

• Have constitutive models populate a set of restart variables
• Print out current time information
• Create a checkpoint class to give to fem_submesh classes with file name information

Each fem_submesh owns and can print out their internal variables. This is because each submesh has a unique interpolation function and can printout based on this information alone.

Constitutive model

• Add a serialise() method using the code snippet.

This function would need to include its parent (information about the element or an index) and a method to construct its data with restart information, using pseudo c++:

class internal_variables
{
public:
    /// Serialise data to file
    void serialise(std::string const& file_tag,
                      std::set<variable::scalar> const& scalars
                      std::set<variable::second> const& tensors);

    /// Deserialise from file to memory
    void deserialise(std::string const& file_tag);
};