Directory Issues with Nested Workflows

[SWeav02]

Hi all,

I'm trying to create a seeded workflow similar to that in the guides that will pass results between runs. The goal is to run a PBE level static-energy calculation followed by an HSE level calculation. I want it to use the WAVECAR from the initial calculation. What I keep finding is that if I create my run_config method similar to below it will just rerun the initial PBE calculation:

class StaticEnergy__Warren__SeededHse(Workflow):
            use_database = False
            @classmethod
            def run_config(
                        cls,
                        structure: Structure,
                        command: str = None,
                        source: dict = None,
                        directory: Path = None,
                        **kwargs,
            ):

                        pbe_results = cls.StaticEnergy__Warren__Pbe.run(
                                   structure=structure,
                                    command=command,
                                    source=source,
                                    directory=directory
                                    ).result()
                        hse_results = cls.StaticEnergy__Warren__Hse.run(
                                    structure=structure,
                                    command=command,
                                    source=source,
                                    directory=directory
                                    ).result()

If I instead save the calculations to two seperate directories (eg. directory=directory / "PBE" and directory = directory / "hse") the calculations will run as expected. Also, if I insert a line to copy the WAVECAR over to the hse directory it will run based on that WAVECAR as expected.

Any idea why the second workflow wouldn't just run after the first one?

[jacksund]

The goal is to run a PBE level static-energy calculation followed by an HSE level calculation. I want it to use the WAVECAR from the initial calculation.

Is this normal to use a PBE WAVECAR as the input for HSE?? Seems off to me, but if you've read other publications that have done it, I guess it's fine. If anything, you could maybe pass along CHGCAR as the starting point and make sure the grids match. Who knows!

(this is a big guess here but...) there's a chance VASP is using your vasprun.xml (or some other file) from PBE and assuming your next calc is same and/or a continuation calc. I wouldn't want to mess with this possibility, so I'd stick to using two directories.

Any idea why the second workflow wouldn't just run after the first one?

What you have coded out is correct and it doesn't re-run the PBE (unless you have a coding error elsewhere). What it's doing is running the HSE calculation in the same directory that the PBE one was ran in because you are giving them the same directory input.

If you are fully convinced that the PBE is just running again, then double check your HSE workflow. Either you are importing/defining it incorrectly or something is up with your "setup" method (or VASP is acting funky like I suggest above). Feel free to share that your workflow code here too.

But I'd recommend sticking to separate directories. Overwriting files by re-running vasp in the same folder can run into confusing issues on where/how files were created. Here's how I'd write the seeded workflow:

import shutil
from simmate.utilities import get_directory


class StaticEnergy__Warren__SeededHse(Workflow):
    use_database = False

    @classmethod
    def run_config(
        cls,
        structure: Structure,
        command: str = None,
        source: dict = None,
        directory: Path = None,
        **kwargs,
    ):
        # SETUP DIRECTORIES
        # note: get_directory is just a utility that creates the directory if
        # it isn't already there and returns the name. Normally it's called
        # within workflow.run() but here we just make the folders up front.
        pbe_dir = get_directory(
            directory / cls.StaticEnergy__Warren__Pbe.name_full
        )
        hse_dir = get_directory(
            directory / cls.StaticEnergy__Warren__Hse.name_full
        )

        # RUN PBE
        pbe_results = cls.StaticEnergy__Warren__Pbe.run(
            structure=structure,
            command=command,
            source=source,
            directory=pbe_dir,
        ).result()

        # copy over WAVECAR
        shutil.copyfile(
            src=pbe_dir / "WAVECAR", 
            dst=hse_dir / "WAVECAR",
        )

        # RUN HSE
        hse_results = cls.StaticEnergy__Warren__Hse.run(
            structure=structure,
            command=command,
            source=source,
            directory=hse_dir,
        ).result()

[SWeav02]

This is what I ended up doing and it works fine. I don't think it's normal to seed HSE with PBE, but I wanted to see what it would do. I also ran into a similar issue when running when running a BadELF workflow that I've been working on. It runs a relaxation, then static energy calc, and then several short workflows for the BadELF analysis. I noticed it wasn't creating an ELFCAR and that the INCAR was still set up for a relaxation when the workflow failed at the BadELF stage. I solved this the same way, by using two directories. Here's what that workflow would have looked like before switching to two directories:

class BaderBadelfRelaxationBase(Workflow):
    """
    Base class for running a PBE quality relaxation followed by a static energy
    and bader or badelf analysis.

    This should NOT be run on its own. It is meant to be inherited from in
    other workflows.
    """

    # We don't need to save anything from this parent workflow
    use_database = False
    relaxation_workflow = None  # This will be defined in inheriting workflows
    population_analysis_workflow = None  # This will be defined in inheriting workflows

    @classmethod
    def run_config(
        cls,
        structure: Structure,
        command: str = None,
        source: dict = None,
        find_empties: bool = True,
        directory: Path = None,
        min_charge: float = 0.15,
        **kwargs,
    ):
        # run a relaxation at PBE quality
        relaxation_result = cls.relaxation_workflow.run(
            structure=structure,
            command=command,
            source=source,
            directory=directory,
        ).result()

        bader_result = cls.population_analysis_workflow.run(
            structure=relaxation_result,
            command=command,
            source=source,
            find_empties=find_empties,
            directory=directory
            min_charge=min_charge,
        ).result()

The relaxation workflow is just a standard relaxation workflow similar to relaxation.vasp.matproj. The population analysis workflows that I'm using look this:

class VaspBadElfBase(Workflow):
    """
    Runs a static energy calculation using an extra-fine FFT grid and then
    carries out Bader analysis on the resulting charge density using the ELFCAR
    as a reference when partitioning. This is the base workflow that all of the
    BadELF workflows are built out from.
    """

    static_energy_prebadelf: Workflow = None  # This must be defined in inheriting class
    use_database = False

    @classmethod
    def run_config(
        cls,
        structure: Structure,
        command: str = None,
        source: dict = None,
        find_empties: bool = False,
        directory: Path = None,
        min_charge: float = 0.15,
        **kwargs,
    ):
        if find_empties:
            # Run static_energy calculation
            prebadelf_result = cls.static_energy_prebadelf.run(
                structure=structure,
                command=command,
                source=source,
                directory=directory,
            ).result()
            # Run badelf on initial output
            PopulationAnalysis__Warren__BadelfInit.run(
                structure=structure,
                directory=prebadelf_result.directory,
            ).result()
            # Find electride sites, place empty atoms, and generate chgcar
            # like files for each atomic site
            PopulationAnalysis__Warren__GetAtomChgcar.run(
                directory=prebadelf_result.directory, 
                analysis_type="badelf",
                min_charge=min_charge,
            )
            # Run Warren lab version of BadELF algorithm
            PopulationAnalysis__Warren__BadelfIonicRadii.run(
                directory=prebadelf_result.directory,
            )

        else:
            prebadelf_result = cls.static_energy_prebadelf.run(
                structure=structure,
                command=command,
                source=source,
                directory=directory,
            ).result()

            # We don't need to generate empty atom files because we're not looking
            # for electride sites here. Just run the badelf algorithm here.
            PopulationAnalysis__Warren__BadelfIonicRadii.run(
                directory=prebadelf_result.directory,
                empty_structure_file="POSCAR",
                partition_file="ELFCAR",
                charge_file="CHGCAR",
            ).result()

Here again the static energy prebadelf workflow that I use is similar to static-energy.vasp.matproj.

Switching the directories definitely works fine, but I'm not sure why this should happen.

[jacksund]

Can you check if there's a "Calculation is already completed. Skipping setup." message in your workflow logs? I bet there's a bug in this section's logic:

simmate/src/simmate/engine/s3_workflow.py

Lines 141 to 178 in 7fe2d38

	# When handling restarted calculations, we check for the final summary
	# file and if it exists, we know the calculation has already completed
	# (and therefore doesn't require a restart). This helps handle nested
	# workflows where we don't know which task to restart at.
	summary_filename = directory / "simmate_summary.yaml"
	is_complete = summary_filename.exists()
	is_dir_setup = cls._check_input_files(directory, raise_if_missing=False)
	# run the setup stage of the task, where there is a unique method
	# if we are picking up from a previously paused run.
	if (not is_restart and not is_complete) or not is_dir_setup:
	cls.setup(directory=directory, **kwargs)
	elif is_restart and not is_complete:
	cls.setup_restart(directory=directory, **kwargs)
	else:
	logging.info("Calculation is already completed. Skipping setup.")
	# now if we have a restart OR have an incomplete calculation that is being
	# restarted, we can check our files and run the external program
	if not is_restart or not is_complete:
	# make sure proper files are present
	cls._check_input_files(directory)
	# run the shelltask and error supervision stages. This method returns
	# a list of any corrections applied during the run.
	corrections = cls.execute(directory, command)
	else:
	logging.info("Calculation is already completed. Skipping execution.")
	# load the corrections from file for reference
	corrections_filename = directory / "simmate_corrections.csv"
	if corrections_filename.exists():
	data = pandas.read_csv(corrections_filename)
	corrections = data.values.tolist()
	else:
	corrections = []
	# OPTIMIZE: this same code is at the start of the execute method.
	# Consider making it into a utility.

When I wrote this section, I assumed 1 directory = 1 vasp calc. I still think this should be a rule, but we can add some user warnings to prevent confusion like what you ran into.

[jacksund]

Also, you may want to use the copy_previous_directory setting in workflows. Here's an example of where I use it:
https://github.com/jacksund/simmate/blob/main/src/simmate/apps/vasp/workflows/electronic_structure/base_full.py

This was a test feature, but I'm guessing there will be a lot more need for it -- so I'll add guides when I can.

[SWeav02]

That explains it! I did get a "Calculation is already completed. Skipping setup." message. I also get the same message for some of the S3Workflows (bader and combining AECCAR0 and AECCAR2) that are being run in the same directory, but those seem to run fine. I'm guessing those are fine, though maybe it would be better to run them in a new directory as well.

[jacksund]

nice, at least we narrowed it down! I need to brainstorm how to best handle this logic with "continuation" calculations. I'm thinking we set a rule that 1 workflow run = 1 directory for all simmate flows. For workflows like Bader + BadELF, we might label them such that they require copy_previous_directory=True or maybe even require specific files copied to work, and prevent new workflows from running inside a directory that already has a simmate_summary.yaml file. There will be added overhead from copying the files + having duplicates, but I think we should prioritize clean directories over the smallest possible outputs. This will be important when we want to load old workflow runs into the database. Thoughts?

[SWeav02]

This seems like a good call to me! The only issue I currently see with some of the BadELF/Bader stuff is that they require some additional commands (like combining AECCAR0/AECCAR2) which are seperate workflows at the moment. My version of BadELF also requires bader to be run beforehand to generate some of the output files. I might switch these from workflows to basic functions. I'll loose some convenience with error readouts when things don't run smoothly, but it would avoid issues with the simmate_summary.yaml file being found in the directory.