upgrading simmate

[hyesol]

Hi!

I recently upgraded Simmate from version 0.14.0 to 0.17.0 and am now encountering issues when trying to run it. When I run a workflow using simmate, an error code appears as follows:


I’ve been using DigitalOcean6 to save and reload data in this environment. Here is the relevant database configuration:

default:
  ENGINE: django.db.backends.postgresql
  HOST: 
  NAME: simmate00
  USER: 
  PASSWORD: 
  PORT: 
  OPTIONS:
    sslmode: require

However, when I run the following:

from simmate.configuration import settings
print(settings.database)

it returns:
{'engine': 'django.db.backends.sqlite3', 'name': PosixPath('/home/WarrenLab/simmate/hyesol_env_CoO-database.sqlite3')}, instead of pointing to the file: /home/WarrenLab/simmate/hyesol_env_CoO-database.yaml

Was there anything I needed to do when upgrading to Simmate 0.17.0 that I may have missed? Could you please help me troubleshoot this issue?

Thank you.

[jacksund]

For the database connection, you're using the old settings config, which I gave the fix for here: #683

As for the InegrityError: FOREIGN KEY constraint failed, this a django/database error. It happens when you try to insert a row where the id or unique column is already present (or used in the past). There are subtle cases where you should use create_or_update on a database object instead of create or save. Are you able to share the workflow? I could help debug and see if its actually a bug on simmate's end

[hyesol]

I have three modified workflows that I want to run as subworkflows within a main workflow.

Modified Workflows:

1. First Relaxation Calculation:

from simmate.workflows.utilities import get_workflow
from simmate.apps.vasp.inputs import PBE_POTCAR_MAPPINGS

original_workflow = get_workflow("relaxation.vasp.matproj")

class Relaxation__Vasp__HSKQuality00(original_workflow):
    version = "2022.1.5"
    potcar_mappings = PBE_POTCAR_MAPPINGS

    # Make the unitcell relatively cubic before relaxing
    standardize_structure = "primitive-LLL"
    symmetry_tolerance = 0.1
    incar = original_workflow.incar.copy()  # Make sure you copy!
    incar.update(
        dict(
            # These settings are the same for all structures regardless of composition.
            PREC="Normal",
            EDIFF=1e-4,
            EDIFFG=1e-2,
            ENCUT=425,  # !!! Should this be based on the element type?
            ISIF=3,
            NSW=100,
            IBRION=2,  # for cases of bad starting sites
            #POTIM=0.02,
            LCHARG=False,
            LWAVE=False,
            KSPACING=0.5,
            ISMEAR=-5,
            LASPH=True,
            LORBIT=11,
            #LREAL=Auto,
            SIGMA=0.5,
            LMAXMIX=4
            )
        )

quality00_workflow=Relaxation__Vasp__HSKQuality00

# make sure we have new settings updated and that we didn't change the original
assert original_workflow.incar != Relaxation__Vasp__HSKQuality00

2. Rigorous Relaxation Calculation:

   from simmate.workflows.utilities import get_workflow
   from simmate.apps.vasp.inputs import PBE_POTCAR_MAPPINGS
   
   original_workflow = get_workflow("relaxation.vasp.matproj")
   
   class Relaxation__Vasp__HSKQuality01(original_workflow):
       version = "2022.1.5"
       potcar_mappings = PBE_POTCAR_MAPPINGS
   
       incar = original_workflow.incar.copy()
       incar.update(
           dict(
               PREC="Normal",
               EDIFF=1e-5,
               ENCUT=450,
               ISIF=3,
               NSW=100,
               IBRION=1,  # For handling poor starting sites
               POTIM=0.5,
               LCHARG=False,
               LWAVE=False,
               KSPACING=0.4,
               ISMEAR=-5,
               LASPH=True,
               LORBIT=11,
               SIGMA=0.5,
               LMAXMIX=4
           )
       )
   
   quality01_workflow = Relaxation__Vasp__HSKQuality01
   
   assert original_workflow.incar != Relaxation__Vasp__HSKQuality01

3. Final Static Calculation:

   from simmate.workflows.utilities import get_workflow
   from simmate.apps.vasp.inputs import PBE_POTCAR_MAPPINGS
   
   original_workflow = get_workflow("static-energy.vasp.matproj")
   
   class Static__Vasp__HSKQuality(original_workflow):
       version = "2022.1.5"
       potcar_mappings = PBE_POTCAR_MAPPINGS
       incar = original_workflow.incar.copy() 
       incar.update(
           dict(
               ALGO="Normal",
               IBRION=-1,  # Locks everything between ionic steps
               NSW=0,  # Main static energy setting
               ISMEAR=-5,  
               SIGMA=0.05,
               LASPH=True,
               LORBIT=11,
               LMAXMIX=4
           )
       )
   
   static_workflow = Static__Vasp__HSKQuality
   
   assert original_workflow.incar != Static__Vasp__HSKQuality

Integrating Subworkflows:

I modified the staged.py script (from staged.py) and updated the subworkflow_names as follows:

subworkflow_names = [
    "/media/synology/user/hyesol/workflow_modified/nested/HSKquality00.py:quality00_workflow",
    "/media/synology/user/hyesol/workflow_modified/nested/HSKquality01.py:quality01_workflow",
    "/media/synology/user/hyesol/workflow_modified/nested/HSKstatic.py:static_workflow"
]

I’ve noticed that when I run the workflows in a local environment without a database connection, everything works smoothly, except that I encounter the following error on the last static calculation step:
AttributeError: 'dict' object has no attribute 'to_toolkit'
and it fails to generate the simmate_summary.yaml file for the static calculation and the main workflow.

On the other hand, when I run the same code in an environment which is connected to DigitalOcean6, I receive the following error code free(): invalid pointer /var/spool/slurmd/job86062/slurm_script: line 8: 371436 Aborted (core dumped) simmate workflows run settings.yaml
I suspect this is a some kind of memory issue with DigitalOcean6, which began after I upgraded simmate in that environment (now I am using hyesol_CoO-settings.yaml)

The reason I’m building this nested DFT workflow is to address an issue where, in the evolutionary algorithm, after a structure is found and undergoes the relaxation.vasp.staged workflow, the energy isn’t converging. I realized that error handlers are not implemented in the relaxation.vasp.staged workflow. With this modified nested DFT calculation, I plan to integrate it into the evolutionary algorithm to handle these convergence issues.

I was also considering modifying the relaxation.vasp.staged code by adding:

original_workflow = get_workflow("relaxation.vasp.matproj")

Sorry for this long long paragraphs... I really appreciate any input!

[jacksund]

Looks like several things are going on.

First, the process of incar.copy() and incar.update() and assert ... != ... was ugly, and after some brainstorming with @SWeav02, I updated it. As of 0.17.0, the new docs for how to update incar settings are here

Here's an example:

from simmate.workflows.utilities import get_workflow

original_workflow = get_workflow("static-energy.vasp.matproj")

class StaticEnergy__Vasp__MyCustomPreset(original_workflow):
    _incar_updates = dict(
        NPAR=1,
        ENCUT=-1,
    )

I modified the staged.py script

For the modified staded.py script, your update looks fine to me. I'd suggest defining your workflow like this:

from simmate.workflows.utilities import get_workflow

original_workflow = get_workflow("relaxation.vasp.staged")

class Relaxation__Vasp__MyCustomStaged(original_workflow):
    subworkflow_names = [
        "/media/synology/user/hyesol/workflow_modified/nested/HSKquality00.py:quality00_workflow",
        "/media/synology/user/hyesol/workflow_modified/nested/HSKquality01.py:quality01_workflow",
        "/media/synology/user/hyesol/workflow_modified/nested/HSKstatic.py:static_workflow",
    ]

It is an open issue for me to simplify building a staged workflow though, so this might be easier in the future: #558

If the error persists, I would have to see the full staged script that you're using and the error traceback to troubleshoot.

I’ve noticed that when I run the workflows in a local environment without a database connection, everything works smoothly, except that I encounter the following error on the last static calculation step:
AttributeError: 'dict' object has no attribute 'to_toolkit'
and it fails to generate the simmate_summary.yaml file for the static calculation and the main workflow.

On the other hand, when I run the same code in an environment which is connected to DigitalOcean6, I receive the following error code free(): invalid pointer /var/spool/slurmd/job86062/slurm_script: line 8: 371436 Aborted (core dumped) simmate workflows run settings.yaml
I suspect this is a some kind of memory issue with DigitalOcean6, which began after I upgraded simmate in that environment (now I am using hyesol_CoO-settings.yaml)

Changing the database backend shouldn't have any effect, so I'm lost on what's happening here. I would address the two things I mention above and then work from there

The reason I’m building this nested DFT workflow is to address an issue where, in the evolutionary algorithm, after a structure is found and undergoes the relaxation.vasp.staged workflow, the energy isn’t converging. I realized that error handlers are not implemented in the relaxation.vasp.staged workflow. With this modified nested DFT calculation, I plan to integrate it into the evolutionary algorithm to handle these convergence issues.

Have you considered adding your workflows to the warren_lab app or modifying the existing qualityN workflows? Inheriting from the materials project workflows is not the best way to add error handlers. If you're only after error handlers, I can probably just add a couple of them to the default workflows. I only excluded them from these workflows because errors were rare in the systems I tested & they would only slow down my searches by wasting time on unrealistic structures

[hyesol]

Changing the database backend shouldn't have any effect, so I'm lost on what's happening here. I would address the two things I mention above and then work from there

When I run VASP through simmate in my hyesol_env_CoO environment (connected to DigitalOcean6, using simmate version 0.17.0), I encounter the error shown above. I suspect this issue is related to the connection with DigitalOcean6, possibly due to an insufficient connection pool size or what not... VASP runs fine on its own, so the problem seems to be specific to simmate workflows. This error only occurs in this environment, while other environments using the same simmate version work without any issues.

Have you considered adding your workflows to the warren_lab app or modifying the existing qualityN workflows? Inheriting from the materials project workflows is not the best way to add error handlers. If you're only after error handlers, I can probably just add a couple of them to the default workflows. I only excluded them from these workflows because errors were rare in the systems I tested & they would only slow down my searches by wasting time on unrealistic structures

I’m inheriting the Materials Project workflows because they come with error handlers and DFT+U options. I know I could just modify them using _incar_updates, but I figured it would be more efficient to inherit from the Materials Project since it offers both features. When using an evolutionary algorithm, isn't it crucial to start with realistic, converged structures so that further structures evolve from a solid foundation? I understand why the error handlers might be excluded, given that it can take around five hours for a 16-atom structure to converge, and the structures themselves may not change significantly. However, if I plan to use the energy values (e.g., for a hull diagram or building a prediction model between energy and structure), ensuring energy convergence becomes critical. I’m also thinking about adding my workflows to the warren_lab app, especially the evolutionary algorithm that includes a full DFT+U calculation.

[jacksund]

DigitalOcean6

I actually don't know what DigitalOcean6 is! Not sure if this is something new, but even if you db is on digitalocean, you shouldn't need to install anything extra except psycopg2 to connect.

This error only occurs in this environment, while other environments using the same simmate version work without any issues.

This makes me think you just need to rebuild your env from scratch -- try to stick 100% to conda-forge. Also, since the error looks like a C-based error rather than a python one (and I only really know python), I asked chatgpt on whim about it:

The "free() invalid pointer" error in Python Django is usually related to low-level memory management issues. It often happens when...

• C Extensions: You are using Python libraries (like NumPy, psycopg2, or any C-based libraries) that involve C extensions. These libraries directly manage memory, and a bug in them can cause this error.
• Multiple Python Versions: If you have multiple Python versions installed or conflicting packages (e.g., mismatched versions of Django and a database driver), they might cause memory corruption.
• Database Driver Issues: This error can be caused by database drivers (e.g., psycopg2 for PostgreSQL). Ensure you're using compatible versions of drivers with your Django and database.
• Misconfigured Virtual Environment: If your virtual environment is corrupted or incorrectly configured, it can lead to such errors.

Try upgrading your packages, especially database drivers, and ensure compatibility between Python, Django, and any C-based libraries you’re using.

When using an evolutionary algorithm, isn't it crucial to start with realistic, converged structures so that further structures evolve from a solid foundation?

Yes, but keep in mind the level of "realistic" and "converged" varies depending on the type of calc. For evolutionary algs, you can still have a successful run with "low quality" calculations (relative to Material Project settings). For example, on a static comp search, you want to identify the best 10 or structures quickly and efficiently. Even if those 10 are in the wrong order at the end, just having the 10 is what's important. Once your search is done, you then take those top structures and do "high quality" calcs on them to really nail down their ranking & relative thermodynamics. So don't take the final hull diagram as-is. You almost always want to do higher quality calcs on the final best structures that are close to / on the hull. (Automating this was on my to-do list... just never got to it)

five hours for a 16-atom structure to converge

Depending on the stoich, this might be way too much! I want to say my benchmarks for 16 atom systems took <1hr per structure for the full staged relaxation

I’m also thinking about adding my workflows to the warren_lab app, especially the evolutionary algorithm that includes a full DFT+U calculation.

Go ahead! Unless Scott says otherwise (maybe he wants to keep down clutter / number of flows), there's no problem with throwing more presets in the app

[hyesol]

Oh yea sorry, idk where digitalocean6 came from, it should be just digitalocean and thank you for all the advices!