Evolutionary Search Questions

[SWeav02]

@jacksund @scott-materials

Yesterday I was finally able to get enough space on WarWulf to get some benchmarks going. I've been running into a couple of issues, most of which I think I can solve, but one error seems to be crashing a number of workers. I haven't seen the error before, but though maybe one of you might recognize it. My current best guess is that we're making too many connections to the database, but I'm not sure. If you have any ideas, let me know.

Also, there are several issues that I'll need to open a mini PR for to make my life easier with checking issues, so keep an eye out for that.

evo_crash.txt

[jacksund]

psycopg2.OperationalError: SSL error: ssl/tls alert bad record mac

Where is your database hosted? You might need to use sslmode: require or remove it if it's already there. See the example here: https://jacksund.github.io/simmate/getting_started/use_a_cloud_database/quick_start/

My current best guess is that we're making too many connections to the database

Are you using a connection pool? If so, you should be able to support >1k connections. https://jacksund.github.io/simmate/getting_started/use_a_cloud_database/build_a_postgres_database/#vi-creating-a-connection-pool

[SWeav02]

Thanks for the quick response! We can mess around with the sslmode: require setting.

Are you using a connection pool? If so, you should be able to support >1k connections.

We are using a connection pool, but I'm not sure what the pool size is set up to be. @scott-materials do you know?

[SWeav02]

Unrelated, but I'm also running into an issue where many workflows (thousands) are failing. A handful of them seem to be from failed DFT due to poor structures, but a vast majority of them give this error:

Exception: Unknown format provided for structure input. <class 'tuple'> was provided.

I tracked this error down and it occurs when the RandomSymStructure creator fails to create a new structure. The new_individual workflow tries to continue anyway and then returns this error. Looking at the worker output, it seems like the main culprit is that many of the spacegroups aren't valid for the first benchmarking composition (Dy1 Cu1). After failing enough times, the random generation must return something other than a structure.

I see that in the RandomSymStructure class there's an option to remove failed spacegroups, but it's by default set to False. @jacksund do you think it would be reasonable for me to set this to True to avoid repeat errors? Or is there another reason it's set to False?

[SWeav02]

I looked at this a bit more, and another issue is that later into the search the creator is more likely to make repeat structures and fail validation. So, in addition to the issue above, I think I might be running this search for too long. I'm using the following settings which I just took from the benchmarks module here.

nfirst_generation = 3,
nsteadystate = 200,
best_survival_cutoff = 10000,
max_structures = 50000,
min_structures_exact = 5000,

@jacksund do you remember if you used these setting for all of your benchmarks? Or did you scale the length with the complexity of the composition? WarWulf has been consistently pretty full so it's difficult for me to test things like this.

[jacksund]

I used the same settings for all of them. Those excessively large values for best_survival_cutoff and max_structures were effectively to make sure the calculation ran until I manually stopped it (after checking that the expected structure had been found w. that benchmark script you linked). nsteadystate was 200 because I want to make sure there were plenty of new jobs in the queue so that workers were never waiting. And a small nfirst_generation is because I was aiming for the best real-time benchmarks, and wanted transformation steadystates to begin as soon as possible

As for the RandomSymStructure, the default is False because there is still a chance that a totally valid spacegroup (maybe even the ground state one!) ends up failing. If a group like that fails and gets removed, you could really hurt your search. I didn't have the time to test & ensure this wasn't happening, so for peace of mind in my benchmarks, I disabled it

Are you able to reproduce the Dy1 Cu1 case using just the RandomSymStructure (outside of a symmetry search)? If so, we can open an issue and tackle fixing it

[SWeav02]

I used the same settings for all of them. Those excessively large values for best_survival_cutoff and max_structures were effectively to make sure the calculation ran until I manually stopped it (after checking that the expected structure had been found w. that benchmark script you linked). nsteadystate was 200 because I want to make sure there were plenty of new jobs in the queue so that workers were never waiting. And a small nfirst_generation is because I was aiming for the best real-time benchmarks, and wanted transformation steadystates to begin as soon as possible.

Awesome, this makes sense, thanks! I'll see if I can devise a way to automatically stop the searches when they find the expected structure since it'll be difficult for me to manually run all 180 searches.

As for the RandomSymStructure, the default is False because there is still a chance that a totally valid spacegroup (maybe even the ground state one!) ends up failing. If a group like that fails and gets removed, you could really hurt your search. I didn't have the time to test & ensure this wasn't happening, so for peace of mind in my benchmarks, I disabled it

Are you able to reproduce the Dy1 Cu1 case using just the RandomSymStructure (outside of a symmetry search)? If so, we can open an issue and tackle fixing it

I wrote a quick script to test this:

from simmate.toolkit.creators.structure.random_symmetry import RandomSymStructure
from simmate.toolkit import Composition

rand_gen = RandomSymStructure(Composition("DyCu"))


failed_count = 0
success_count = 0
for i in range(1000):
    structure = rand_gen.create_structure()
    if structure is False:
        failed_count += 1
    else:
        success_count += 1

print(f"Failure rate: {failed_count/10}%")

Generation fails ~70% of the time. I tested this on NaCl and Fe as well with similar results. Increasing the number of atoms to Na2Cl2 decreases the failure rate to ~30%. I think as we get far into a small composition search, the possible structures are more likely to have already been generated and new submissions don't pass validation.

I think this is the correct behavior. There just aren't that many ways to build a structure out of 1 or 2 atoms. My search was probably going way too long and ran out of possible structures. I think the thing to fix here is the error/warning that's given to users. We should probably manually check for this situation and return an error suggesting that the search may be near completion.

I'm sure there is also a way to prevent repeat failed generation if a spacegroup truly can't be used, but the structure generation for these small compositions is fast enough that this issue isn't high on my to-do list.

[jacksund]

I'll see if I can devise a way to automatically stop the searches

I started outlining the module way back. But go with a new stop condition class:
https://github.com/jacksund/simmate/blob/main/src/simmate/apps/evolution/stop_conditions/stop_conditions.py

I wrote a quick script to test this

under the hood, RandomSymStructure uses RSLSmartVolume which in turn uses Compositon.volume_estimate. I bet if you mess with the predicted volume via the packing factor, this error will largely go away. I can't remember the specifics, but I know I messed with the packing factor to improve searches in the 10-20 atom range, but might be causing issues for the 1-5 range like you're seeing. Setting a packing factor that is based on number of atoms (within RandomSymStructure) is what I expect fixes this

Also maybe double check my logic... I might have packing factor backwards? I expected higher packing = more dense, but my old comments on this method suggest the opposite. Probably need to switch this to total_volume /=packing_factor and switch the default from 1.35 to ~0.75:

simmate/src/simmate/toolkit/base_data_types/composition.py

Line 142 in 91af10b

total_volume *= packing_factor

I think as we get far into a small composition search, the possible structures are more likely to have already been generated and new submissions don't pass validation.

Agreed. 1-3 atom structures should largely be pulled from sources like MatProj. Even in our variable composition searches, we just skip 1 atom systems.

the structure generation for these small compositions is fast enough

yep this is where I was at as well. It's a negligible amount of time for small systems

[jacksund]

our default (0.74)

That's a typo in the docstring. It was the original default but it is now 1.35 because it led to much better searches:

simmate/src/simmate/toolkit/base_data_types/composition.py

Line 106 in 91af10b

packing_factor: float = 1.35,

Meanwhile the distance cutoff matrix uses 0.5 to be more forgiving with atom placements:

simmate/src/simmate/toolkit/base_data_types/composition.py

Line 149 in 91af10b

packing_factor: float = 0.5,

the latter probably has a larger effect

They are closely tied to one another. If you decrease the volume, you need a more forgiving distance matrix to get the same success rate. I wouldn't say one has more of an effect over the other but could be wrong

Again, I'd like to leave the defaults untouched for >5atoms and just fix your specific issue with DyCu. It's going to take more than just 'successful creation rate' rate for me to change the defaults because I ran benchmarks and found the current settings lead to faster calculations & lower energy + more diverse structures on average:
https://github.com/jacksund/simmate/tree/91af10b06c5bc37f045aa4d3b15d5453377affa0/benchmarks/creators

[SWeav02]

That's a typo in the docstring. It was the original default but it is now 1.35 because it led to much better searches:

Sorry, to clarify I actually meant the default of 1.35 is now 0.74 if we switch from

total_volume *= packing_factor

to

total_volume /= packing_factor

But yeah we don't need to change it if it led to better search results. Neither packing_factor variable had any noticeable effect on the success rate of structure creation in my testing anyways.

Honestly, I don't think I'm going to sink any more time into this for now since the problem it solves only affects the smaller compositions. They generate fast enough that it doesn't save much time. Plus, I'm not sure what else I would investigate since the distance cutoffs in distance_matrix_estimate are based off of the packing_factor which I just tested. The bigger problem for me is stopping searches once they've found the expected structure, so I'll switch to working on the StopCondition class instead.

[SWeav02]

so I'll switch to working on the StopCondition class instead.

BTW, on the topic of the StopCondition class, do you think we should move the code in the check_stop_condition method to the BasicStopConditon class? And/or add a section to check_stop_condition that loops over any user provided StopConditions? We could also add the StopCondition loop to the workflow itself, but that doesn't seem as in line with the code that's currently there.

[jacksund]

yep, go ahead and move that method over to the BasicStopCondition class. You can delete/edit as much as you want in the stop condition module and design it how you see fit. It's pretty clear I outlined that module and then never followed through haha. I think I stepped away from it in order to keep the yaml submission files pretty -- specifically to avoid stop_condition_name + stop_condition_kwargs keys since editing these would be very common even for beginners, unlike the selector + validator:

simmate/src/simmate/apps/evolution/models/fixed_composition.py

Lines 51 to 54 in 91af10b

	selector_name = table_column.CharField(max_length=200, null=True, blank=True)
	selector_kwargs = table_column.JSONField(default=dict, null=True, blank=True)
	validator_name = table_column.CharField(max_length=200, null=True, blank=True)
	validator_kwargs = table_column.JSONField(default=dict, null=True, blank=True)

Looping vs not-looping is also up to you. Eventually we want a loop though. Just know that if you loop of list of conditions, I would like the input to be dictionary of stop_conditions={stop_condition1: kwargs1, stop_condition2: kwargs2, ....} and avoid stop_condition_names=[....] + stop_condition_kwargs=[{...}, {...}, ...]`. Need to keep those yaml files user friendly

[SWeav02]

Looping vs not-looping is also up to you. Eventually we want a loop though. Just know that if you loop of list of conditions, I would like the input to be dictionary of stop_conditions={stop_condition1: kwargs1, stop_condition2: kwargs2, ....} and avoid stop_condition_names=[....] + stop_condition_kwargs=[{...}, {...}, ...]`. Need to keep those yaml files user friendly

I just saw this, but luckily I came to the same conclusion and already started designing it this way 😌