Particle Coordinate is NaN occasionally upon molecule hitting PBCs

[alexberlaga]

I am running large numbers of simulations of three peptide-like molecules in water (starting in a triple-helical conformation). In my simulations, I increase the temperature every by 1 K/nanosecond. In a small number of these simulation (yet still significant because I am running so many of them), I get a "Particle Coordinate is NaN" error. This happens exclusively when one or more of the molecules goes through the periodic boundary, but far from every time this happens. Frustratingly, if I try to rerun the simulation from a checkpoint file, I cannot reproduce the same error -- it seems to happen randomly. When I visualize the trajectory before the error, there are no abnormalities and no statistical differences that I can discern between it and another trajectory that runs its full length without the NaN issue. I do not think my box size is too small -- increasing it does not help. I attached my code to this comment -- the issue is either in run_annealing.py or sim_annealing_lib.py. Do you know what could be causing such an issue? Thanks!
Peptoid Code.zip

[alexberlaga]

After trying many things, increasing the friction coefficient to 1/ps seems to make this PBC issue go away. I am not yet fully convinced that it's not a coincidence, but the more simulations I run with this friction coefficient, the more I'm confident that it's the case.

[peastman]

Hopefully that's all it takes. For what it's worth, 1/ps is the default value I usually recommend. Your previous value of 0.01/ps is tiny. It takes hundreds of picoseconds to exchange energy between the system and the heat bath. That doesn't make it less accurate, but it does mean any inaccuracy in the simulation (like from the finite time step) has a lot more opportunity to cause problems.

[alexberlaga]

Could such an error occur because of a very low constraint tolerance (maybe the integrator is doing something crazy to enforce a desired temperature)? I noticed that mine is 1e-8, while others in these posts have used tolerances of 1e-5 or 1e-6.

[peastman]

Your code uses mixed precision, right? If so, it shouldn't be a problem. 1e-8 is smaller than it's possible to achieve in single precision, which in that case could lead to problems. But in mixed precision the only issue would be slower performance since it needs more iterations to converge the constraints.

[alexberlaga]

It seems like this could be a CVPack issue, rather than an openmm issue... when I added a new "force" through CVPack that simply keeps track of certain distances [ cvpack.Distance( ... , pbc=True) ], these errors started popping up again decently often.