New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Implement 'dump_modify maxfiles' feature #907

Merged
merged 6 commits into from May 12, 2018

Conversation

Projects
None yet
3 participants
@akohlmey
Member

akohlmey commented May 11, 2018

Purpose

This implements a feature where a trajectory is recorded frequently but with limited history. This is implemented by storing filenames of created files with a 'one-file-per-step' setting active and one file per step being written. Once the maximum number of files is reached, the oldest file in the list is deleted.
This supersedes and and closes #604
Unlike the feature in #604, this variant requires only minimal changes to the code and is compatible with all dumps styles that support the multifile feature and also compatible with writing one file per processor or one file per group of processors.

Author(s)

Axel Kohlmeyer (Temple U)

Backward Compatibility

new feature and off by default.

Post Submission Checklist

Please check the fields below as they are completed

  • The feature or features in this pull request is complete
  • Suitable new documentation files and/or updates to the existing docs are included
  • One or more example input decks are included
  • The source code follows the LAMMPS formatting guidelines

@akohlmey akohlmey assigned sjplimp and unassigned akohlmey May 11, 2018

@jrgissing

This comment has been minimized.

Collaborator

jrgissing commented May 11, 2018

I agree this is more general than #604

however, it has been quite handy to download just one xyz file with the latest timesteps

notably, in this version, during debugging, files would quickly build up as runs exited at different times

this could be mitigated by appending filenames with the integers 1-maxfiles instead of timestep (which is less important in this case, and usually included in dump file)

(implementation note: this would also involve a file 'rename' loop once maxfile is reached)

@akohlmey

This comment has been minimized.

Member

akohlmey commented May 11, 2018

converting multiple xyz or atom style or custom dump files into a single file is trivial to do with 'cat' in combination with dump_modify pad, you can have zero padded step numbers, so that those are automatically in the correct order.

as for accumulating "debug" files, it is equally trivial to put something like shell rm dump-debug-*.xyz at the beginning of a script, so that those debug outputs are purged when a new run is started.

there really isn't much of a need to put added complexity inside LAMMPS. one just has to make good use of existing features and combine them accordingly. i was actually surprised by the simplicity of adding the changes in this PR and how well it fits into the overall design of dump file management in LAMMPS.

@jrgissing

This comment has been minimized.

Collaborator

jrgissing commented May 11, 2018

i like those solutions.

however, I believe the primary use case for something like this is rapid debugging. in my case, i used this at high frequency while fine-tuning parameters. basically, i might still be tempted to have one xyz to avoid file clutter and the extra 'cat' step

at the least, i'd guess your above solutions might have to be mentioned in the docs for users to effectively use this for debugging

@akohlmey

This comment has been minimized.

Member

akohlmey commented May 11, 2018

sorry but you are asking us now to put explanations of basic file handling workflows into the documentation of an application. next you ask us to explain how to debug with gdb or valgrind or how to back up and synchronize projects between clusters and local archives. where would this stop?

i understand your fondness for your own solution and frustrations about not being aware of alternatives to the solution you came up with (and probably spent a significant amount of effort on implementing it).
But it is in the LAMMPS' spirit of things to stick with simple solutions whenever possible, and this is a good example of it.

@sjplimp

This comment has been minimized.

Contributor

sjplimp commented May 11, 2018

@jrgissing If you want to contribute a couple of paragraphs of doc text to add to
the dump_modify command doc page to illustrate how to use this along with commands
like cat, etc in scenarios you are envisioning, that would be great ...

@akohlmey

This comment has been minimized.

Member

akohlmey commented May 11, 2018

@sjplimp ok with me. i am piggybacking a windows portability bugfix on this one. my change to the AtC library broke on windows due to the idiosyncratic way how it supports common POSIX features.

@jrgissing

This comment has been minimized.

Collaborator

jrgissing commented May 11, 2018

@sjplimp i actually would agree that such an explanation wouldn't belong there; hopefully my concerns are groundless

@sjplimp sjplimp merged commit ca6920b into lammps:master May 12, 2018

5 checks passed

lammps/pull-requests/build-docs-pr head run ended
Details
lammps/pull-requests/kokkos_omp head run ended
Details
lammps/pull-requests/openmpi-pr head run ended
Details
lammps/pull-requests/serial-pr head run ended
Details
lammps/pull-requests/shlib-pr head run ended
Details

@akohlmey akohlmey deleted the akohlmey:dump_maxfile branch May 15, 2018

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment