Track

[ovhpa]

Dear Prof. Rohl,

I have finally completed GDIS's tracking feature.
This allow to launch USPEX or VASP calculations and track the progress on real time.
I have also modified the task system so that several calculations can be launched at the same time.
The tracking is a little different for VASP and USPEX though.

For ex. this is two USPEX calculations done simultaneously.
The main graph, ALL, shows all structure at one depending on the total energy per atom.
In red is the structure that belong to the BESTIndividuals file, ie. that are considered as best by USPEX.
In green are the structures generation from the generation currently under calculation.

The second graph is the BEST structure (also from the BESTIndividuals file)

I keep the BEST graph because it is sometimes tricky to select a structure from the main graphic (which can be very dense).
Then is the convex hull graph, which have several new features:

As for the main graph, green squares are the structures generation from the generation currently under calculation.
But this time, the structures are plotted against a "formation energy".
Given compound AxBy, the energy is given as:

E_f = E[A_xB_y] - ( x/n E[A_n] + y/m E[B_m] )

Where An and Bm are the minimum energy structure containing only A and only B, respectively, from the USPEX calculation results.
It is also possible to use an external references, by using a chem.in file.
Its syntax is straightforward, for example, it can contain a line with:

1 2 -16.625834

in which case the reference for the species 1 (the first species in USPEX AtomType order) have 2 atoms and an total energy of -16.625834eV.
If the species 1 is A, this means the reference is [A₂].
Each species is defined on a single line, and any missing species reference will be taken from USPEX calculation.
Additionally, in case no external reference is provided, the graph is first calculated using total energy per atom, until at least one of each single species have been calculated.

As previously, each structure can be select in the graph:

And the corresponding structure can be use directly.
For example, I decided to use the structure for a VASP calculation:

Note that the VASP tracking is different.
Each SCF energy is presented in a single graph, in which the diamond symbols correspond to the ionic steps, each link to the corresponding structure.

Each task is register to GDIS task manager, but I modified the code slightly so that each process has it own PID and can be killed independently:

This was not possible before due to the synchronous spawning of task. This is a reason why I do them asynchronously (and use the waitpid system function).
If this is stable enough, I think it would be nice to change all the calculation related tasks to use this strategy.

Anyway, I think the code is quite stable now, almost ready for the next USPEX workshop - I will do a round of bug hunting before giving the green light ;)

For the future GDIS version, I am very sorry that I have encounter some delays m(_ _)m
I will explain the reasons in a separate mail, but it is still ongoing!

Sincerely,

[ovhpa]

I forgot to mention:
About VASP tracking, when the vasprun.xml file is overwritten by a new calculation, the tracking is not stopped, which allow to track several VASP calculations that are performed in the same directory.
For example in a USPEX calculation using VASP optimization, I have been able to follow all the VASP optimizations from a single GDIS tracking window over more than a week.

[arohl]

Looks great! I look forward to you committing the changes. Please do change everything to asynchronous if you have the time. Andrew Professor Andrew Rohl DPhil FACS Director | ARC Training Centre for Transforming Maintenance through Data Science Director | Curtin Institute for Computation Chemistry | School of Molecular and Life Sciences Curtin University Tel | +61 8 9266 3124 Fax | +61 8 9266 2300 Mobile | 0401 103 620 Email | andrew.rohl@curtin.edu.au<mailto:andrew.rohl@curtin.edu.au> Web | computation.curtin.edu.au<https://computation.curtin.edu.au> CRICOS Provider Code 00301J On 31 May 2019, at 9:28 pm, Okadome Valencia <notifications@github.com<mailto:notifications@github.com>> wrote: Dear Prof. Rohl, I have finally completed GDIS's tracking feature. This allow to launch USPEX or VASP calculations and track the progress on real time. I have also modified the task system so that several calculations can be launched at the same time. The tracking is a little different for VASP and USPEX though. For ex. this is two USPEX calculations done simultaneously. The main graph, ALL, shows all structure at one depending on the total energy per atom. In red is the structure that belong to the BESTIndividuals file, ie. that are considered as best by USPEX. In green are the structures generation from the generation currently under calculation. [track010]<https://user-images.githubusercontent.com/36496189/58705778-23a3c780-83eb-11e9-8d67-d3206a829f4e.png> The second graph is the BEST structure (also from the BESTIndividuals file) [track011]<https://user-images.githubusercontent.com/36496189/58706229-5ef2c600-83ec-11e9-81ea-4fdd64a24d39.png> I keep the BEST graph because it is sometimes tricky to select a structure from the main graphic (which can be very dense). Then is the convex hull graph, which have several new features: [track012]<https://user-images.githubusercontent.com/36496189/58706318-b002ba00-83ec-11e9-81d5-232c375e547d.png> As for the main graph, green squares are the structures generation from the generation currently under calculation. But this time, the structures are plotted against a "formation energy". Given compound AxBy, the energy is given as: Ef = E[AxBy] - ( x/n E[An] + y/m E[Bm] ) Where An and Bm are the minimum energy structure containing only A and only B, respectively, from the USPEX calculation results. It is also possible to use an external references, by using a chem.in file. Its syntax is straightforward, for example, it can contain a line with: 1 2 -16.625834 in which case the reference for the species 1 (the first species in USPEX AtomType order) have 2 atoms and an total energy of -16.625834eV. If the species 1 is A, this means the reference is [A2]. Each species is defined on a single line, and any missing species reference will be taken from USPEX calculation. Additionally, in case no external reference is provided, the graph is first calculated using total energy per atom, until at least one of each single species have been calculated. As previously, each structure can be select in the graph: [track110]<https://user-images.githubusercontent.com/36496189/58707169-125cba00-83ef-11e9-837d-deb98d19d722.png> And the corresponding structure can be use directly. For example, I decided to use the structure for a VASP calculation: [track114]<https://user-images.githubusercontent.com/36496189/58707251-4b952a00-83ef-11e9-855b-1b4adbc9cc0a.png> Note that the VASP tracking is different. Each SCF energy is presented in a single graph, in which the diamond symbols correspond to the ionic steps, each link to the corresponding structure. Each task is register to GDIS task manager, but I modified the code slightly so that each process has it own PID and can be killed independently: [track112]<https://user-images.githubusercontent.com/36496189/58707620-6ae08700-83f0-11e9-8e8a-25f12efc165c.png> This was not possible before due to the synchronous spawning of task. This is a reason why I do them asynchronously (and use the waitpid system function). If this is stable enough, I think it would be nice to change all the calculation related tasks to use this strategy. Anyway, I think the code is quite stable now, almost ready for the next USPEX workshop - I will do a round of bug hunting before giving the green light ;) For the future GDIS version, I am very sorry that I have encounter some delays m(_ _)m I will explain the reasons in a separate mail, but it is still ongoing! Sincerely,

…

________________________________ You can view, comment on, or merge this pull request online at: #23 Commit Summary * file_uspex: FIX mixed_symbol was not properly set to default (FALSE) at the begining of each set. fix #2 * gl_graph: remove gl_print_window, replaced by newest pango_print. * graphs: add ability to change indivudual symbol colors. fix #1 * sgio.c: minor fix on some compiler warning. * graphs: FIX plot system for gui_plot. fix #6 Generalize new graph method. fix #7 * file_vasp.c: add forces and stress reading. fix #8 * file_vasp: first tracking implementation: model frames & values. * file_vasp: fix tracking so it can be used from begining of calculation. fix #10 * file_vasp: first implementation of continuous vasprun.xml tracking (follow the calculationS related to that file). also fix #10 ... again. * file_vasp: very classic omission of 'git add .' m(_ _)m * file_vasp: add a SCF graph visual to reading vasprun.xml, including during tracking. * file_vasp: FIX continuous tracking of vasp calculations + FIX SCF graph update. * file_vasp: continuous tracking of vasp calculations (rare BUG fix). * file_vasp: sanitize memory in tracking + fix a _BUG_ in animation of frames for subsequent (not first) vasprun.xml files of a continous tracking. * file_uspex: first tracking implementation of USPEX job. * file_uspex: tracking of variable composition jobs (USPEX). * file_uspex: small fix. * file_uspex: small fix II. * file_uspex: many fixes. * file_uspex: precompiler mistake. m(_ _)m * file_uspex: implicit reference for convex hull calculation + load VASP4 format + changed USPEX launch. * file_uspex: read chem.in file (if present) for explicit reference in convex hull calculation + cosmetic changes (100000.000 energy value in Individuals file). * file_uspex: applied previous changes to tracking. * file_uspex & file_vasp: small fix. * file_uspex: small improvement. * uspex: launch & track 1/3. * uspex: launch & track 2/3 (+ FIX). * uspex: launch & track 3/3 + simultaneous USPEX launches + new async task + cosmetic changes. * vasp: launch & track 1/2. * vasp: small fix. * uspex: small fix. File Changes * M src/file.c<https://github.com/arohl/gdis/pull/23/files#diff-0> (9) * M src/file.h<https://github.com/arohl/gdis/pull/23/files#diff-1> (12) * M src/file_uspex.c<https://github.com/arohl/gdis/pull/23/files#diff-2> (1571) * M src/file_uspex.h<https://github.com/arohl/gdis/pull/23/files#diff-3> (7) * M src/file_vasp.c<https://github.com/arohl/gdis/pull/23/files#diff-4> (993) * M src/file_vasp.h<https://github.com/arohl/gdis/pull/23/files#diff-5> (30) * M src/gdis.h<https://github.com/arohl/gdis/pull/23/files#diff-6> (3) * M src/gl_graph.c<https://github.com/arohl/gdis/pull/23/files#diff-7> (692) * M src/graph.h<https://github.com/arohl/gdis/pull/23/files#diff-8> (14) * M src/gui_image.h<https://github.com/arohl/gdis/pull/23/files#diff-9> (1) * M src/gui_main.c<https://github.com/arohl/gdis/pull/23/files#diff-10> (57) * M src/gui_plots.c<https://github.com/arohl/gdis/pull/23/files#diff-11> (736) * M src/gui_task.c<https://github.com/arohl/gdis/pull/23/files#diff-12> (4) * M src/gui_tree.c<https://github.com/arohl/gdis/pull/23/files#diff-13> (43) * M src/gui_uspex.c<https://github.com/arohl/gdis/pull/23/files#diff-14> (77) * M src/gui_vasp.c<https://github.com/arohl/gdis/pull/23/files#diff-15> (48) * M src/image.c<https://github.com/arohl/gdis/pull/23/files#diff-16> (8) * M src/interface.h<https://github.com/arohl/gdis/pull/23/files#diff-17> (2) * M src/main.c<https://github.com/arohl/gdis/pull/23/files#diff-18> (4) * M src/model.c<https://github.com/arohl/gdis/pull/23/files#diff-19> (13) * M src/pak.h<https://github.com/arohl/gdis/pull/23/files#diff-20> (7) * M src/plots.c<https://github.com/arohl/gdis/pull/23/files#diff-21> (777) * M src/plots.h<https://github.com/arohl/gdis/pull/23/files#diff-22> (6) * M src/sgio.c<https://github.com/arohl/gdis/pull/23/files#diff-23> (4) * A src/t1.xpm<https://github.com/arohl/gdis/pull/23/files#diff-24> (25) * A src/t2.xpm<https://github.com/arohl/gdis/pull/23/files#diff-25> (25) * A src/t3.xpm<https://github.com/arohl/gdis/pull/23/files#diff-26> (25) * M src/task.c<https://github.com/arohl/gdis/pull/23/files#diff-27> (61) * M src/task.h<https://github.com/arohl/gdis/pull/23/files#diff-28> (2) * A src/track.h<https://github.com/arohl/gdis/pull/23/files#diff-29> (32) * A src/track.xpm<https://github.com/arohl/gdis/pull/23/files#diff-30> (25) Patch Links: * https://github.com/arohl/gdis/pull/23.patch * https://github.com/arohl/gdis/pull/23.diff — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub<#23?email_source=notifications&email_token=ACTZWLASXIY4D7GNXZXVZM3PYERWPA5CNFSM4HR2XZB2YY3PNVWWK3TUL52HS4DFUVEXG43VMWVGG33NNVSW45C7NFSM4GW6U53Q>, or mute the thread<https://github.com/notifications/unsubscribe-auth/ACTZWLBBH67PQC4DEXSBB33PYERWPANCNFSM4HR2XZBQ>.

[ovhpa]

I made all task asynchronous.
Additionally, I change the display of task status a little.
Now the "status file" is updated live, seeking through it for update - if any.
I fear some slow down if the status file become very large, but on the other hand it is quite enjoyable to read the status of calculation together with its output:

I'm still looking for bugs, and I met with some strange behaviour with CENTOS distribution: the output of the ps command to see the CPU and memory usage of a task is unusable.
However, I think we don't have to fix that on GDIS side.

[ovhpa]

I think it is now quite stable:

I had to (temporary) remove the update of BEST graph though.
It was not so stable so I replace with a function that wipe graph data and reconstruct BEST graph for each generation. It's a little slower but much more stable.

I think it is a ready enough version for merge, please tell me if you encounter some problems afterwards.