Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

move atomicrex interface development to contrib #65

Merged
merged 59 commits into from Mar 1, 2021
Merged

Conversation

Leimeroth
Copy link
Member

As discussed I will continue the development of an atomicrex interface to pyiron in pyiron_contrib.
Essentially there are 2 points where I still need help before this works like other pyiron jobs imo.
The probably more difficult one is storing and loading the jobs in hdf5. I used InputLists whenever possible so I hope it isn't too hard.
The second point is how to find the executable in an intelligent way. Right now it fails because the automatic search path uses the module path and splits it at the wrong place for the atomicrex module if it stays in contrib/atomistic to find it in the resources. So I think either place it one level higher, redefining that function or changing the pattern in the resources directory is the solution and I am not sure which is the better way to go.

@Leimeroth Leimeroth added the help wanted Extra attention is needed label Feb 4, 2021
@coveralls
Copy link

coveralls commented Feb 4, 2021

Pull Request Test Coverage Report for Build 607362118

  • 0 of 0 changed or added relevant lines in 0 files are covered.
  • No unchanged relevant lines lost coverage.
  • Overall coverage remained the same at 47.909%

Totals Coverage Status
Change from base Build 606193312: 0.0%
Covered Lines: 378
Relevant Lines: 789

💛 - Coveralls


## Class defined for future addition of other codes
## Not sure which functionality (if any) can be extracted yet, but a similar pattern is followed in other pyiron modules
class PotentialFittingBase(GenericJob):
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Like it, probably once #51 is in, we want to include it here. Can you already move that class to something like pyiron_contrib/atomistic/job/potential_fitting.py?

self.output = Output()

"""
def to_hdf(self, hdf=None, group_name=None):
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should call super().to_hdf(hdf=hdf, group_name=group_name) and similarly pass those to the other to_hdf. Otherwise it should work.

return False

def to_xml_element(self):
xml = ET.Element(f"{self.prop}")
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

f-strings are python 3.6+ and we nominally support down to 3.4, so we either remove them or bump our python requirements. Am open to both.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am fine with dropping 3.4 and 3.5 in particular for pyiron_contrib.

class Poly(InputList):
def __init__(self, identifier, cutoff, species):
super().__init__(table_name=f"Poly_{identifier}")
self.identifier = identifier
Copy link
Contributor

@pmrv pmrv Feb 4, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These will end up as elements in the InputList. If you don't want this call object.__setattr__ like here to define them as instance variables. Accessing them afterwards works with the normal syntax though.

If you want them as elements this is fine though (and will also take care of saving them).

structure.from_hdf(hdf, group_name = g)
self.append(structure)

def to_ase_db(self, db):
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

See #51 again. It's ok unify those a bit later though.

Copy link
Contributor

@pmrv pmrv left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for contributing this!

@Leimeroth Leimeroth marked this pull request as draft February 5, 2021 10:06
@jan-janssen
Copy link
Member

https://github.com/pyiron/pyiron_base/blob/master/pyiron_base/job/generic.py#L1475 - this is the function that should be overwritten.

@jan-janssen
Copy link
Member

I created a small mybinder environment for testing https://github.com/jan-janssen/pyiron-atomicrex

@jan-janssen
Copy link
Member

So the executable part seems to be working now, it only fails to store data in the HDF5 file:

/srv/conda/envs/notebook/lib/python3.7/site-packages/pyiron_base/generic/inputlist.py in to_hdf(self, hdf, group_name)
    568 
    569         self._type_to_hdf(hdf)
--> 570         hdf["data"] = self.to_builtin()
    571         hdf["read_only"] = self.read_only
    572 

/srv/conda/envs/notebook/lib/python3.7/site-packages/pyiron_base/generic/hdfio.py in __setitem__(self, key, value)
    745                 title=posixpath.join(self.h5_path, key),
    746                 overwrite="update",
--> 747                 use_json=True,
    748             )
    749 

/srv/conda/envs/notebook/lib/python3.7/site-packages/h5io/_h5io.py in write_hdf5(fname, data, overwrite, compression, title, slash, use_json)
    113         _triage_write(title, data, fid, comp_kw, str(type(data)),
    114                       cleanup_data, slash=slash, title=title,
--> 115                       use_json=use_json)
    116 
    117     # Will not be empty if any extra data to be written

/srv/conda/envs/notebook/lib/python3.7/site-packages/h5io/_h5io.py in _triage_write(key, value, root, comp_kw, where, cleanup_data, slash, title, use_json)
    149             _triage_write(
    150                 'key_{0}'.format(key), sub_value, sub_root, comp_kw,
--> 151                 where + '["%s"]' % key, cleanup_data=cleanup_data, slash=slash)
    152     elif isinstance(value, (list, tuple)):
    153         title = 'list' if isinstance(value, list) else 'tuple'

/srv/conda/envs/notebook/lib/python3.7/site-packages/h5io/_h5io.py in _triage_write(key, value, root, comp_kw, where, cleanup_data, slash, title, use_json)
    149             _triage_write(
    150                 'key_{0}'.format(key), sub_value, sub_root, comp_kw,
--> 151                 where + '["%s"]' % key, cleanup_data=cleanup_data, slash=slash)
    152     elif isinstance(value, (list, tuple)):
    153         title = 'list' if isinstance(value, list) else 'tuple'

/srv/conda/envs/notebook/lib/python3.7/site-packages/h5io/_h5io.py in _triage_write(key, value, root, comp_kw, where, cleanup_data, slash, title, use_json)
    230 
    231         err_str = 'unsupported type %s (in %s)' % (type(value), where)
--> 232         raise TypeError(err_str)
    233 
    234 ##############################################################################

TypeError: unsupported type <class 'pyiron_atomistics.atomistics.structure.atoms.Atoms'> (in <class 'dict'>["p_id2674"]["_structure"])

Basically it does not like the structures to be stored in the inputlist. @pmrv do we plan to add this? or should we just store the structures just like we do it in the structure master?

@pmrv
Copy link
Contributor

pmrv commented Feb 5, 2021

Basically it does not like the structures to be stored in the inputlist. @pmrv do we plan to add this? or should we just store the structures just like we do it in the structure master?

We have plans for this, see here, but have nothing concrete right now to revive it. For now we either store it as in the StructureContainer or use #51, which I'll merge later today.

@jan-janssen
Copy link
Member

@Leimeroth Can you explain me the difference between ARStructureList and StructureList ?

@jan-janssen jan-janssen mentioned this pull request Feb 5, 2021
@Leimeroth
Copy link
Member Author

Leimeroth commented Feb 8, 2021

@Leimeroth Can you explain me the difference between ARStructureList and StructureList ?

I wanted to inherit ARStructureList from StructureList and reuse StructureList when trying to extend the code for Lammps. But since the plan is to move to StuctureContainer it is probably unnecessary to keep the StructureList.

@jan-janssen
Copy link
Member

@Leimeroth To my understanding both the executable issue and the HDF5 issue is fixed now. What I would be interested now is an example which uses atomicrex to fit a potential and then uses the resulting potential for a very simple LAMMPS calculation. If it helps you can create a fork of https://github.com/jan-janssen/pyiron-atomicrex and just upload the notebook there then we can easily test it on mybinder.

@Leimeroth
Copy link
Member Author

Leimeroth commented Feb 8, 2021

@Leimeroth To my understanding both the executable issue and the HDF5 issue is fixed now. What I would be interested now is an example which uses atomicrex to fit a potential and then uses the resulting potential for a very simple LAMMPS calculation. If it helps you can create a fork of https://github.com/jan-janssen/pyiron-atomicrex and just upload the notebook there then we can easily test it on mybinder.

Running the job works now, but it is not possible to load it using Project.load().
Error:

---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-30-e3dcce638ade> in <module>
----> 1 job = pr.load("CuEAM_ArLBFGS_1000Steps_test")

~/git_projects/pyiron_base/pyiron_base/project/generic.py in load(self, job_specifier, convert_to_object)
    769             s.logger.warning("Job '%s' does not exist and cannot be loaded", job_specifier)
    770             return None
--> 771         return self.load_from_jobpath(
    772             job_id=job_id, convert_to_object=convert_to_object
    773         )

~/git_projects/pyiron_atomistics/pyiron_atomistics/project.py in load_from_jobpath(self, job_id, db_entry, convert_to_object)
    207             GenericJob, JobCore: Either the full GenericJob object or just a reduced JobCore object
    208         """
--> 209         job = super(Project, self).load_from_jobpath(
    210             job_id=job_id, db_entry=db_entry, convert_to_object=convert_to_object
    211         )

~/git_projects/pyiron_base/pyiron_base/project/generic.py in load_from_jobpath(self, job_id, db_entry, convert_to_object)
    790         if job_id:
    791             job = jobpath(db=self.db, job_id=job_id, user=self.user)
--> 792             job = job.load_object(
    793                 convert_to_object=convert_to_object, project=job.project_hdf5.copy()
    794             )

~/git_projects/pyiron_base/pyiron_base/job/core.py in load_object(self, convert_to_object, project)
    547             with project.open("..") as job_dir:
    548                 job_dir._mode = "a"
--> 549                 return self.to_object(project=job_dir, job_name=self._name)
    550         return self
    551 

~/git_projects/pyiron_base/pyiron_base/job/core.py in to_object(self, object_type, **qwargs)
    482                 + "\" is empty, so it can not be loaded."
    483             )
--> 484         return self.project_hdf5.to_object(object_type, **qwargs)
    485 
    486     def get(self, name):

~/git_projects/pyiron_base/pyiron_base/generic/hdfio.py in to_object(self, class_name, **qwargs)
   1341             self["TYPE"] = str(class_object)
   1342 
-> 1343         obj = self.create_instance(class_object, **qwargs)
   1344         obj.from_hdf(
   1345                 hdf = self.open(".."),

~/git_projects/pyiron_base/pyiron_base/generic/hdfio.py in create_instance(self, cls, **kwargs)
   1310         init_args.update(kwargs)
   1311 
-> 1312         return cls(**init_args)
   1313 
   1314     def to_object(self, class_name=None, **qwargs):

TypeError: __init__() got an unexpected keyword argument 'project'

@jan-janssen
Copy link
Member

@pmrv I guess the current issue with the HDF5 storage is that the classes are derived from InputList, so when they are reloaded from the HDF5 they are just input lists and not the specific class. In certain cases like the potential class it is not even possible to initialise the class with an empty dictionary, as depending on the type of potential - Lennard Jones or EAM - the classes have different utility functions. So the question is: Do we already have utility functions in the Inputlist to restore the original class which was previously derived from the inputlist?

@Leimeroth
Copy link
Member Author

The hdf storage of the general input still doesn't work and I can't figure out why. input.atom_types and input.fit_algorithm both inherit from InputList but are not written to hdf correclty, while it works completely fine for potential and fit properties of the structures, which do the same as far as I can tell. Could you maybe have a look at it @pmrv or @jan-janssen.

@pmrv
Copy link
Contributor

pmrv commented Feb 24, 2021

I can't straight-away see what the problem is, but I'm not sure if AtomTypes and AtomicrexAlgorithm need to be sub-types of InputList, I would either have them as normal InputLists or as their own type. For AtomicrexAlgorithm the problem might be related to the discussion here at the bottom.

Will have a deeper look tomorrow.

@jan-janssen
Copy link
Member

@Leimeroth Can we merge it now? And can you upload the corresponding notebook to https://github.com/pyiron/pyiron_potentialfit/tree/master/day_2 ? then we can use it to build a test environment of the workshop

@Leimeroth Leimeroth marked this pull request as ready for review March 1, 2021 06:47
@Leimeroth Leimeroth merged commit 5815b1d into master Mar 1, 2021
@delete-merged-branch delete-merged-branch bot deleted the atomicrex branch March 1, 2021 06:48
@jan-janssen jan-janssen restored the atomicrex branch March 1, 2021 07:09
@jan-janssen
Copy link
Member

@Leimeroth Currently I use the atomicrex branch from pyiron_contrib for the continuous integration and building the docker container. So please continue making changes on this branch.

@Leimeroth
Copy link
Member Author

Leimeroth commented Mar 1, 2021

Something just broke the hdf storage in my notebook. I think it is the change from InputList to Datacontainer that cause this.
Now when I run it I get an error:

Traceback (most recent call last):

  File "/home/niklas/anaconda3/lib/python3.8/site-packages/IPython/core/interactiveshell.py", line 3427, in run_code
    exec(code_obj, self.user_global_ns, self.user_ns)

  File "<ipython-input-18-5bb85cd10cac>", line 6, in <module>
    job.run()

  File "/home/niklas/git_projects/pyiron_base/pyiron_base/generic/util.py", line 213, in decorated
    return function(*args, **kwargs)

  File "/home/niklas/git_projects/pyiron_base/pyiron_base/job/generic.py", line 669, in run
    self._run_if_new(debug=debug)

  File "/home/niklas/git_projects/pyiron_base/pyiron_base/job/generic.py", line 1329, in _run_if_new
    self.save()

  File "/home/niklas/git_projects/pyiron_base/pyiron_base/job/generic.py", line 1175, in save
    self.to_hdf()

  File "/home/niklas/git_projects/pyiron_contrib/pyiron_contrib/atomistic/atomicrex/atomicrex_job.py", line 53, in to_hdf
    self.potential.to_hdf(hdf=self.project_hdf5)

  File "/home/niklas/git_projects/pyiron_base/pyiron_base/generic/datacontainer.py", line 595, in to_hdf
    v.to_hdf(hdf=hdf, group_name=k)

  File "/home/niklas/git_projects/pyiron_base/pyiron_base/generic/datacontainer.py", line 595, in to_hdf
    v.to_hdf(hdf=hdf, group_name=k)

  File "/home/niklas/git_projects/pyiron_base/pyiron_base/generic/datacontainer.py", line 595, in to_hdf
    v.to_hdf(hdf=hdf, group_name=k)

  File "/home/niklas/git_projects/pyiron_base/pyiron_base/generic/datacontainer.py", line 595, in to_hdf
    v.to_hdf(hdf=hdf, group_name=k)

  File "/home/niklas/git_projects/pyiron_base/pyiron_base/generic/datacontainer.py", line 578, in to_hdf
    hdf = hdf.create_group(group_name)

  File "/home/niklas/git_projects/pyiron_base/pyiron_base/generic/hdfio.py", line 369, in create_group
    h_new = self[name].copy()

  File "/home/niklas/git_projects/pyiron_base/pyiron_base/generic/hdfio.py", line 822, in __getitem__
    if item in self.list_nodes():

  File "/home/niklas/git_projects/pyiron_base/pyiron_base/generic/hdfio.py", line 529, in list_nodes
    return self.list_all()["nodes"]

  File "/home/niklas/git_projects/pyiron_base/pyiron_base/generic/hdfio.py", line 513, in list_all
    iopy_nodes = self._filter_io_objects(groups)

  File "/home/niklas/git_projects/pyiron_base/pyiron_base/generic/hdfio.py", line 951, in _filter_io_objects
    [group for group in groups if self._get_h5io_type(group) in h5io_types]

  File "/home/niklas/git_projects/pyiron_base/pyiron_base/generic/hdfio.py", line 951, in <listcomp>
    [group for group in groups if self._get_h5io_type(group) in h5io_types]

  File "/home/niklas/git_projects/pyiron_base/pyiron_base/generic/hdfio.py", line 929, in _get_h5io_type
    return str(eval("".join([self._h5_group, name, "._v_title"])))

  File "<string>", line 1
    store.root.PotentialDF1.potential.electron_densities__index_1.rho_CuCu__index_0.parameters__index_5.node_0.5__index_0._v_title
                                                                                                                ^
SyntaxError: invalid decimal literal

@jan-janssen
Copy link
Member

@pmrv Can you take a look at this? @Leimeroth I guess for the moment it is fine to use pyiron_base=0.1.48

@pmrv
Copy link
Contributor

pmrv commented Mar 1, 2021

@pmrv Can you take a look at this? @Leimeroth I guess for the moment it is fine to use pyiron_base=0.1.48

Can have a look in the afternoon. It seems that keys with decimal places in them break the HDF5 interface.

@pmrv
Copy link
Contributor

pmrv commented Mar 2, 2021

Something just broke the hdf storage in my notebook. I think it is the change from InputList to Datacontainer that cause this.
Now when I run it I get an error:

Traceback (most recent call last):

  File "/home/niklas/anaconda3/lib/python3.8/site-packages/IPython/core/interactiveshell.py", line 3427, in run_code
    exec(code_obj, self.user_global_ns, self.user_ns)

  File "<ipython-input-18-5bb85cd10cac>", line 6, in <module>
    job.run()

  File "/home/niklas/git_projects/pyiron_base/pyiron_base/generic/util.py", line 213, in decorated
    return function(*args, **kwargs)

  File "/home/niklas/git_projects/pyiron_base/pyiron_base/job/generic.py", line 669, in run
    self._run_if_new(debug=debug)

  File "/home/niklas/git_projects/pyiron_base/pyiron_base/job/generic.py", line 1329, in _run_if_new
    self.save()

  File "/home/niklas/git_projects/pyiron_base/pyiron_base/job/generic.py", line 1175, in save
    self.to_hdf()

  File "/home/niklas/git_projects/pyiron_contrib/pyiron_contrib/atomistic/atomicrex/atomicrex_job.py", line 53, in to_hdf
    self.potential.to_hdf(hdf=self.project_hdf5)

  File "/home/niklas/git_projects/pyiron_base/pyiron_base/generic/datacontainer.py", line 595, in to_hdf
    v.to_hdf(hdf=hdf, group_name=k)

  File "/home/niklas/git_projects/pyiron_base/pyiron_base/generic/datacontainer.py", line 595, in to_hdf
    v.to_hdf(hdf=hdf, group_name=k)

  File "/home/niklas/git_projects/pyiron_base/pyiron_base/generic/datacontainer.py", line 595, in to_hdf
    v.to_hdf(hdf=hdf, group_name=k)

  File "/home/niklas/git_projects/pyiron_base/pyiron_base/generic/datacontainer.py", line 595, in to_hdf
    v.to_hdf(hdf=hdf, group_name=k)

  File "/home/niklas/git_projects/pyiron_base/pyiron_base/generic/datacontainer.py", line 578, in to_hdf
    hdf = hdf.create_group(group_name)

  File "/home/niklas/git_projects/pyiron_base/pyiron_base/generic/hdfio.py", line 369, in create_group
    h_new = self[name].copy()

  File "/home/niklas/git_projects/pyiron_base/pyiron_base/generic/hdfio.py", line 822, in __getitem__
    if item in self.list_nodes():

  File "/home/niklas/git_projects/pyiron_base/pyiron_base/generic/hdfio.py", line 529, in list_nodes
    return self.list_all()["nodes"]

  File "/home/niklas/git_projects/pyiron_base/pyiron_base/generic/hdfio.py", line 513, in list_all
    iopy_nodes = self._filter_io_objects(groups)

  File "/home/niklas/git_projects/pyiron_base/pyiron_base/generic/hdfio.py", line 951, in _filter_io_objects
    [group for group in groups if self._get_h5io_type(group) in h5io_types]

  File "/home/niklas/git_projects/pyiron_base/pyiron_base/generic/hdfio.py", line 951, in <listcomp>
    [group for group in groups if self._get_h5io_type(group) in h5io_types]

  File "/home/niklas/git_projects/pyiron_base/pyiron_base/generic/hdfio.py", line 929, in _get_h5io_type
    return str(eval("".join([self._h5_group, name, "._v_title"])))

  File "<string>", line 1
    store.root.PotentialDF1.potential.electron_densities__index_1.rho_CuCu__index_0.parameters__index_5.node_0.5__index_0._v_title
                                                                                                                ^
SyntaxError: invalid decimal literal

@Leimeroth Can you try if this error persists with this branch?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
help wanted Extra attention is needed
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

4 participants