Seeking clarifications

[matk86]

Since i dont have much experience doing the building stuff, could someone explain:

• The drawbacks in the design of the current builder in pymatpro

• what new features(other than mpi support and dependency specification capabilities) must the new builder support? As far as i understand the one in pymatpro implements the producer consumer model using the multiprocessing package.

• It would help my understanding a lot if someone could give me a concrete fully sketched out example of a builder with complicated dependencies.

• What package dependencies are allowed? for example the current get_db function in the helpers module is a reimplementaion of the function in matgendb.utils module. So should we use those from matgendb package or make everything self-contained?

Thanks

[shyuep]

• Drawbacks - too many to list.
• General suggestion is that pymatpro should be a thin layer. Everything that can be "outsourced" to public packages should be so. If pymatgen-db already has a db access, improve on that rather than writing your own in pymatpro. If builders need a "workflow" style process, use fireworks. The more code is public, the more eyes will look at the code. The reason why pymatpro is the way it is today is because privacy promotes laziness. There are no tests, people write all sorts of things badly, and no one ever looks at them.
• Dependencies - use as many public, well-maintained code bases as you like. The more, the better.

[dwinston]

I agree with @shyuep on shunting MP-specific stuff to public packages. The idea of this public repository specifically is to shunt materials-science-agnostic stuff from pymatgen-db so that more eyes look at the code that essentially just builds on pymongo to manage database building and maintenance -- I couldn't find a good open library for this, and the audience is anyone who uses Mongo and Python (like https://github.com/materialsvirtuallab/flamyngo).

The idea of https://github.com/materialsproject/emmet is to trim pymatpro and outsource stuff to a public repo. emmet should depend on this repo and on pymatgen-db, etc. (emmet:maggma as atomate:fireworks).