Formalizing the Schema #37

dgasmith · 2018-03-27T21:21:28Z

Description

To kick this off again, I have started creating a json-schema implementation based the discussed schema along with documentation via Sphinx and RTD. As a forward I should note that the implementation here is an attempt to build something concrete. Many things can be changed, but some sort of core needs to be formed to move forward.

This has three basic parts:

A small python module to help organize the schema and collection previous versions.
The start of a set of examples that will also be used for testing.
A RTD-based website that is auto generated from the current repo and started porting the current markdown files to this website. This will be up once this PR is merged.

The presented schema aims to provide a very basic interface with a few notable points such as lack of unit support. Before continuing to build out this schema I wanted to get intermediate feedback about the current setup and general organization.

Questions

Is there a good way to produce html-based documentation of schema automatically? I found several python/sphinx solution, but they do not seem to work. I also like swagger.io for REST interfaces; however, it doesn't seem to match up to the requirements of this schema.

Status

Ready to go

Continue to move to repo snapshot Finshes transition to project Finishes moving to a module finshed modulizing

langner · 2018-03-27T22:26:09Z

.gitignore

@@ -99,3 +99,11 @@ ENV/

 # mypy
 .mypy_cache/
+
+# Vim leftover


Not everyone uses vim, and this is not strictly related to the schema. Maybe put this in your private ignore list? (.git/info/exclude)

Sure, can do. I normally see these left in the .gitignore files, is there a reason to move them out?

IMO projects should have just entries relevant to their code. But of course it's not super important.

langner · 2018-03-27T22:31:02Z

qc_schema/dev/molecule.py

+            }
+        },
+        "masses": {
+            "description": "The mass of the molecule, canonical weights assumed if not given.",


The name is plural but the description has singular mass.

langner · 2018-03-27T22:34:07Z

qc_schema/dev/properties/mp_properties.py

+    "The doubles portion of the MP2 correlation energy including same-spin and opposite-spin correlations."
+}
+
+mp_properties['mp2_total_correlation_energy'] = {


Would be useful to also have a total MP energy property. I know it's easy to sum up, but some people like to see the total energy.

Sure, missed this.

berquist · 2018-03-28T13:33:02Z

qc_schema/dev/properties/scf_properties.py

+scf_properties = {}
+
+scf_properties["scf_one_electron_energy"] = {
+    "description": "The one-electron energy contribution [H] to the total SCF energy.",


Here and later on, what is [H]?

The core Hamiltonian, I dont really look at these descriptions as complete (they are not intended to be), but more about the general scaffold to build on.

langner · 2018-03-28T15:58:52Z

qc_schema/dev/molecule.py

+            "$ref": "#/definitions/provenance"
+        }
+    },
+    "required": ["symbols", "geometry"],


Is this actually enforced somehow?

If you run this through a json-schema validator you will get a raise like the following:

E jsonschema.exceptions.ValidationError: 'symbols' is a required property E Failed validating 'required' in schema['properties']['molecule']:

Im not sure how to require this at other levels.

Top-level Makefile

loriab · 2018-03-27T22:04:00Z

docs/source/tech_specs.rst

+How do we store large lists of objects (such as lists of atoms or bonds?)
+
+ 1) As a table of values (these values do very well in HDF5 as well)
+ 2) As a set of arrays


Should probably emphasize that 2) has been chosen.

On a personal note, I was using 1) in psi4 as closer to local layout. Switched over to 2) after LBNL mtg and found it perfectly easy to work in and much tidier to inspect.

loriab · 2018-03-27T22:10:13Z

docs/source/spec_components.rst

+ * Repeat of input components
+ * Driver return - Return of the requested driver (energy/gradient/etc)
+ * Properties - Other properties/values constructed as by products of the computation
+ * Provenance - Code, computer, user information, actual settings used by the code (lots


Request that this be a list so that multiple programs can contribute provenance to a calc.

loriab · 2018-03-27T23:22:19Z

qc_schema/dev/properties/mp_properties.py

+    "description": "The singles portion of the MP2 correlation energy. Zero except in ROHF."
+}
+
+mp_properties["mp2_double_energy"] = {


should this and the previous be "singles" and "doubles"?

loriab · 2018-03-30T06:12:50Z

qc_schema/dev/molecule.py

+            }
+        },
+        "geometry": {
+            "description": "The 3N XYZ coordinates of the atoms involved.",


explicitly mention flat (3N, ) rather than nester (N, 3), I think.

loriab · 2018-03-30T06:15:44Z

qc_schema/dev/properties/scf_properties.py

+scf_properties["scf_one_electron_energy"] = {
+    "description": "The one-electron energy contribution to the total SCF energy.",
+    "type": "number"
+}


If this is meant to refer to Hartree--Fock, not HF-or-DFT, I suggest we be explicit now and use "HF".

What if we define that is as simply \sum{D * H}? I worry that splitting HF and SCF into two parts could get much more complex.

ghutchis

The other thing missing IMHO was Bob Hanson's discussion about having a required 'magic header' that indicates that this JSON is, in fact, a QC_JSON version 1.0.

I think that's useful for handling files 'in the wild' for parsers, but also to ensure we can gracefully upgrade the spec over time.

ghutchis · 2018-03-30T14:05:41Z

Topology/README.md

@@ -23,15 +23,15 @@ The following are optional fields and default values (option, more a list of pos
  - `comment` (str) - Any additional comment one would attach to the molecule.


I think, based on #35 that it makes sense to have an "identifier" section, with comment, molecule name, formula, InChI, SMILES, etc. as optional. This would probably also include provenance and DOI.

Looking at my notes, I think these were grouped in the discussion.

Are you thinking of comment as a dictionary? I was considering moving up a few of those to top level optional fields.

No, I'm thinking that there's an obvious request for identifiers, and I think comment/title have usually been identifiers in QC codes.

So my suggestion is that there's an explicit identifier object - something like:

"identifier" : { "name": "aspirin", "comment": "training set", "formula": "C9H8O4", "smiles": "O=C(C)Oc1ccccc1C(=O)O", "inchikey": "BSYNRYMUTXBXSQ-UHFFFAOYSA-N" }

ghutchis · 2018-03-30T14:14:53Z

docs/source/index.rst

+
+**Translators:**
+ - `cclib <http://cclib.github.io>`_
+


openbabel <http://openbabel.org/>_

ghutchis · 2018-03-30T14:15:46Z

docs/source/index.rst

+ - `Avogadro <https://avogadro.cc>`_
+ - `Molecular Design Toolkit <https://github.com/Autodesk/molecular-design-toolkit>`_
+ - `VTK <http://www.vtk.org>`_
+


Jmol / JSmol <http://jmol.org/>_

ghutchis · 2018-03-30T14:18:20Z

docs/source/tech_specs.rst

+              ['N', 7, 14.20, 0, [0.214,12.124,1.12], [0,0,0]],
+              ...}
+
+    // 2) Storing fields as arrays: much more compact, but harder to read and edit


Perhaps its harder to read/edit as a human, but not in a program. Please don't use this language. As @loriab mentioned, this was the choice at the workshop and there are a lot of benefits on the "writing code" side.

I was mostly porting over current Markdown documentation to RST as an example, but there seems to be bringing up a bit of old discussion that is indeed solved. Should we covert the solved topics over to a simpler discussion of why the choice was made?

Yes, I think that's a good idea. As people implement, I don't think there should be ambiguity and I think a discussion of the choice is good.

(Incidentally, I'd suggest when the schema is done, that this be published in J. Cheminf.)

ghutchis · 2018-03-30T14:19:39Z

docs/source/tech_specs.rst

+
+.. code:: python
+
+    // 1) Storing fields as tables: creates an mmCIF/PDB-like layout


Please no. Let's not even suggest something remotely like the mess of PDB files.

ghutchis · 2018-03-30T14:22:00Z

qc_schema/dev/molecule.py

+            "description": "The name of the molecule.",
+            "type": "string"
+        },
+        "comment": {


Let's make these part of an object for "identifiers" to reflect feedback from #35

ghutchis · 2018-03-30T14:22:56Z

qc_schema/dev/molecule.py

+            }
+        },
+        "fix_com": {
+            "description": "Whether to adjust to the molecule to the COM or not.",


COM? I assume you mean "center of mass" but let's be explicit.

ghutchis · 2018-03-30T14:24:29Z

qc_schema/dev/properties/scf_properties.py

+"""
+}
+
+scf_properties["scf_dipole_moment"] = {


Does this have to be 'scf' dipole moment? Maybe I'm pedantic, but what if I run CCSD(T) - most programs don't print out two different dipole moments IIRC.

I think its good to clarify if someone does say CCSD bond orders. It can become unclear if you stored the SCF or CCSD dipoles.

Won't that lead to a proliferation of new objects? Maybe there should be a properties object:

property: { type: "dipole moment" method: "SCF" value: ... }

It doesn't seem like too many more fields as we will have energy definitions for these slots anyways. I might be missing your point on the object side of things however.

dgasmith · 2018-03-30T14:50:31Z

For the magic header, poking around there doesn't seem to be a full standard. However, finding a few examples this following seems popular:

{
  "name": "Schema name",
  "description": "A quick description",
  "url": "http://schema_host.org/schemas/v0.1/something.schema",
  "version": "v1.0",
}

Should this be spun off into its own issue however?

ghutchis · 2018-03-30T14:55:53Z

I'd probably go with name/version/url/description in that order, but implementations will go with alphabetical order.

Can we drop the description part? Then you'd have:

{
  "name": "QC_JSON",
  "url": "http://schema_host.org/schemas/v0.1/something.schema",
  "version": "v1.0",
}

That seems suitably unique for file/magic sensing.

ghutchis · 2018-03-30T14:56:24Z

IMHO, this is a great start and I'm in favor of merging this and starting new issues / pull requests.

dgasmith · 2018-03-30T15:29:31Z

Im in favor of merging as well. Certainly a lot left to do, but we can break the current topics off into issues.

cryos · 2018-03-30T15:40:11Z

This is looking great, and would agree that merging, and then breaking off into smaller PRs/discussions would be beneficial.

dgasmith · 2018-03-30T16:58:45Z

Going to pull this in and setup the docs. Please spin off issues or PR's as needed to fix current issues.

dgasmith added 10 commits March 21, 2018 15:42

First pass at a simple JSON Schema

e71d410

Toying around with testing systems

72e4e80

Adds various built in definitions

6843ee5

Adds a simple travis YAML

ba958ef

Reformats error definition

d4fc8f5

Starts the move to a small py repo for organizational purposes

a748f51

Adds MP and SCF properties

9d5faad

Continue refactoring to a git module

a71dc8e

Continue to move to repo snapshot Finshes transition to project Finishes moving to a module finshed modulizing

Starts documentation

1163ae8

Starts the generation of schema docs

0010693

langner reviewed Mar 27, 2018

View reviewed changes

MP2 total energy and travis fix

9a933f7

berquist reviewed Mar 28, 2018

View reviewed changes

Top-level Makefile

f7f7ab1

langner reviewed Mar 28, 2018

View reviewed changes

dgasmith and others added 4 commits March 28, 2018 12:14

Adds several validation failure tests

3106cc2

Writes topology keys out as a table

12e0188

Merge pull request #1 from berquist/dgasmith-schema

50c703d

Top-level Makefile

Updates travis to build docs via make

ca8cdbc

loriab reviewed Mar 30, 2018

View reviewed changes

Minor tweaks from Lori's feedback

8f92170

ghutchis reviewed Mar 30, 2018

View reviewed changes

dgasmith added 2 commits March 30, 2018 11:00

Magic 'header', multi description, and more partners

585e132

Updates the tech specs rst to reflect the current big/small discussion

a21678a

dgasmith mentioned this pull request Mar 30, 2018

Multi-method properties #39

Open

dgasmith merged commit bf61e6b into MolSSI:master Mar 30, 2018

		@@ -23,15 +23,15 @@ The following are optional fields and default values (option, more a list of pos
		- `comment` (str) - Any additional comment one would attach to the molecule.


		.. code:: python

		// 1) Storing fields as tables: creates an mmCIF/PDB-like layout

Formalizing the Schema #37

Formalizing the Schema #37

Conversation

dgasmith commented Mar 27, 2018 • edited

Description

Questions

Status

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ghutchis left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dgasmith commented Mar 30, 2018

ghutchis commented Mar 30, 2018

ghutchis commented Mar 30, 2018

dgasmith commented Mar 30, 2018

cryos commented Mar 30, 2018

dgasmith commented Mar 30, 2018

dgasmith commented Mar 27, 2018 •

edited