Merge d833f5c into cc644f3

README.md

@@ -118,6 +118,29 @@ The before and after structures match within an expected tolerance; note the rou
 | --- | --- | --- |
 | ![Zn8B8Pb4O24,volume=623,uid=bc2d](https://user-images.githubusercontent.com/45469701/169936372-e14a8bba-698a-4fc9-9d4b-fc5e1de7d67f.png) | <img src=https://user-images.githubusercontent.com/45469701/169936297-57f5afb6-c4ae-4d8a-8cbb-33dcaf190b98.png width=400> | <img src=https://user-images.githubusercontent.com/45469701/169937021-f6f60169-6965-4db1-9bbd-e8744521d570.png width=400> |
 
+## Limitations and Design Considerations
+
+There are some limitations and design considerations for `xtal2png`. While the round-off
+error is a necessary evil for encoding to a PNG file format, the unrounded NumPy arrays
+can be used directly instead if supported by the image model of interest. We choose a
+$64\times64$ representation by default which supports up to 52 sites within a unit cell.
+The maximum number of sites can be adjusted which changes the size of the
+representation. A square representation is used for greater compatibility with the
+common limitation of image-based models supporting only square image arrays. The choice
+of the default sidelength as a base-2 number (i.e. $2^6$) reflects common conventions of
+low-resolution images for image-based machine learning tasks. While the
+distance matrix does not directly contribute to the reconstruction in the current
+implementation of `xtal2png`, it serves a number of purposes. First, similar to the unit
+cell volume and space group information, it can provide additional guidance to the
+algorithm. A corresponding example would be the role of background vs. foreground in
+classification of wolves vs. huskies; oftentimes classification algorithms will pay
+attention to the background (such as presence of snow) in predicting the animal class.
+Likewise, providing contextual information such as volume, space group, and a distance
+matrix is additional information that can help the models to capture the essence of
+particular crystal structures. In a future implementation, we plan to reconstruct
+Euclidean coordinates from the distance matrices and homogenize (e.g. via weighted
+averaging) the explicit fractional coordinates with the reconstructed coordinates.
+
 ## Installation
 
 ### Anaconda (`conda`) installation (recommended)

docs/index.md

@@ -21,7 +21,20 @@ aria-label="Issue sparks-baird/xtal2png on GitHub">Issue</a>
 <a class="github-button" href="https://github.com/sparks-baird/xtal2png/discussions" data-icon="octicon-comment-discussion" data-size="large" aria-label="Discuss sparks-baird/xtal2png on GitHub">Discuss</a>
 <br><br>
 
-> Manuscript coming soon! (est. July 22, 2022)
+The latest advances in machine learning are often in natural language such as with long
+short-term memory networks (LSTMs) and transformers or image processing such as with
+generative adversarial networks (GANs), variational autoencoders (VAEs), and guided
+diffusion models; however, transfering these advances to adjacent domains such as
+materials informatics often takes years. `xtal2png` encodes and decodes crystal
+structures via grayscale PNG images (see e.g. \autoref{fig:64-bit}) by writing and
+reading the necessary information for crystal reconstruction (unit cell, atomic
+elements, atomic coordinates) as a square matrix of numbers, respectively. This is akin
+to making/reading a QR code for crystal structures, where the `xtal2png` representation
+is invertible. The ability to feed these images directly into image-based pipelines
+allows you, as a materials informatics practitioner, to get streamlined results for new
+state-of-the-art image-based machine learning models applied to crystal structure.
+
+> Manuscript coming soon!
 
  <!-- ![GitHub Repo stars](https://img.shields.io/github/stars/sparks-baird/xtal2png?style=social) ![GitHub followers](https://img.shields.io/github/followers/sgbaird?style=social) ![GitHub issues](https://img.shields.io/github/issues-raw/sparks-baird/xtal2png) ![GitHub closed issues](https://img.shields.io/github/issues-closed-raw/sparks-baird/xtal2png) -->
 

reports/paper.bib

@@ -68,6 +68,23 @@ @misc{Pythonista2021
   file = {C\:\\Users\\sterg\\Zotero\\storage\\8AMZF6UQ\\Pythonista.html}
 }
 
+@article{renInvertibleCrystallographicRepresentation2022a,
+  title = {An Invertible Crystallographic Representation for General Inverse Design of Inorganic Crystals with Targeted Properties},
+  author = {Ren, Zekun and Tian, Siyu Isaac Parker and Noh, Juhwan and Oviedo, Felipe and Xing, Guangzong and Li, Jiali and Liang, Qiaohao and Zhu, Ruiming and Aberle, Armin G. and Sun, Shijing and Wang, Xiaonan and Liu, Yi and Li, Qianxiao and Jayavelu, Senthilnath and Hippalgaonkar, Kedar and Jung, Yousung and Buonassisi, Tonio},
+  year = {2022},
+  month = jan,
+  journal = {Matter},
+  volume = {5},
+  number = {1},
+  pages = {314--335},
+  issn = {2590-2385},
+  doi = {10.1016/j.matt.2021.11.032},
+  abstract = {Realizing general inverse design could greatly accelerate the discovery of new materials with user-defined properties. However, state-of-the-art generative models tend to be limited to a specific composition or crystal structure. Herein, we present a framework capable of general inverse design (not limited to a given set of elements or crystal structures), featuring a generalized invertible representation that encodes crystals in both real and reciprocal space, and a property-structured latent space from a variational autoencoder (VAE). In three design cases, the framework generates 142 new crystals with user-defined formation energies, bandgap, thermoelectric (TE) power factor, and combinations thereof. These generated crystals, absent in the training database, are validated by first-principles calculations. The success rates (number of first-principles-validated target-satisfying crystals/number of designed crystals) ranges between 7.1\% and 38.9\%. These results represent a significant step toward property-driven general inverse design using generative models, although practical challenges remain when coupled with experimental synthesis.},
+  langid = {english},
+  keywords = {general inverse design,generalized crystallographic representation,generative model,invertible crystallographic representation,machine learning,property-structured latent space,solid-state materials,thermoelectrics,variational autoencoder},
+  file = {C\:\\Users\\sterg\\Zotero\\storage\\C597QIUA\\Ren et al_2022_An invertible crystallographic representation for general inverse design of.pdf;C\:\\Users\\sterg\\Zotero\\storage\\3A7B58GN\\S2590238521006251.html}
+}
+
 @misc{riebesellPymatviz2022,
   title = {Pymatviz},
   author = {Riebesell, Janosh},
@@ -138,6 +155,37 @@ @article{wangCompositionallyrestrictedAttentionbasedNetwork2020
   doi = {10.26434/chemrxiv.11869026.v1}
 }
 
+@article{weiningerSMILESChemicalLanguage1988,
+  title = {{{SMILES}}, a Chemical Language and Information System. 1. {{Introduction}} to Methodology and Encoding Rules},
+  author = {Weininger, David},
+  year = {1988},
+  month = feb,
+  journal = {Journal of Chemical Information and Computer Sciences},
+  volume = {28},
+  number = {1},
+  pages = {31--36},
+  publisher = {{American Chemical Society}},
+  issn = {0095-2338},
+  doi = {10.1021/ci00057a005},
+  file = {C\:\\Users\\sterg\\Zotero\\storage\\JDGWYLHJ\\Weininger_1988_SMILES, a chemical language and information system.pdf;C\:\\Users\\sterg\\Zotero\\storage\\ZL2984XI\\ci00057a005.html}
+}
+
+@article{xieCrystalDiffusionVariational2022,
+  title = {Crystal {{Diffusion Variational Autoencoder}} for {{Periodic Material Generation}}},
+  author = {Xie, Tian and Fu, Xiang and Ganea, Octavian-Eugen and Barzilay, Regina and Jaakkola, Tommi},
+  year = {2022},
+  month = mar,
+  journal = {arXiv:2110.06197 [cond-mat, physics:physics]},
+  eprint = {2110.06197},
+  eprinttype = {arxiv},
+  primaryclass = {cond-mat, physics:physics},
+  abstract = {Generating the periodic structure of stable materials is a long-standing challenge for the material design community. This task is difficult because stable materials only exist in a low-dimensional subspace of all possible periodic arrangements of atoms: 1) the coordinates must lie in the local energy minimum defined by quantum mechanics, and 2) global stability also requires the structure to follow the complex, yet specific bonding preferences between different atom types. Existing methods fail to incorporate these factors and often lack proper invariances. We propose a Crystal Diffusion Variational Autoencoder (CDVAE) that captures the physical inductive bias of material stability. By learning from the data distribution of stable materials, the decoder generates materials in a diffusion process that moves atomic coordinates towards a lower energy state and updates atom types to satisfy bonding preferences between neighbors. Our model also explicitly encodes interactions across periodic boundaries and respects permutation, translation, rotation, and periodic invariances. We significantly outperform past methods in three tasks: 1) reconstructing the input structure, 2) generating valid, diverse, and realistic materials, and 3) generating materials that optimize a specific property. We also provide several standard datasets and evaluation metrics for the broader machine learning community.},
+  archiveprefix = {arXiv},
+  langid = {english},
+  keywords = {Computer Science - Machine Learning,Condensed Matter - Materials Science,Physics - Computational Physics},
+  file = {C\:\\Users\\sterg\\Zotero\\storage\\VBQ9U4WF\\Xie et al_2022_Crystal Diffusion Variational Autoencoder for Periodic Material Generation.pdf}
+}
+
 @article{xieCrystalGraphConvolutional2017,
   title = {Crystal Graph Convolutional Neural Networks for an Accurate and Interpretable Prediction of Material Properties},
   author = {Xie, Tian and Grossman, Jeffrey C.},

reports/paper.md

@@ -42,10 +42,14 @@ bibliography: paper.bib
 The latest advances in machine learning are often in natural language such as with long
 short-term memory networks (LSTMs) and transformers or image processing such as with
 generative adversarial networks (GANs), variational autoencoders (VAEs), and guided
-diffusion models. Using `xtal2png` [@xtal2png] to encode/decode crystal structures via grayscale PNG images (see
-e.g. \autoref{fig:64-bit}) is akin to making/reading a QR code for crystal structures.
-This allows you, as a materials informatics practitioner, to get streamlined results for
-new state-of-the-art image-based machine learning models applied to crystal structure.
+diffusion models. `xtal2png` encodes and decodes crystal structures via grayscale PNG
+images (see e.g. \autoref{fig:64-bit}) by writing and reading the necessary information
+for crystal reconstruction (unit cell, atomic elements, atomic coordinates) as a square
+matrix of numbers, respectively. This is akin to making/reading a QR code for crystal
+structures, where the `xtal2png` representation is an invertible representation. The
+ability to feed these images directly into image-based pipelines allows you, as a
+materials informatics practitioner, to get streamlined results for new state-of-the-art
+image-based machine learning models applied to crystal structure.
 
 ![A real-size $64\times64$ pixel `xtal2png` representation of a crystal structure.\label{fig:64-bit}](figures/Zn8B8Pb4O24,volume=623,uid=bc2d.png)
 
@@ -61,24 +65,30 @@ informatics (chemical-formula-based predictions) was not publicly realized until
 approximately two-and-a-half years later, with peer-reviewed publications dating in late
 2020/ mid-2021
 [@goodallPredictingMaterialsProperties2020;@wangCompositionallyRestrictedAttentionbased2021].
-Another example of state-of-the-art algorithm domain transfer is refactoring image-processing models for crystal structure applications, with
-introduction [@kipfSemisupervisedClassificationGraph2016], domain transfer (preprint)
-[@xieCrystalGraphConvolutional2017], and peer-reviewed domain transferred
-[@xieCrystalGraphConvolutional2018] publication dates of Sep 2016, Oct 2017, and Apr
-2018, respectively. Here, we focus on the latter application: state-of-the-art domain
-transfer from image-processing to crystal structure.
-
-`xtal2png` [@xtal2png]
-([https://github.com/sparks-baird/xtal2png](https://github.com/sparks-baird/xtal2png))
-is a Python package that allows you to encode/decode a crystal structure to/from a
-grayscale PNG image for direct use with image-based machine learning models. Let's take
-[Google's image-to-image diffusion model,
+Another example of state-of-the-art algorithm domain transfer is refactoring
+image-processing models for crystal structure applications, with introduction, domain
+transfer (preprint), and peer-reviewed domain transferred publication dates of Sep 2016,
+Oct 2017, and Apr 2018 ([@kipfSemisupervisedClassificationGraph2016],
+[@xieCrystalGraphConvolutional2017], and [@xieCrystalGraphConvolutional2018]),
+respectively. Here, we focus on the latter application: state-of-the-art domain transfer
+from image-processing to crystal structure.
+
+`xtal2png` is a Python package that allows you to convert between a crystal structure
+and a grayscale PNG image for direct use with image-based machine learning models. Let's
+take [Google's image-to-image diffusion model,
 Palette](https://iterative-refinement.github.io/palette/)
-[@sahariaPaletteImagetoImageDiffusion2022]. Rather than dig into the code spending
-hours, days, or weeks modifying, debugging, and playing GitHub phone tag with the
-developers before you can (maybe) get preliminary results, `xtal2png` lets you get those
-results using the default instructions on the repository, assuming the instructions can
-be run without error.
+[@sahariaPaletteImagetoImageDiffusion2022], which supports unconditional image
+generation, conditional inpainting, and conditional JPEG restoration tasks with
+correlaries for crystal generation, structure prediction, and structure relaxation,
+respectively. Rather than dig into the code spending hours, days, or weeks modifying,
+debugging, and playing GitHub phone tag with the developers before you can (maybe) get
+preliminary results, `xtal2png` lets you get those results using the default
+instructions on the repository, assuming the instructions can be run without error.
+While there are other invertible representations for crystal structure
+[@xieCrystalDiffusionVariational2022,@renInvertibleCrystallographicRepresentation2022a]
+as well as cross-domain conversions such as converting between molecules as strings
+[@weiningerSMILESChemicalLanguage1988], to our knowledge, this is the first package that
+enables conversion between a crystal structure and an image file format.
 
 ![(a) upscaled example image and (b) legend of the `xtal2png` encoding.\label{fig:example-and-legend}](figures/example-and-legend.png)
 
@@ -93,17 +103,30 @@ symmetry, and atomic elements and coordinates which are each scaled individually
 according to the information type. An upscaled version of the PNG image and a legend of
 the representation are given in \autoref{fig:example-and-legend}. Due to the encoding of
 numerical values as grayscale PNG images (allowable values are integers between 0 and
-255), a small round-off error is present during a single round of encoding and decoding.
+255), a round-off error is present during a single round of encoding and decoding.
 An example comparing an original vs. decoded structure is given in
 \autoref{fig:original-decoded}.
 
+There are some limitations and design considerations for `xtal2png` which are described
+in `xtal2png`'s [documentation](https://xtal2png.readthedocs.io/en/latest/index.html) in
+the Overview section.
+
+At this time, it is unclear to what extent deviation from the aforementioned design
+choices will affect performance. We intend to use hyperparameter optimization to
+determine an optimal configuration for crystal structure generation tasks using the
+`xtal2png` representation.
+
 ![(a) Original and (b) `xtal2png` decoded visualizations of
 [`mp-560471`](https://materialsproject.org/materials/mp-560471/) / $Zn_2B_2PbO_6$. Images were generated using [ase visualizations](https://wiki.fysik.dtu.dk/ase/ase/visualize/visualize.html). \label{fig:original-decoded}](figures/original-decoded.png){ width=50% }
 
 The significance of the representation lies in being able to directly use the PNG
 representation with image-based models which often do not directly support custom
-dataset types, potentially saving days or weeks during the process of obtaining
-preliminary results on a newly released model.
+dataset types. We expect the use of `xtal2png` as a screening tool for such models to
+save significant user time of code refactoring and adaptation during the process of
+obtaining preliminary results on a newly released model. After obtaining preliminary
+results, you get to decide whether it's worth it to you to take on the
+higher-cost/higher-expertise task of modifying the codebase and using a more customized
+approach. Or, you can stick with the results of xtal2png. It's up to you!
 
 We plan to apply `xtal2png` to a probabilistic diffusion generative model as a
 proof-of-concept and present our findings in the near future.