Skip to content

Support writing structures to compressed JSON (.json.gz .json.bz2 .json.xz .json.lzma) - #3003

Merged
janosh merged 3 commits into
masterfrom
struct-to-json-gzip-bz2
May 20, 2023
Merged

Support writing structures to compressed JSON (.json.gz .json.bz2 .json.xz .json.lzma)#3003
janosh merged 3 commits into
masterfrom
struct-to-json-gzip-bz2

Conversation

@janosh

@janosh janosh commented May 20, 2023

Copy link
Copy Markdown
Member

ffae4ea support writing structures to compressed JSON files
ba99183 check structure equality writing and reading from .json.gz .json.bz2 .json.xz .json.lzma

Related: #2994 helps with reducing test file sizes

@janosh janosh added enhancement A new feature or improvement to an existing one io Input/output functionality labels May 20, 2023
@janosh
janosh enabled auto-merge (squash) May 20, 2023 15:20
@janosh

janosh commented May 20, 2023

Copy link
Copy Markdown
Member Author

Minimal example

from pymatgen.core import Lattice, Structure

FeO = Structure(
    lattice=Lattice.cubic(5),
    species=("Fe", "O"),
    coords=((0, 0, 0), (0.5, 0.5, 0.5)),
)

structure.to("FeO.json.gz")
structure.to("FeO.json.bz2")

@janosh

janosh commented May 20, 2023

Copy link
Copy Markdown
Member Author

@ml-evs Could you have a quick look at 0357d74? The Optimade test is still flaky.

@ml-evs

ml-evs commented May 20, 2023

Copy link
Copy Markdown
Contributor

@ml-evs Could you have a quick look at 0357d74? The Optimade test is still flaky.

Test changes look fine, from what I can see of the failures, maybe its some interaction where each job in the build matrix is hitting MP with the same query at the same time with the same IP and one is timing out? Could just boost OptimadeRester("mp", timeout=16) or something for the tests. I think a timeout of 5 (the current default) is equivalent to one TCP packet retry (which defaults to 3 seconds, see https://docs.python-requests.org/en/latest/user/advanced/#timeouts).

EDIT: Actually this probably isn't the case as the case as you're using pytest split etc so the build is quite staggered.

@janosh
janosh merged commit c84bd59 into master May 20, 2023
@janosh
janosh deleted the struct-to-json-gzip-bz2 branch May 20, 2023 16:01
@janosh

janosh commented May 20, 2023

Copy link
Copy Markdown
Member Author

@ml-evs Thanks for the quick reply! If the flakiness persists, I'll try increasing the timeout.

lbluque pushed a commit to lbluque/pymatgen that referenced this pull request May 23, 2023
…on.xz .json.lzma) (materialsproject#3003)

* support writing structures to compressed JSON files

.json
.json.gz
.json.bz2
.json.xz
.json.lzma

* check structure equality writing and reading from .json .json.gz .json.bz2 .json.xz .json.lzma
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement A new feature or improvement to an existing one io Input/output functionality

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants