Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support writing structures to compressed JSON (.json.gz .json.bz2 .json.xz .json.lzma) #3003

Merged
merged 3 commits into from
May 20, 2023

Conversation

janosh
Copy link
Member

@janosh janosh commented May 20, 2023

ffae4ea support writing structures to compressed JSON files
ba99183 check structure equality writing and reading from .json.gz .json.bz2 .json.xz .json.lzma

Related: #2994 helps with reducing test file sizes

@janosh janosh added enhancement A new feature or improvement to an existing one io Input/output functionality labels May 20, 2023
@janosh janosh enabled auto-merge (squash) May 20, 2023 15:20
@janosh
Copy link
Member Author

janosh commented May 20, 2023

Minimal example

from pymatgen.core import Lattice, Structure

FeO = Structure(
    lattice=Lattice.cubic(5),
    species=("Fe", "O"),
    coords=((0, 0, 0), (0.5, 0.5, 0.5)),
)

structure.to("FeO.json.gz")
structure.to("FeO.json.bz2")

@janosh
Copy link
Member Author

janosh commented May 20, 2023

@ml-evs Could you have a quick look at 0357d74? The Optimade test is still flaky.

@ml-evs
Copy link
Contributor

ml-evs commented May 20, 2023

@ml-evs Could you have a quick look at 0357d74? The Optimade test is still flaky.

Test changes look fine, from what I can see of the failures, maybe its some interaction where each job in the build matrix is hitting MP with the same query at the same time with the same IP and one is timing out? Could just boost OptimadeRester("mp", timeout=16) or something for the tests. I think a timeout of 5 (the current default) is equivalent to one TCP packet retry (which defaults to 3 seconds, see https://docs.python-requests.org/en/latest/user/advanced/#timeouts).

EDIT: Actually this probably isn't the case as the case as you're using pytest split etc so the build is quite staggered.

@janosh janosh merged commit c84bd59 into master May 20, 2023
@janosh janosh deleted the struct-to-json-gzip-bz2 branch May 20, 2023 16:01
@janosh
Copy link
Member Author

janosh commented May 20, 2023

@ml-evs Thanks for the quick reply! If the flakiness persists, I'll try increasing the timeout.

lbluque pushed a commit to lbluque/pymatgen that referenced this pull request May 23, 2023
…on.xz .json.lzma) (materialsproject#3003)

* support writing structures to compressed JSON files

.json
.json.gz
.json.bz2
.json.xz
.json.lzma

* check structure equality writing and reading from .json .json.gz .json.bz2 .json.xz .json.lzma
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement A new feature or improvement to an existing one io Input/output functionality
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants