BUG: Update GIZMO frontend to handle newer GIZMO versions

[mtryan83]

I'd like to suggest two changes to the GIZMO frontend to fix what I believe is a potential bug and to handle updates to the hdf5 interface (this was mentioned on the Slack #help channel in December(?) but I no longer have access to that post and can't reference it). I'm starting with a draft because a) technically these are two separate issues and could go into two separate PR's, but the first is so minor it seemed unnecessary and b) I'm not sure I've resolved the 2nd issue in the proper yt fashion.

1. dmetal check - GIZMO runs with METALS flag enabled but without COOL_METAL_LINES_BY_SPECIES, GALSF_FB_FIRE_RPROCESS, GALSF_FB_FIRE_AGE_TRACERS, or STARFORGE_FEEDBACK_TRACERS enabled will have a per-particle metallicity field (loaded as dmetal) with shape (NUM_PARTICLES, ). Thus, the fh[dmetal].shape[1] call will fail with an IndexError. I've added a check to ensure fh[dmetal] is 2d first (fh[dmetal].ndim>1).

2. Overriding Gadget _parse_parameter_file - newer versions of GIZMO have changed some of the header fields (described uner Reading Snapshots in the GIZMO doc) used by yt for determining the cosmology of the simulation. I've copied this function from the Gadget/data_structures.py with modifications. Now it checks if the ComovingIntegrationOn flag exists - if so, use that to determine if this is a cosmological run and pull cosmological parameters using updated names. Otherwise, fall back to checking for OmegaLambda and setting cosmological flag from that. Copying the whole function seems overkill though, is there a better way to do this?

Creating tests for both bugs should be straightforward (I think they'd be similar to the test_gizmo_64 test), but it doesn't look like the sample datasets are made with the newer GIZMO versions and I'm not sure how to provide any. I'll figure out the tests if that can get resolved.

PR Summary

PR Checklist

• New features are documented, with docstrings and narrative docs
• Adds a test for any bugs fixed. Adds tests for new features.

[jzuhone]

Hi @mtryan83, this is most welcome! I'll try to have a look at this in the next few days.

[neutrinoceros]

Thank you for working on this !
I agree that this work qualifies as a fix and could in principle be included in a bugfix release, but it needs to retain backward compatibility with previous versions of GIZMO. I see you attempted to satisfy this requirement already, but failing tests seem to indicate that it's not working yet.

Anyway, I know it's still in draft mode, but here are some preliminary remarks and questions.

[mtryan83]

Thank you for working on this ! I agree that this work qualifies as a fix and could in principle be included in a bugfix release, but it needs to retain backward compatibility with previous versions of GIZMO. I see you attempted to satisfy this requirement already, but failing tests seem to indicate that it's not working yet.

Anyway, I know it's still in draft mode, but here are some preliminary remarks and questions.

So looking at the some of the failing tests (the non gizmo tests), it looks like the problem is files that were previously only being considered an OWLS dataset or an EAGLE dataset are now also being considered GIZMO datasets. I don't know enough about OWLS and EAGLE formats to know if that's correct or not. Is that a test problem or I'm not distinguishing GIZMO files well enough?

For the gizmo tests, like yt.frontends.gizmo.tests.test_outputs.PixelizedProjectionValues_snap_N64L16_135_all_('gas', 'density')_0_None where/how is that even run?

Lastly, regarding most of your comments, the _parse_parameter_file function was copied directly from gadget/data_structures.py. I then made a few minimal changes (basically the stuff in the try block, lines 85-105, and the if not self.cosmological_simulation condition+message, line 116-121 ) so that it could use the updated stuff. So most of these other changes should be back-ported (side-ported?) to the Gadget reader too, but I would assume that's out of scope for this PR.

[neutrinoceros]

So most of these other changes should be back-ported (side-ported?) to the Gadget reader too, but I would assume that's out of scope for this PR.

agreed to consider it out of scope, but good to know !

[neutrinoceros]

note that image test failures are unrelated (#4540), and should pass next time this branch is updated

[mtryan83]

The macos test failure doesn't look like it's real.

Also, since I have once again been reminded why tests are important, how should I go about writing one for this case? I think it can just be a few unit tests, but obviously none of the sample datasets would trigger this.

[neutrinoceros]

The macos test failure doesn't look like it's real.

Indeed, it's just flaky. Rerunning fixed it.

[neutrinoceros]

As for testing, I would advise adding a new sample dataset if you can manage.

[jzuhone]

So most of these other changes should be back-ported (side-ported?) to the Gadget reader too, but I would assume that's out of scope for this PR.

I agree that it's out of scope, but given the significant amount of code duplication here in _parse_parameter_file it would be nice if we could do something about it soon. The best way to make sure we do it correctly whenever that happens is to add a new test dataset as @neutrinoceros suggested which uses the new functionality here and we can help you through getting it up on https://yt-project.org/data and getting a test written, which is probably just going to be a small answer test like the existing ones. A small-ish dataset, if possible, would be best.

[mtryan83]

I'm working on writing a test now, using a single snapshot from the gizmo Zeldovich pancake test problem (~20 MB).

For the _is_valid portion, should I use 2 files, one with GIZMO_version and one without? They would be identical otherwise (so having two seems like a bit of a waste), and would also be testing the metallicity code and the cosmology code. If I only use one file, I'm not sure how to test both the GIZMO_version code and the metallicity code without modifying the hdf5 file inside the test function, which seems like a bad idea.

[neutrinoceros]

20Mb is small enough that we can afford more than one file, especially for such a critical part of the logic. I say go for it.

[mtryan83]

snapshot_076_no_gizver.tar.gz
snapshot_076_wi_gizver.tar.gz
Ok, I've added a new test function: test_gizmo_zeldovich which requires test files gizmo_zeldovich/snapshot_076_XX_gizver.hdf5 where XX is either wi or no. I've attached them separately here (I couldn't fit them in one file, combined they were 3MB over the file limit).

For reference, the files can be generated using the gizmo zeldovich pancake test problem with the additional config flags COOLING and METAL. The _no_gizver then has the GIZMO_version header field manually removed. Snapshot 076 was picked randomly.

[mtryan83]

Just FYI, as of right now the only "failing" test is the new test I've developed, which is failing due to the missing files. What do I need to do to get them added as test datasets @neutrinoceros ?

[neutrinoceros]

Sorry I meant to reply earlier and forgot. There's a whole process to get the files permanently stored on our server, and it starts with a PR to the yt-project/website repository. I advise you take a look at previous PRs there. I'll have more time to provide more details tomorrow if need be !

[mtryan83]

I just put in a PR for the website. Is this the right process?

[neutrinoceros]

So far, so good 😊
Note: I'm on vacation for a couple days. I'll pick it up when I come back unless someone else does it while I'm gone !

[neutrinoceros]

alright, I see John took care of your companion PR. Now what we need is to do in this PR is to update yt/sample_data_registry.json with the new entries and check that we're able to load them through yt.load_sample
After that it should be all good.

[neutrinoceros]

import yt; ds = yt.load_sample("gizmo_zeldovich")

appears to be working (well done !). Now waiting to see if all tests pass

[neutrinoceros]

I do not know why the test is still failing with FileNotFoundError despite the file being correctly hosted on the website server. Maybe further action is required to install it on the machine running Jenkins CI ? @Xarthisius can you have a look please ?

[Xarthisius]

@yt-fido test this please.

[neutrinoceros]

LGTM !
@jzuhone, take your time if you still want to review this, otherwise feel free to merge !