Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Simplify and correct tipsy sub-chunking. #2278

Merged
merged 2 commits into from
Jul 5, 2019

Conversation

matthewturk
Copy link
Member

Fixes #2276.

The tipsy chunking was not correctly subchunking, and this seemed to only show up when the particular ordering and sizes of the particle types was set in a specific way. This would always have a "hard fail", so there have to my knowledge this has not been the source of any quiet failures.

What this does is simplify the particle offset calculation. Instead of our previous way of doing things, which I could not quite parse out (and which was probably dating from before we did any subchunking) the new way is to compute a global offset for each particle type and then offset into that based on data_file.start. I also terminate earlier when trying to read particles that aren't in a sub-chunk.

This uses data from @claytonstrawn .

My verification script, which I ran with changing CHUNKSIZE in yt/geometry/particle_geometry_handler.py to a big number and leaving it at the small number:

import yt
import os

if os.path.isfile("testnihaoinput/g8.26e11/g8.26e11.00320.ewah"):
    print("removing ewah")
    os.unlink("testnihaoinput/g8.26e11/g8.26e11.00320.ewah")
if os.path.isfile("testnihaoinput/g8.26e11/g8.26e11.00320-hsml"):
    print("removing hsml")
    os.unlink("testnihaoinput/g8.26e11/g8.26e11.00320-hsml")

#ds = yt.load("testnihaoinput/g2.63e10/g2.63e10.00320")
ds = yt.load("testnihaoinput/g8.26e11/g8.26e11.00320")

dd = ds.all_data()

import h5py

f = h5py.File("output_big.h5", "w")

for ft, fn in ds.field_list:
    if ft not in f: f.create_group(ft)
    print("Creating ", ft, fn, dd[ft, fn].size)
    f[ft].create_dataset(fn, data=dd[ft, fn])
f.close()

The results are identical between the two. I should note that for some reason reading "smoothing_length" doesn't work, which is a bit confusing for me.

@matthewturk matthewturk added bug code frontends Things related to specific frontends demeshening Removing the global mesh for particles labels Jun 14, 2019
@matthewturk matthewturk added this to the 4.0 milestone Jun 14, 2019
size = self._pdtypes[ptype].itemsize
field_offsets[ptype] = pos
params = self.ds.parameters
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This line is triggering the flake8 error since params is no longer used.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should be fixed now!

@matthewturk matthewturk merged commit 33dcf44 into yt-project:yt-4.0 Jul 5, 2019
@matthewturk matthewturk deleted the tipsy_io_fix branch July 18, 2019 20:56
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug code frontends Things related to specific frontends demeshening Removing the global mesh for particles
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants