BUG: bring back support to group folder structure in RAMSES

[cphyc]

PR Summary

This brings back support to RAMSES folder structures like output_XXXXX/group_YYYYY. Fixes #3785.
In particular, the output number should be read off the XXXXX part, not the YYYYY part.

PR Checklist

• Adds a test for any bugs fixed. Adds tests for new features.

Note: I have tested the bugfix on a (massive) simulation with the following script. Unfortunately, we do not have any simulation dataset with the group structure on yt-data so testing would imply more work that I can reasonably put

import yt

yt.mylog.setLevel(40)
eps = 0.0001
bbox = [[0.5-eps]*3, [0.5+eps]*3]

def test_me(path):
    print(f"Testing {path=}")
    ds = yt.load(path, bbox=bbox)
    ds.index

    assert len(ds.index.domains) > 0

    ds.r["density"]
    ds.r["DM", "particle_identity"]

test_me("/data84/obelisk/RHD/OUTPUT_DIR/output_00075")
test_me("/data84/obelisk/RHD/OUTPUT_DIR/output_00075/group_00001/")
test_me("/data84/obelisk/RHD/OUTPUT_DIR/output_00075/group_00001/info_00075.txt")
test_me("/data84/obelisk/RHD/OUTPUTS/output_00075/info_00075.txt")
test_me("/data84/obelisk/RHD/OUTPUTS/output_00075")

[neutrinoceros]

I think it'd be worth trying to simplify this, especially if we can't afford to test it thoroughly.

[neutrinoceros]

I feel much more confortable with the simplified implementation, thanks ! would it be doable to add a small unit test for the RAMSESFileSanitizer class that wouldn't require pushing another dataset to the database ?

[neutrinoceros]

why not use the builtin pytest fixture tmp_path here ?

[neutrinoceros]

I anticipate this should fail on windows

[neutrinoceros]

There's a more "pytest-ic" way to write such tests with parametrised fixtures. Here's an example: https://github.com/neutrinoceros/yt_idefix/blob/14052de1a2d1a6404fcc7c4164a4aa4d1ba283a0/tests/conftest.py#L147

I can help refactoring this particular case if you want

[neutrinoceros]

@cphyc I'll take that back. I'm convinced these tests could be written in a simpler form but I haven't been able to find how. I'd be prone to accept this PR as is (provided tests actually pass on windows)

[neutrinoceros]

@cphyc I see that your tests are also failing on Jenkins because you forgot to ignore them on nose :)

[neutrinoceros]

pre-commit fails because of an incompatibility between black and click that happened this week (psf/black#2964)
it'll be fixed on yt's main branch tomorrow with the auto-update bot. In the mean time I'm patching it locally.

[neutrinoceros]

Also got an instability from matplotlib + pillow. CI should be stabilised globally by #3882

[lumberbot-app]

Owee, I'm MrMeeseeks, Look at me.

There seem to be a conflict, please backport manually. Here are approximate instructions:

1. Checkout backport branch and update it.

git checkout yt-4.0.x
git pull

2. Cherry pick the first parent branch of the this PR on top of the older branch:

git cherry-pick -x -m1 a1bce43eeb6ad6d3dfe15db0acf2971c2053a93a

3. You will likely have some merge/cherry-pick conflict here, fix them and commit:

git commit -am 'Backport PR #3811: BUG: bring back support to group folder structure in RAMSES'

4. Push to a named branch:

git push YOURFORK yt-4.0.x:auto-backport-of-pr-3811-on-yt-4.0.x

5. Create a PR against branch yt-4.0.x, I would have named this PR:

"Backport PR #3811 on branch yt-4.0.x (BUG: bring back support to group folder structure in RAMSES)"

And apply the correct labels and milestones.

Congratulations — you did some good work! Hopefully your backport PR will be tested by the continuous integration and merged soon!

Remember to remove the Still Needs Manual Backport label once the PR gets merged.

If these instructions are inaccurate, feel free to suggest an improvement.

[neutrinoceros]

manually backported as #3886