Parsing of list of dictionaries, i.e. with that N (...) wrappers

[Gnadi27]

Great tool and solid parsing for OpenFoam dictionary files. But I ran into two limitations with the same root cause:
For OpenFoam files (v2312 in my case) like polymesh/boundary or a decomposed case using the collated file handler, the files have a different structure, where instead of dictionaries they are lists of dictionaries, where the "keys" are wrapped inside a N (...) structure - which the foamlib grammar fails to unwrap and thus doesn't catch the keys inside such files.

Here and example polyMesh/boundary:

 /*--------------------------------*- C++ -*----------------------------------*\
| =========                 |                                                 |
| \\      /  F ield         | OpenFOAM: The Open Source CFD Toolbox           |
|  \\    /   O peration     | Version:  2312                                  |
|   \\  /    A nd           | Website:  www.openfoam.com                      |
|    \\/     M anipulation  |                                                 |
\*---------------------------------------------------------------------------*/
FoamFile
{
    version     2.0;
    format      ascii;
    arch        "LSB;label=32;scalar=64";
    class       polyBoundaryMesh;
    location    "constant/polyMesh";
    object      boundary;
}
// * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * //


6
(
    outlet
    {
        type            patch;
        startFace       221124;
        nFaces          1229;
    }

    inlet
    {
        type            patch;
        startFace       222353;
        nFaces          156;
    }
...
)

and for a decompose -fileHandler collated U-Fieldfile (processors4/0/U):

/*--------------------------------*- C++ -*----------------------------------*\
| =========                 |                                                 |
| \\      /  F ield         | OpenFOAM: The Open Source CFD Toolbox           |
|  \\    /   O peration     | Version:  2312                                  |
|   \\  /    A nd           | Website:  www.openfoam.com                      |
|    \\/     M anipulation  |                                                 |
\*---------------------------------------------------------------------------*/
FoamFile
{
    version     2.0;
    format      binary;
    arch        "LSB;label=32;scalar=64";
    class       decomposedBlockData;
    location    "0";
    object      U;
    data.format ascii;
    data.class  volVectorField;
}
// * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * //


// processor0
1281
(FoamFile
{
    version     2.0;
    format      ascii;
    arch        "LSB;label=32;scalar=64";
    class       volVectorField;
    location    "0";
    object      U;
}
dimensions      [0 1 -1 0 0 0 0];

internalField   uniform (1744.48 0 0);

boundaryField
{
    outlet
    {
        type            zeroGradient;
    }
    inlet
    {
    ...
)
...

[gerlero]

@Gnadi27 thanks for reporting! Actually, foamlib should be able to parse these kinds of files, so this is likely a bug.

Could you share the full files (as attachments) and the errors you get?

[Gnadi27]

processors4/0/U.C
constant/polyMesh/boundary.C

When I run a simple test script like this:

#!/usr/bin/env python3
"""Run this inside the OpenFOAM case directory."""

from pathlib import Path
import sys
import foamlib as fl
#import inspect
#print("foamlib imported from:", inspect.getfile(fl))

case = fl.FoamCase(Path(__file__).parent)

collated0U = fl.FoamFile("./processors4/0/U")
print(list(collated0U.keys())) #This returns: [None]

boundaryFile = fl.FoamFile("./constant/polyMesh/boundary")
print(list(boundaryFile.keys())) #This returns: [None]

collated0U["internalField"]["value"] = [111, 222, 333] #This errors out with: KeyError: ('internalField',)

The complete terminal output is this:

[None]
[None]
Traceback (most recent call last):
  File "/home/mgnadlinger/cases/foamlib_playground/fluid/foamtest.py", line 19, in <module>
    collated0U["internalField"]["value"] = [111, 222, 333] #This errors out
    ~~~~~~~~~~^^^^^^^^^^^^^^^^^
  File "/home/<User>/cases/foamlib_playground/env/lib/python3.12/site-packages/multicollections/abc.py", line 218, in __getitem__
    return self.getone(key)
           ^^^^^^^^^^^^^^^^
  File "/home/<User>/cases/foamlib_playground/env/lib/python3.12/site-packages/multicollections/abc.py", line 147, in wrapper
    return meth(self, key)
           ^^^^^^^^^^^^^^^
  File "/home/<User>/cases/foamlib_playground/env/lib/python3.12/site-packages/multicollections/abc.py", line 207, in getone
    return next(iter(self.getall(key)))
                     ^^^^^^^^^^^^^^^^
  File "/home/<User>/cases/foamlib_playground/env/lib/python3.12/site-packages/multicollections/abc.py", line 147, in wrapper
    return meth(self, key)
           ^^^^^^^^^^^^^^^
  File "/home/<User>/cases/foamlib_playground/env/lib/python3.12/site-packages/foamlib/_files/files.py", line 485, in getall
    ret = parsed.getall(keywords)
          ^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/<User>/cases/foamlib_playground/env/lib/python3.12/site-packages/multicollections/abc.py", line 147, in wrapper
    return meth(self, key)
           ^^^^^^^^^^^^^^^
  File "/home/<User>/cases/foamlib_playground/env/lib/python3.12/site-packages/foamlib/_files/_parsing/parsed.py", line 102, in getall
    return [entry.data for entry in self._parsed.getall(keywords)]
                                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/<User>/cases/foamlib_playground/env/lib/python3.12/site-packages/multicollections/abc.py", line 147, in wrapper
    return meth(self, key)
           ^^^^^^^^^^^^^^^
  File "/home/<User>/cases/foamlib_playground/env/lib/python3.12/site-packages/multicollections/__init__.py", line 53, in getall
    raise KeyError(key)
KeyError: ('internalField',)

[gerlero]

@Gnadi27 Here reporting back after trying out the files you provided before:

• boundary: everything seems OK (just used the pattern I posted before and it seems to work)
• U: might qualify as a bug/enhancement, since foamlib's parser currently doesn't have any special logic for those data structures (they look like a count of characters followed by a list of keyword entries that fills exactly that many characters)

Do you know if OpenFOAM will accept those files without the initial counts of characters of each? Cause maybe you could get rid of the counts before writing back to the files...

[Gnadi27]

Sadly no, removing these "integer count of character headers" results in an error when starting the simulation:

Reading field U

[0]
[0]
[0] --> FOAM FATAL IO ERROR: (openfoam-2312 patch=240625)
[0] incorrect first token, expected <int> or '(', found on line 23: punctuation '{'
[0]
[0]
[0] file: processors4/0/U at line 24.
[0]
[0]     From Foam::Istream& Foam::List<T>::readList(Foam::Istream&) [with T = char]
[0]     in file lnInclude/ListIO.C at line 300.
[0]
FOAM parallel run exiting

OpenFoam is expecting to find these counts and the counts must match the character count in their brackets.

[Gnadi27]

also removing just the intgers, but leaving the round ( .. ) brackets, results in OpenFoam getting stuck at reading the file

[gerlero]

@Gnadi27 Thanks, it was worth a try!

I wasn't (nor still am) familiar with the collated format, and I looked it up and there's not many instances of people wanting to parse it (nor can e.g. PyFoam parse it either). Sure there's not a different way to do what you want that doesn't involve parsing (and especially writing to) these files?

That said, however, this feature is definitely in scope for foamlib. By that, I mean, you're welcome to open an issue or pull request and we can start working out a possible implementation. I'm probably gonna have to rely on your code and/or input for most of the implementation though, given my lack of familiarity with these data structures.

[Gnadi27]

Supporting collated decomposed files is most definitely just a nice-to-have. By just going with the "normal" uncollated openFoam fileHandler this issue can be completely avoided.

The only real advantage of using the collated fileHandler I am aware of, is that the case directory looks cleaner and is easier to manually go through, compared to having 256 processor folders. And that there might be and advantage of reduced I/O operations because of overall less files to open and close, but in the grand scheme of simulation time, that's not really that big of a deal I suppose.