SDDL2: locally-consumed field not usable as array length or parameter within same record

[Tabaie]

The documentation describes using a local field as an array length or record parameter as valid syntax, but it is rejected with a semantic error.

Repro using examples from documentation:

From arrays-collections.md, "Non-Instant-Parse Arrays" section:

Record Data() = {
  count: UInt32LE,
  values: Int32LE[count]  # Size depends on local field
}

data: Data()

Fails with:

semantic error: Undefined variable: 'count'
  3 |   values: Int32LE[count]  # Size depends on local field
    |                   ~~~~~

The #139 workaround (field[is_present]) also implies dynamic array lengths were expected to work. Tested on 2f9487062b9e86f3ac6799b1e50d87c8dd07d593.

[daniellerozenblit]

Hi, thanks for your feedback.

The SDDL2 documentation currently doesn't clearly differentiate between the full language specification (i.e., what the language is designed to express) and the features that are actually supported today. Non-instant parse records, in particular, are not currently supported.

I'm actively working on revamping the documentation to make the distinction between the language trajectory and the currently supported feature set much clearer. This should help avoid confusion like this going forward.

Apologies for the inconvenience, and thanks again for raising this!

[Tabaie]

Thank you @daniellerozenblit for the quick and clear response. Is there a way to do this kind of count-first arrays currently? Or is this the canonical way and I just have to wait until it's supported?

[daniellerozenblit]

SDDL does currently support count-first arrays. The approach depends on your structure:

If you just need a single count + array at the top level, you can declare them directly:

count: UInt32LE
values: Int32LE[count]

If you have repeated count-prefixed data chunks, define a parameterized Record and pass the count in:

Record Data(COUNT) = {
  field_a: Int32LE,
  field_b: Int32LE,
  field_c: Int32LE[COUNT]
}
count_0: UInt32LE
data_0: Data(count_0)
count_1: UInt32LE
data_1: Data(count_1)

Note that records cannot directly reference top-level fields — you need to pass them as parameters.

[Tabaie]

I have multi-dimensional arrays so there definitely will be a length parameter inside records.

[jkbonfield]

I was hopeful given the above reply, but sadly this only works for the top-level still however.

For example this works:

$ hd _
00000000  05 68 65 6c 6c 6f 05 77  6f 72 6c 64              |.hello.world|
0000000c

$ cat eg.sddl2
record PStr(len) {
  data: Byte[len]
}

l1: UInt8
t1: PStr(l1)
l2: UInt8
t2: PStr(l2)

$ ./zli compress --force --profile sddl2 --profile-arg eg.sddl2 _ -o /dev/null
Compressed 12 -> 32 (0.38x) in 0.105 ms, 0.11 MB/s

If I move those lengths into a record, it doesn't:

$ cat eg.sddl2
record PStr(len) {
  data: Byte[len]
}

record HelloWorld() {
    l1: UInt8,
    t1: PStr(l1),
    l2: UInt8,
    t2: PStr(l2)
}

hw: HelloWorld

$ ./zli compress --force --profile sddl2 --profile-arg eg.sddl2 _ -o /dev/null
Unhandled Exception:
	eg.sddl2:7:14-16: semantic error: Undefined variable: 'l1'
  7 |     t1: PStr(l1),
    |              ~~

This means we can't use p-string style encodings within a record. Is there a plan to add this feature to SDDL2?

It may be possible to work around this if there is some looping syntax that can be applied at the main level.

[daniellerozenblit]

@jkbonfield There are definitely plans to add this feature. This is a construct that we refer to as a scan record (see the north star vision here).

I don't have an exact timeline for the availability of this feature, but this is actually something that I am actively working on. I can update this issue once available!