Add Byte Array specialization by Waguramu · Pull Request #134 · Klebert-Engineering/simfil

Waguramu · 2026-02-04T10:34:58Z

Note

Medium Risk
Touches core parsing/tokenization, value/operator dispatch, and model serialization/JSON encoding, so regressions could affect query evaluation and persistence. Changes are well-scoped and covered by new unit tests.

Overview
Adds first-class bytes support to simfil end-to-end: a new ByteArray scalar type, b"..."/b'...' literals (including \xNN escapes), typeof/# support, as bytes casting, and comparison semantics against other bytes and numeric values.

Extends the model layer to store bytes in ModelPool (new column + serialization) and introduces a tagged JSON representation ({"_bytes":true,"hex":...}) with round-trip parsing/printing; also refactors typed node resolution to an ADL-based Model::resolve<T> API (replacing the old resolveObject/resolveArray helpers). Documentation and tests are updated to cover the new bytes behavior.

^{Written by Cursor Bugbot for commit eb25799. This will update automatically on new commits. Configure here.}

github-actions · 2026-02-04T10:37:12Z

Test Results

1 files ±0 1 suites ±0 6m 47s ⏱️ -4s
88 tests +2 88 ✅ +2 0 💤 ±0 0 ❌ ±0
93 runs +2 93 ✅ +2 0 💤 ±0 0 ❌ ±0

Results for commit eb25799. ± Comparison against base commit c50a0f3.

♻️ This comment has been updated with latest results.

src/model/string-pool.cpp

include/simfil/model/model.h

cursor · 2026-02-04T10:41:21Z

include/simfil/byte-array.h

+                if (static_cast<unsigned char>(bytes[i]) != 0)
+                    return std::nullopt;
+            }
+        }


Big-endian decode logic truncates wrong bytes for overflow

High Severity

The decodeBigEndianI64 function incorrectly handles byte arrays longer than 8 bytes. It checks if trailing bytes (indices 8+) are zero and then uses the leading 8 bytes. In big-endian, trailing bytes are the least significant, so this logic rejects valid values that fit in 64 bits (like 256 in a 10-byte array) while accepting values that overflow (like 2^72 truncated to its high bytes). The check for overflow needs to verify the leading excess bytes are zero (or proper sign extension), not the trailing ones.

Waguramu · 2026-02-04T10:42:42Z

src/model/model.cpp

        strings_(std::move(strings))
    {
        columns_.stringData_.reserve(detail::ColumnPageSize*4);
+        columns_.byteArrayData_.reserve(detail::ColumnPageSize*4);


Reuse stringData column instead.

Waguramu · 2026-02-04T10:44:11Z

src/model/model.cpp

    clear_and_shrink(columns.strings_);
    clear_and_shrink(columns.stringData_);
+    clear_and_shrink(columns.byteArrays_);
+    clear_and_shrink(columns.byteArrayData_);


Waguramu · 2026-02-04T10:44:32Z

src/model/model.cpp

+        auto idx = n.addr().index();
+        if (auto err = checkBounds(impl_->columns_.byteArrays_))
+            return tl::unexpected<Error>(*err);
+        auto& val = impl_->columns_.byteArrays_[idx];


Access stringData column

Waguramu · 2026-02-04T10:44:57Z

src/model/model.cpp

+ModelNode::Ptr ModelPool::newValue(simfil::ByteArray const& value)
+{
+    impl_->columns_.byteArrays_.emplace_back(Impl::StringRange{
+        (uint32_t)impl_->columns_.byteArrayData_.size(),


Also stringData column

src/model/string-pool.cpp

johannes-wolf · 2026-02-04T11:40:27Z

include/simfil/operator.h

+        return l == r.toDisplayString();
+    }
+
+    auto operator()(const ByteArray& l, int64_t r) const


I do not think this should be allowed:

We do not allow comparing strings to numbers without casting

toDisplayString() changes format depending on content and should not be used for comparison at all

(Same for double.)

johannes-wolf · 2026-02-04T11:41:04Z

include/simfil/operator.h

+        return l < r.toDisplayString();
+    }
+
+    auto operator()(const ByteArray& l, int64_t r) const


See previous comment about ByteArray to number conversion.

johannes-wolf · 2026-02-04T11:41:42Z

include/simfil/operator.h


+    auto operator()(const ByteArray& v) const -> std::string
+    {
+        return v.toDisplayString();


Imo. this should always use toHex().

johannes-wolf · 2026-02-04T11:44:09Z

include/simfil/operator.h


+    auto operator()(const ByteArray&) const -> std::string_view
+    {
+        static auto n = "string"sv;


Why is the type "string"? Since ByteArray acts differently (hex format, ...)
it should get its own type name or be handled exactly like string.

johannes-wolf · 2026-02-04T11:45:50Z

Do we really need to make the ByteArray accessible via simfil queries?

Waguramu · 2026-02-04T14:04:07Z

Do we really need to make the ByteArray accessible via simfil queries?

The relevant issue:

The NDS.Live specifies in the RoadCharacteristicsLayer the RoadLocationId attribute as array of bytes that stores numeric value, in case of HERE implementation the long will be encoded in the array of bytes, please see attached.

There are basically two use cases for decoding a byte array to numeric value:
   - Attribute inspection - the numeric representation should be visible as well
   - Search - the numeric value should used as an input value

Proposal:
- Introduce byte array as a specialization of the simfil string.
- Byte array value returns big endian decimal if the value fits into 8 bytes.
- Byte array value returns hexadecimal if the value doesn't fit into 8 bytes.
- In the Livesource ensure that all zserio fields of type byte array are converted to simfil byte arrays.

Could you take a look? Maybe you've got a great idea @johannes-wolf

include/simfil/operator.h

johannes-wolf · 2026-02-04T16:02:17Z

Do we really need to make the ByteArray accessible via simfil queries?

The relevant issue:

The NDS.Live specifies in the RoadCharacteristicsLayer the RoadLocationId attribute as array of bytes that stores numeric value, in case of HERE implementation the long will be encoded in the array of bytes, please see attached.

There are basically two use cases for decoding a byte array to numeric value:
   - Attribute inspection - the numeric representation should be visible as well
   - Search - the numeric value should used as an input value

Proposal:
- Introduce byte array as a specialization of the simfil string.
- Byte array value returns big endian decimal if the value fits into 8 bytes.
- Byte array value returns hexadecimal if the value doesn't fit into 8 bytes.
- In the Livesource ensure that all zserio fields of type byte array are converted to simfil byte arrays.

Could you take a look? Maybe you've got a great idea @johannes-wolf

Can we use the native simfil int & string values instead? Livesource could do the conversion to either an int or a string depending on the content/length.

include/simfil/operator.h

include/simfil/value.h

josephbirkner · 2026-02-09T09:50:10Z

TODO:

Add a Bytes value type which is returned by ValueType4CType<ByteArray>
Add a possibility to declare a byte array through the language (b"c0ffee")
Be consistent with cross-type operator logic
- typeof b"ff" -> "bytes"
- b"89899" as string -> "89899"
- "A normal string" as bytes -> b"41206E6F726D616C20737472696E67"
- b"89899" > 5 -> true
- b"89899" > "normal-string" -> false (incompatible comparison)
JSON conversion as tagged object (similar to _multimap): {'_bytes': true, 'number': <number-conv-if-possible>, 'data': 'data-as-base64'}
Move toDisplayString() to the erdblick inspection converter.

…>. Add a possibility to declare a byte array through the language. Fix consistency with cross-type operator logic. Add JSON conversion as tagged object (similar to _multimap). Remove toDisplayString().

…o byte-array

include/simfil/value.h

include/simfil/byte-array.h

include/simfil/operator.h

…g. Fix ByteArray display format not round-trippable with literal syntax (parse b string literal as hex bytes).

cursor

Cursor Bugbot has reviewed your changes and found 2 potential issues.

^{Bugbot Autofix is OFF. To automatically fix reported issues with Cloud Agents, enable Autofix in the Cursor dashboard.}

include/simfil/operator.h

src/token.cpp

PR comments are addressed :)

sonarqubecloud · 2026-02-12T18:01:36Z

Quality Gate passed

Issues
17 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

github-actions · 2026-02-12T18:10:56Z

Package	Line Rate	Branch Rate	Health
include.simfil	24%	10%	❌
include.simfil.model	89%	57%	✔
src	74%	46%	➖
src.model	74%	43%	➖
Summary	41% (6174 / 14923)	25% (4030 / 15857)	❌

johannes-wolf

Looking good!

Waguramu added 2 commits February 4, 2026 09:58

Add byte array specialization and processing

9ec0d0b

Add byte array

1240d9d

cursor bot reviewed Feb 4, 2026

View reviewed changes

Waguramu commented Feb 4, 2026

View reviewed changes

Waguramu requested review from johannes-wolf and josephbirkner February 4, 2026 10:46

Waguramu self-assigned this Feb 4, 2026

johannes-wolf previously requested changes Feb 4, 2026

View reviewed changes

Remove extraneous output

5d1bf1e

Waguramu changed the base branch from main to v0.6.3 February 4, 2026 13:58

cursor bot reviewed Feb 4, 2026

View reviewed changes

include/simfil/operator.h Outdated Show resolved Hide resolved

include/simfil/operator.h Show resolved Hide resolved

include/simfil/operator.h Show resolved Hide resolved

Remove byteArrayData_ column

988514e

cursor bot reviewed Feb 5, 2026

View reviewed changes

include/simfil/operator.h Show resolved Hide resolved

include/simfil/value.h Outdated Show resolved Hide resolved

Introduce a generic ADL-based Model::resolve function.

b5e58d9

josephbirkner and others added 4 commits February 10, 2026 07:20

Address PR comments.

6b01ea8

Add a Bytes value type which is returned by ValueType4CType<ByteArray…

74b02fc

…>. Add a possibility to declare a byte array through the language. Fix consistency with cross-type operator logic. Add JSON conversion as tagged object (similar to _multimap). Remove toDisplayString().

Merge remote-tracking branch 'origin/improvement/generic-resolve' int…

0d13c00

…o byte-array

Fix mp_key usage.

bd975b3

cursor bot reviewed Feb 10, 2026

View reviewed changes

include/simfil/value.h Show resolved Hide resolved

johannes-wolf reviewed Feb 10, 2026

View reviewed changes

include/simfil/byte-array.h Outdated Show resolved Hide resolved

include/simfil/operator.h Outdated Show resolved Hide resolved

Do not use k* prefixes. Use fmt::format instead of custom hex encodin…

2484898

…g. Fix ByteArray display format not round-trippable with literal syntax (parse b string literal as hex bytes).

cursor bot reviewed Feb 12, 2026

View reviewed changes

include/simfil/operator.h Outdated Show resolved Hide resolved

src/token.cpp Outdated Show resolved Hide resolved

Allow ByteArray conversion for strings, bool and int

f59f6f7

Waguramu added 2 commits February 12, 2026 16:34

Fix duplication

9f4b9a5

Fix tests

85304dc

Fix tests

eb25799

johannes-wolf self-requested a review February 13, 2026 11:34

johannes-wolf approved these changes Feb 13, 2026

View reviewed changes

Waguramu merged commit 6976c70 into v0.6.3 Feb 13, 2026
7 checks passed

Waguramu deleted the byte-array branch February 13, 2026 14:14

Conversation

Waguramu commented Feb 4, 2026 • edited by cursor bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions bot commented Feb 4, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Test Results

Uh oh!

Uh oh!

Uh oh!

cursor bot Feb 4, 2026

Choose a reason for hiding this comment

Big-endian decode logic truncates wrong bytes for overflow

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

johannes-wolf commented Feb 4, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Waguramu commented Feb 4, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

johannes-wolf commented Feb 4, 2026

Uh oh!

Uh oh!

Uh oh!

josephbirkner commented Feb 9, 2026 • edited by Waguramu Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

cursor bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

sonarqubecloud bot commented Feb 12, 2026

Quality Gate passed

Uh oh!

github-actions bot commented Feb 12, 2026

Uh oh!

johannes-wolf left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Waguramu commented Feb 4, 2026 •

edited by cursor bot

Loading

github-actions bot commented Feb 4, 2026 •

edited

Loading

johannes-wolf commented Feb 4, 2026 •

edited

Loading

josephbirkner commented Feb 9, 2026 •

edited by Waguramu

Loading