Skip to content

column descriptor deserialisation logic added #337

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 6 commits into from
Jun 17, 2025
Merged

column descriptor deserialisation logic added #337

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
36732d7
column descriptor deserialisation logic added
Krmjn09 Jun 16, 2025
1b4d862
removing eslint error
Krmjn09 Jun 16, 2025
68596a5
removed estlint warnings
Krmjn09 Jun 16, 2025
4b3331b
Update modules/rntuple.mjs
Krmjn09 Jun 16, 2025
386261f
Update rntuple.mjs
Krmjn09 Jun 16, 2025
30dbd9a
updated with local variable
Krmjn09 Jun 16, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
16 changes: 16 additions & 0 deletions demo/node/rntuple_test.js
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -43,3 +43,19 @@ else {
console.log(`OK: Field ${i}: ${field.fieldName} (${field.typeName})`);
}
}

// Column Check

if (!rntuple.builder?.columnDescriptors?.length)
console.error('FAILURE: No columns deserialized');
else {
console.log(`OK: ${rntuple.builder.columnDescriptors.length} column(s) deserialized`);
for (let i = 0; i < rntuple.builder.columnDescriptors.length; ++i) {
const column = rntuple.builder.columnDescriptors[i];
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The tests for both columns and fields should be made more comprehensive (it can be in a separate PR).
E.g, for fields, verify that the field name and types are those that you expect; for columns verify that the field ID and column types are the ones you expect etc.
Does not need to be for every field and column, but at least for the first and last field they should be there. Ideally you should test all the properties of the descriptors.

if (!column.fieldId)
console.error(`FAILURE: Column ${i} is missing fieldId`);
else
console.log(`OK: Column ${i} fieldId: ${column.fieldId} `);
}
}

49 changes: 49 additions & 0 deletions modules/rntuple.mjs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -127,6 +127,9 @@ deserializeHeader(header_blob) {

// List frame: list of field record frames
this._readFieldDescriptors(reader);

// List frame: list of column record frames
this._readColumnDescriptors(reader);
}

deserializeFooter(footer_blob) {
Expand Down Expand Up @@ -219,6 +222,52 @@ fieldListIsList = fieldListSize < 0;
this.fieldDescriptors = fieldDescriptors;
}

_readColumnDescriptors(reader) {
const columnListSize = reader.readS64(),
columnListIsList = columnListSize < 0;
if (!columnListIsList)
throw new Error('Column list frame is not a list frame, which is required.');
const columnListCount = reader.readU32(); // number of column entries
console.log('Column List Count:', columnListCount);
const columnDescriptors = [];
for (let i = 0; i < columnListCount; ++i) {
const columnRecordSize = reader.readS64(),
coltype = reader.readU16(),
bitsOnStorage = reader.readU16(),
fieldId = reader.readU32(),
flags = reader.readU16(),
representationIndex = reader.readU16();
console.log(`Column Record Size: ${columnRecordSize}`);
let firstElementIndex = null, minValue = null, maxValue = null;
if (flags & 0x1) firstElementIndex = reader.readU64();
if (flags & 0x2){
minValue = reader.readF64();
maxValue = reader.readF64();
}


const column = {
coltype,
bitsOnStorage,
fieldId,
flags,
representationIndex,
firstElementIndex,
minValue,
maxValue
};
column.isDeferred = function() {
return (this.flags & 0x01) !== 0;
};
column.isSuppressed = function() {
return this.firstElementIndex !== null && this.firstElementIndex < 0;
};

columnDescriptors.push(column);
}
this.columnDescriptors = columnDescriptors;
}

}

/** @summary Very preliminary function to read header/footer from RNTuple
Expand Down