Grand Row Unification

[hawkfish]

There are 2-3 different row representations in the code use for aggregation, joins and sorting, and it would be helpful architecturally if they could be combined in some fashion to avoid code duplication and standardise the representations.

There is an existing discussion covering the initial discussions. A quick summary of the requirements so far:

• Ability to handle both data and aggregates
• Ability to handle either native or memcmp representations
• Ability to spool to disk for external sorting and scalable joins and aggregation

[hawkfish]

One problem that just occurred to me is that we have no way of serialising/deserialising aggregate states. IIRC Postgres requires aggregate states to be compound types (e.g., STRUCTs?) which makes serialisation generic. To do this we would have to add two more aggregate methods for states.

[Mytherin]

We have discussed serialization/deserialization of aggregate states into BLOBs before for the purpose of enabling incremental computation of aggregates. Specifically the idea was to have some functionality for extracting an aggregate state as a BLOB on the SQL level, rather than the actual result of the aggregate, e.g. you could do this:

SELECT key, STATE(SUM(value))
FROM table
GROUP BY key

This would give you the aggregate state of a specific type ("sum_state") stored as a blob. We could then expose the other aggregate functions on this state, specifically combine and finalize. This allows you to perform incremental computation of aggregates. For example, you could create a table holding this state:

CREATE TABLE partial_state AS
SELECT key, STATE(SUM(value)) state
FROM 'file1.parquet'
GROUP BY key

Then when new information comes in, you can do an upsert to either insert new states (if key is not present) or combine (if the specific key is present):

INSERT INTO partial_state
SELECT key, STATE(SUM(value)) state
FROM 'file2.parquet'
GROUP BY key
ON CONFLICT (key) DO UPDATE SET state=COMBINE_STATE(state, excluded.state)
WHERE key=excluded.key;

At any point, you can retrieve the current aggregates by calling finalize on them:

SELECT key, STATE_FINALIZE(state)
FROM partial_state

Perhaps that idea could be semi-combined with this idea, if we are adding functions to serialize/deserialize states into binary objects.

[github-actions]

This issue is stale because it has been open 90 days with no activity. Remove stale label or comment or this will be closed in 30 days.

[github-actions]

This issue was closed because it has been stale for 30 days with no activity.