Support ingestion of transaction/correlation ids

[lukekim]

Enable the engine to ingest transaction or correlation ids, like order_id or trace_id.

To ingest unique ids today, they would need to be set as categories, which is not scalable beyond a few ids. The initial version of this feature would simply enable them to be ingested, and round-tripped through the data APIs. In the second iteration, they would be available for reference in reward functions, and the third the potential to automatically correlate and flatten multiple rows of data on the id.

Proposed manifest

dataspaces:
  - from: coinbase
    name: btcusd
    identifiers:
      - transaction_id
      - tick_id
    measurements:
      ...

alternative proposal:

dataspaces:
  - from: coinbase
    name: btcusd
    transactions:
      identifiers:
        - transaction_id
        - tick_id
    measurements:
      ...

alternative proposal:

dataspaces:
  - from: coinbase
    name: btcusd
    transactions:
      identifiers:
        - transaction_id
        - tick_id
      operator: and | or
    measurements:
      ...

alternative proposal:

dataspaces:
  - from: coinbase
    name: btcusd
    correlation_ids:
      - transaction_id
      - tick_id
    measurements:
      ...

[phillipleblanc]

Enable the engine to ingest transaction or correlation ids, like order_id or trace_id.

Are these useful as inputs to the model? Or only as inputs to the reward function/correlating multiple rows.

To ingest unique ids today, they would need to be set as categories, which is not scalable beyond a few ids. The initial version of this feature would simply enable them to be ingested, and round-tripped through the data APIs. In the second iteration, they would be available for reference in reward functions, and the third the potential to automatically correlate and flatten multiple rows of data on the id.

For the initial version, are we sending them through to the AI Engine - or just storing them on the Go side?

Proposed manifest

dataspaces:
  - from: coinbase
    name: btcusd
    identifiers:
      - transaction_id
      - tick_id
    measurements:
      ...

I like this version 👍

[lukekim]

@phillipleblanc the later.

Iteration 1. Just storing them, and round-tripping back through API/Dashboard for viewing.

Iteration 2. Maybe as input to reward function, not sure how valuable that would be yet.

Iteration 3. Multiple row correlation/merging.

[Corentin-pro]

I like the 2 first manifest proposals.

I do not think categories the same way as they add new column dimension to the dataset and they have a fixed set of possible values that can be common between records/entries.