transfer participants extraction for scan bigquery #1244

stephencompall-DA · 2025-06-26T20:22:26Z

SQL to retrieve all unique parties that were party to a transfer from ledger beginning up to record time, from Scan BigQuery. We only look at receivers because being a receiver at one point is the only way to become a sender.

Pull Request Checklist

Cluster Testing

If a cluster test is required, comment /cluster_test on this PR to request it, and ping someone with access to the DA-internal system to approve it.
If a hard-migration test is required (from the latest release), comment /hdm_test on this PR to request it, and ping someone with access to the DA-internal system to approve it.

PR Guidelines

Include any change that might be observable by our partners or affect their deployment in the release notes.
Specify fixed issues with Fixes #n, and mention issues worked on using #n
Include a screenshot for frontend-related PRs - see README or use your favorite screenshot tool

Merge Guidelines

Make the git commit message look sensible when squash-merging on GitHub (most likely: just copy your PR description).

Signed-off-by: Stephen Compall <stephen.compall@digitalasset.com>

moritzkiefer-da · 2025-06-26T20:24:20Z

We only look at receivers because being a receiver at one point is the only way to become a sender.

I might not have full context here but reward minting is a self-transfer where the only output is your sender change and you are not listed explicitly as a receiver. Aren't you missing that?

Signed-off-by: Stephen Compall <stephen.compall@digitalasset.com>

stephencompall-DA · 2025-06-26T20:52:04Z

I might not have full context here but reward minting is a self-transfer where the only output is your sender change and you are not listed explicitly as a receiver. Aren't you missing that?

It depends on what we mean by "transfer participant". Also, can those be incorporated simply by including the .transfer.sender in the union from the same choice arguments?

Copilot

Pull Request Overview

This PR introduces an SQL test resource to extract unique transfer participants from Scan BigQuery using JSON extraction. Key changes include adding a new SQL query that declares configuration parameters, extracts JSON arrays from transfer exercises, and filters records based on migration_id and record time.

apps/app/src/test/resources/transfer-participants.sql

Signed-off-by: Stephen Compall <stephen.compall@digitalasset.com>

stephencompall-DA · 2025-06-26T20:58:47Z

apps/app/src/test/resources/transfer-participants.sql

+  FROM daml_TransferOutput_jsons
+       INNER JOIN UNNEST(daml_TransferOutput_jsons.TransferOutput_array) AS TransferOutput;


Pulled from the docs here (this is JSON, not a struct, but a similar idea applies)

Longer explanation of the underlying syntax.

Signed-off-by: Stephen Compall <stephen.compall@digitalasset.com>

isegall-da · 2025-06-27T00:01:45Z

apps/app/src/test/resources/transfer-participants.sql

+         AND (e.migration_id < migration_id
+           OR (e.migration_id = migration_id
+               AND e.record_time <= UNIX_MICROS(as_of_record_time)))


Does this buy us anything given that the request was "all", and we're anyway putting "<= now" ?

A couple things:

As this is concurrent with ingestion, it's best to choose a time where ingestion has likely stabilized, so that there's no chance of partial data.

If you refactor or alter the query later, you can compare results knowing it's against the same set of transactions.

apps/app/src/test/resources/transfer-participants.sql

moritzkiefer-da · 2025-06-30T07:52:18Z

It depends on what we mean by "transfer participant". Also, can those be incorporated simply by including the .transfer.sender in the union from the same choice arguments?

Yes just including transfer.sender sounds fine to me.

…transfer-participants [skip ci] Signed-off-by: Stephen Compall <stephen.compall@digitalasset.com>

isegall-da · 2025-06-30T16:58:14Z

We only look at receivers because being a receiver at one point is the only way to become a sender.

I might not have full context here but reward minting is a self-transfer where the only output is your sender change and you are not listed explicitly as a receiver. Aren't you missing that?

@moritzkiefer-da I'm missing something here. How come you're not listed as the receiver? It's a self-transfer, so you are the receiver, aren't you? Or is it implemented as a "send to nobody", so all outputs are change to the sender?

moritzkiefer-da · 2025-06-30T17:58:32Z

@moritzkiefer-da I'm missing something here. How come you're not listed as the receiver? It's a self-transfer, so you are the receiver, aren't you? Or is it implemented as a "send to nobody", so all outputs are change to the sender?

send to nobody so it only has the sender change output

Signed-off-by: Stephen Compall <stephen.compall@digitalasset.com>

@moritzkiefer-da

- suggested by @moritzkiefer-da; thanks Signed-off-by: Stephen Compall <stephen.compall@digitalasset.com>

stephencompall-DA · 2025-06-30T19:17:41Z

Yes just including transfer.sender sounds fine to me.

5c20a9c

moritzkiefer-da

Query lgtm, I have two questions:

What is the plan for testing these queries?
Have we double checked what the intent is with the people requesting this query? We do have all transfer senders and receivers now. But there are two things I'm unsure about:
2.1. Is the intent really all transfer senders and receivers or is it anyone that ever held CC? Because the latter seems like a much simpler query where you just go over all created events and get the holder. It's almost identical in terms of result. Tap makes a difference but that's not even available on mainnet. Importantly this one is also robust against us adding more choices that create amulet other than AmuletRules_Transfer.
2.2. How do we want to treat lock holders? They are not receivers so if we want to include them we need to do more.

ray-roestenburg-da · 2025-07-02T09:03:09Z

apps/app/src/test/resources/transfer-participants.sql

+WITH daml_Transfer_jsons AS (
+       SELECT JSON_VALUE(e.argument,
+                         -- .transfer.sender
+                         '$.record.fields[0].value.record.fields[0].value.party')


is it possible to create functions for these kind of 'paths', or is that too far? probably easy to use for others.

I'm not sure, it depends on what functionality is available in temp function signatures.

maybe have a look?

I've changed total-supply to include this (as well as this file); what do you think?

I like it! I think it gives a good basis for reuse and ease of use.

…icipants [skip ci] Signed-off-by: Stephen Compall <stephen.compall@digitalasset.com>

Signed-off-by: Stephen Compall <stephen.compall@digitalasset.com>

initial transfer participants extraction for scan bigquery [skip ci]

cb1c1f0

Signed-off-by: Stephen Compall <stephen.compall@digitalasset.com>

stephencompall-DA self-assigned this Jun 26, 2025

stephencompall-DA marked this pull request as draft June 26, 2025 20:22

stephencompall-DA changed the title ~~initial transfer participants extraction for scan bigquery~~ transfer participants extraction for scan bigquery Jun 26, 2025

splat outputs, retrieve receivers, uniquify [skip ci]

b03e95b

Signed-off-by: Stephen Compall <stephen.compall@digitalasset.com>

stephencompall-DA marked this pull request as ready for review June 26, 2025 20:52

stephencompall-DA requested review from Copilot, isegall-da, rautenrieth-da and ray-roestenburg-da June 26, 2025 20:52

Copilot AI reviewed Jun 26, 2025

View reviewed changes

apps/app/src/test/resources/transfer-participants.sql Outdated Show resolved Hide resolved

unnest *and* inner join needed, thanks Copilot [skip ci]

a1192cd

Signed-off-by: Stephen Compall <stephen.compall@digitalasset.com>

stephencompall-DA commented Jun 26, 2025

View reviewed changes

stephencompall-DA added 2 commits June 26, 2025 20:59

label result column [skip ci]

c9bce52

Signed-off-by: Stephen Compall <stephen.compall@digitalasset.com>

make the join a bit clearer [skip ci]

d14fcb2

Signed-off-by: Stephen Compall <stephen.compall@digitalasset.com>

isegall-da reviewed Jun 27, 2025

View reviewed changes

apps/app/src/test/resources/transfer-participants.sql Outdated Show resolved Hide resolved

isegall-da reviewed Jun 27, 2025

View reviewed changes

apps/app/src/test/resources/transfer-participants.sql Outdated Show resolved Hide resolved

Merge commit '06bbc9ce34497f8c6f07313adbdec32656686377' into s11/703-…

3bc8b20

…transfer-participants [skip ci] Signed-off-by: Stephen Compall <stephen.compall@digitalasset.com>

stephencompall-DA added 2 commits June 30, 2025 18:55

subquery for transfer sender [skip ci]

33ccecd

Signed-off-by: Stephen Compall <stephen.compall@digitalasset.com>

unify senders and receivers [skip ci]

5c20a9c

- suggested by @moritzkiefer-da; thanks Signed-off-by: Stephen Compall <stephen.compall@digitalasset.com>

moritzkiefer-da approved these changes Jul 1, 2025

View reviewed changes

ray-roestenburg-da approved these changes Jul 2, 2025

View reviewed changes

ray-roestenburg-da reviewed Jul 2, 2025

View reviewed changes

stephencompall-DA added 7 commits July 2, 2025 22:13

Merge remote-tracking branch 'origin/main' into s11/703-transfer-part…

5dd3704

…icipants [skip ci] Signed-off-by: Stephen Compall <stephen.compall@digitalasset.com>

daml_record_path [skip ci]

cdd700e

Signed-off-by: Stephen Compall <stephen.compall@digitalasset.com>

actually working array-of-ints case [skip ci]

58322c2

Signed-off-by: Stephen Compall <stephen.compall@digitalasset.com>

change all the total supply calls to use it [skip ci]

84bbe2d

Signed-off-by: Stephen Compall <stephen.compall@digitalasset.com>

SELECT can't occur in a JSONPath calculation [skip ci]

f85b70a

Signed-off-by: Stephen Compall <stephen.compall@digitalasset.com>

copy utilities to transfer-participants [skip ci]

20e86d1

Signed-off-by: Stephen Compall <stephen.compall@digitalasset.com>

remove LIMIT [static]

9554e08

Signed-off-by: Stephen Compall <stephen.compall@digitalasset.com>

stephencompall-DA merged commit e173691 into main Jul 3, 2025
41 checks passed

stephencompall-DA deleted the s11/703-transfer-participants branch July 3, 2025 20:09

		FROM daml_TransferOutput_jsons
		INNER JOIN UNNEST(daml_TransferOutput_jsons.TransferOutput_array) AS TransferOutput;

transfer participants extraction for scan bigquery #1244

transfer participants extraction for scan bigquery #1244

Uh oh!

Conversation

stephencompall-DA commented Jun 26, 2025

Pull Request Checklist

Cluster Testing

PR Guidelines

Merge Guidelines

Uh oh!

moritzkiefer-da commented Jun 26, 2025

Uh oh!

stephencompall-DA commented Jun 26, 2025

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

moritzkiefer-da commented Jun 30, 2025

Uh oh!

isegall-da commented Jun 30, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

moritzkiefer-da commented Jun 30, 2025

Uh oh!

stephencompall-DA commented Jun 30, 2025

Uh oh!

moritzkiefer-da left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

isegall-da commented Jun 30, 2025 •

edited

Loading