feat: composite queries #4003

crusso · 2023-05-25T18:05:34Z

Add new form of composite query that:

can call other queries and composite queries
can't be called from updates
can't do async expressions (no way to queue up a callback since state is lost)
can't call local async or async* (because those might contain update calls)

From the (informal) doc:

Composite query functions

Although queries can be fast, when called from a frontend, yet trusted though slower, when called from an actor, they are also limited in what they can do. In particular, they cannot themselves issue further messages, including queries.

To address this limitation, the Internet Computer supports another flavour of query function called a composite query. Like plain queries, the state changes made by a composite query are transient, isolated and never committed. Moreover, composite queries cannot call update functions, including those implicit in async expressions (which require update calls under the hood). Unlike plain queries, composite queries can call query functions and composite query functions, on the same and other actors, but only provided those actors reside on the same subnet.

As a contrived example, consider generalising the previous Counter actor to a class of counters. Each instance of the class provides an additional composite query to sum the values of a given array of counters:

actor class Counter () {

  var count = 0;

  // ...

  public shared query func peek() : async Nat {
    count
  };

  public shared composite query func sum(counters : [Counter]) : async Nat {
    var sum = 0;
    for (counter in counters.vals())  {
      sum += await counter.peek();
    };
    sum
  }

}

Declaring sum as a composite query enables it call the peek queries of its argument counters.

While update message can call plain query functions, they cannot call composite query functions. This distinction, which is dictated by the current capabilites of the IC, explains why query functions and composite query functions are regarded as distinct types of shared functions.

Note that the composite query modifier is reflected in the type of a composite query function:

  sum : shared composite query ([Counter]) -> async Nat

Since only a composite query can call another composite query, you may be wondering how any composite query gets called at all? The answer to this chicken-and-egg problem is that composite queries are initiated outside the IC, typically by an application (such as a browser frontend) sending an ingress message invoking a composite query on a backend actor on the IC.

c.f. dfinity/interface-spec#144

…uery/3 annotations in idl (codegen and rts)

…failure of ICQC; execute test on drun

…nstance)

…t like Write (i.e. InUpdate) for now

… to update calls)

github-actions · 2023-06-12T17:16:09Z

Comparing from c09b751 to aaa9cfc:
In terms of gas, 3 tests regressed, 1 tests improved and the mean change is -0.0%.
In terms of size, 4 tests regressed and the mean change is +0.0%.

doc/md/examples/grammar.txt

…/ICQC

crusso · 2023-06-28T17:00:45Z

@rvanasa added you as a reviewer just so you are aware of what needs to change for VSCode, if anything (there's a new keyword composite that can appear in several places (see grammar.txt in PR)

luc-blaeser · 2023-06-29T07:31:05Z

doc/md/language-manual.md

@@ -1264,7 +1272,10 @@ The argument type of `inspect` depends on the interface of the enclosing actor.

 -   `arg : Blob`: the raw, binary content of the message argument;

-   `msg : <variant>`: a variant of *decoding* functions, where `<variant> == {…; #<id>: () → T; …}` contains one variant per shared function, `<id>`, of the actor. The variant’s tag identifies the function to be called; The variant’s argument is a function that, when applied, returns the (decoded) argument of the call as a value of type `T`.
+-   `msg : <variant>`: a variant of *decoding* functions, where `<variant> == {…; #<id>: () → T; …}` contains one variant per `shared` or `shared query` function, `<id>`, of the actor
+    (`shared composite query` functions are ignored).


This means that inspect is not called for composite queries? Or is the message argument for shared composite query simply mapped to shared query?

Inspect is indeed not called for composite queries (in my understanding) so we don't even include them in the variant type.

Just to clarify, even for querie method, inspect is only called if the method is invoked as an (ingress) update call (with consensus). It's not used for the faster query calls (no-concensus). Very confusing.

Since composite query methods can't be called with (ingress) update calls, just (ingress) query calls, there is no need to include them here.

luc-blaeser · 2023-06-29T07:33:55Z

doc/md/language-manual.md

-shared stable switch system throw to_candid true try type var while with
+
+actor and assert async async* await await* break case catch class
+composite continue debug debug_show do else flexible false for


Including composite as reserved keyword impacts backward compatibility (former identifiers named composite become invalid). This would technically be a breaking change, isn't it?

Yes, it is, but probably won't break many. Maybe we should bump to 0.10?

Another alternative would be to use composite_query (ugly IMO) or query* (confusing, subtle IMO)

I think composite query is much nicer to read. (Another possibility would be to introduce non-reserved keywords but the complexity is not worth it.) I would keep the current syntax.

luc-blaeser · 2023-06-29T07:36:32Z

doc/md/language-manual.md

@@ -2022,6 +2036,17 @@ The context type is a record to allow extension with further fields in future re

 :::

+Shared functions have different capabilities dependent on their qualification as `shared`, `shared query` or `shared composite query`.
+
+A `shared` function may call any `shared` or `shared query` function, but no `shared composite query` function.


I heard that update calls cannot invoke composite queries. Just for curiosity: Is it a technical reason? Or a conceptual aspect?

When an update calls a query, the query is actually executed in replicated mode for concensus on the result (state change is still discarded).
Similary, inspect message is only called for query functions that are executed in update mode (with concensus), not ordinary fast queries.
I suspect that replicated composite queries are much harder to implement and haven't been tackled yet, and might not be.

Thanks for the explanation.

luc-blaeser · 2023-06-29T07:39:38Z

doc/md/language-manual.md

+All varieties of shared functions may call unshared functions.
+
+Composite queries, though composable, can only be called externally (from a frontend) and cannot be initiated from an actor.
+


Maybe we should mention that each composite query call rolls back its state on function exit, and also does not pass state changes to sub-query or sub-composite-query calls. Therefore, repeated calls (which includes recursive calls) have different semantics than in classical procedural execution (like with update calls).

Will do, I was shying away from this ;->

Maybe, we could also just reference the general design description of that feature, because it is not a Motoko-specific aspect?

I've added some clarification, plus an example.

Great explanation!

luc-blaeser · 2023-06-29T07:42:26Z

src/codegen/compile.ml

@@ -7910,6 +7918,8 @@ module FuncDec = struct
        Lifecycle.trans env Lifecycle.Idle
      | Type.Shared Type.Query ->
        Lifecycle.trans env Lifecycle.PostQuery
+      | Type.Shared Type.Composite ->
+        Lifecycle.trans env Lifecycle.Idle


Just to double check: The subsequent state of the lifecycle is Idle and not PostQuery for composite queries?

Yes, this is an unfortunate hack. For queries, you can just generate the atomic body and enter PostQuery. But composite queries are not atomic, so you need to enter some state after every callback. Unfortunately, the code for generating callbacks always enters InUpdate and its hard to make the transition conditional.
I couldn't see any (easy) way to enter a different state without complicated the rest of the code to track whether we are in an ordinary callback or a composite callback. I could may set and test a global but that seemed gross too.

As far as I know, the Lifecycle stuff is just for sanity checking, so I think we are just losing some accuracy here.

luc-blaeser · 2023-06-29T07:50:45Z

src/mo_frontend/typing.ml

     | C.ErrorCap
     | C.QueryCap _
-     | C.NullCap ->
+     | C.NullCap
+     | _ ->
        error env at "M0047"
          "send capability required, but not available (need an enclosing async expression or function body)"


Probably, the error message is not very specific if the call between the modes update/query/composite query is not supported. But this is only cosmetics, one could always polish error messages...

Yes, I was considering revising those (maybe splitting send capability into update query and composite query capability.
Maybe this is the right time to do it.

This is nothing important, could be also done later. I guess we can do that as part of an error message improvement work (together as a team across the entire compiler).

luc-blaeser

I like that Motoko offers composite query calls with static checking.
Nice compiler implementation and documentation of this feature.

crusso · 2023-06-29T20:03:38Z

@luc-blaeser, if you could take another (quick) look, I've improved the error messages (a bit) and added some more doc about side-effects.

contrecoeur indeed.

doc/md/language-manual.md

test/run-drun/idl-composite.mo

luc-blaeser · 2023-06-30T10:00:24Z

@luc-blaeser, if you could take another (quick) look, I've improved the error messages (a bit) and added some more doc about side-effects.

Very clear explanation. I would say it is ready to merge...

ggreif · 2023-06-30T10:16:53Z

doc/md/actors-async.md

+```
+
+because none of the local updates to `state` are visible to any of the callers or callees.
+


~~Maybe add a note that composite queries cannot be run in replicated mode (as opposed to normal queries).~~
Line 1290 has it.

src/idllib/compile_js.ml

crusso added 16 commits May 25, 2023 18:30

composite queries

74102a7

allow local async function calls (but not in composite queries)

40ad1ca

add negative typing tests for composite queries

e58b205

adjust comment

b4d1bf2

test composites not callable from non-composites; support composite_q…

a4599ac

…uery/3 annotations in idl (codegen and rts)

skip ic-ref-run while I don't have it

78e7583

modify @add_cycles to only call add cycles when non-zero, preventing …

bf2da46

…failure of ICQC; execute test on drun

extend test to composite query on a different canister (actor class i…

bf72ae9

…nstance)

suppress get_refund when no cycles sent; treat Composite lifetime jus…

5b1a6e9

…t like Write (i.e. InUpdate) for now

fix grammar

5482a48

exclude composite queries from message inspection (which only applied…

8337b48

… to update calls)

newlines

0d76905

missing newline

ef78ae5

Merge branch 'master' into claudio/ICQC

2774383

extend printers.ml to T_COMPOSITE

342cf0a

commit blessed test output

e4dc92b

crusso added 7 commits June 22, 2023 10:02

Merge branch 'master' into claudio/ICQC

649d35e

Merge branch 'master' into claudio/ICQC

28f6f83

update test & output post merge

6a22a56

Merge branch 'master' into claudio/ICQC

45f3b67

specify composite queries; with other fixes

305cb85

add example

d2347b8

Merge branch 'master' into claudio/ICQC

9475d37

crusso commented Jun 27, 2023

View reviewed changes

doc/md/examples/grammar.txt Outdated Show resolved Hide resolved

crusso added 5 commits June 28, 2023 13:02

more doc

913b383

Merge branch 'claudio/ICQC' of github.com:dfinity/motoko into claudio…

69325ca

…/ICQC

fix grammar

e94299a

refactor <query>

073a2ef

partially revert change

e76dd85

Merge branch 'master' into claudio/ICQC

126051f

This was referenced Jun 29, 2023

Add composite keyword dfinity/node-motoko#75

Merged

Add composite keyword dfinity/vscode-motoko#221

Merged

luc-blaeser reviewed Jun 29, 2023

View reviewed changes

luc-blaeser approved these changes Jun 29, 2023

View reviewed changes

crusso added 3 commits June 29, 2023 13:24

test of semantics

43930ba

more doc and and example of bizarre semantics@

ecf17e1

improve error messages

55bc9f6

Changelog++

6ddcfb7

crusso commented Jun 30, 2023

View reviewed changes

doc/md/language-manual.md Outdated Show resolved Hide resolved

Update doc/md/language-manual.md

940d87b

crusso commented Jun 30, 2023

View reviewed changes

test/run-drun/idl-composite.mo Outdated Show resolved Hide resolved

crusso commented Jun 30, 2023

View reviewed changes

test/run-drun/idl-composite.mo Outdated Show resolved Hide resolved

crusso added 2 commits June 30, 2023 10:51

Update test/run-drun/idl-composite.mo

8b659de

Update test/run-drun/idl-composite.mo

bf2dee1

ggreif reviewed Jun 30, 2023

View reviewed changes

crusso commented Jun 30, 2023

View reviewed changes

src/idllib/compile_js.ml Outdated Show resolved Hide resolved

Update src/idllib/compile_js.ml

aaa9cfc

crusso added the automerge-squash When ready, merge (using squash) label Jun 30, 2023

mergify bot merged commit 2d9902f into master Jun 30, 2023
8 of 9 checks passed

mergify bot removed the automerge-squash When ready, merge (using squash) label Jun 30, 2023

mergify bot deleted the claudio/ICQC branch June 30, 2023 12:21

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: composite queries #4003

feat: composite queries #4003

crusso commented May 25, 2023 •

edited

github-actions bot commented Jun 12, 2023 •

edited

crusso commented Jun 28, 2023

luc-blaeser Jun 29, 2023

crusso Jun 29, 2023

crusso Jun 29, 2023

luc-blaeser Jun 29, 2023

crusso Jun 29, 2023 •

edited by ggreif

crusso Jun 29, 2023

luc-blaeser Jun 29, 2023

luc-blaeser Jun 29, 2023

crusso Jun 29, 2023

luc-blaeser Jun 29, 2023

luc-blaeser Jun 29, 2023

crusso Jun 29, 2023

luc-blaeser Jun 29, 2023

crusso Jun 29, 2023

luc-blaeser Jun 29, 2023

luc-blaeser Jun 29, 2023

crusso Jun 29, 2023

luc-blaeser Jun 29, 2023 •

edited

crusso Jun 29, 2023

luc-blaeser Jun 29, 2023

luc-blaeser left a comment

crusso commented Jun 29, 2023 •

edited

luc-blaeser commented Jun 30, 2023

ggreif Jun 30, 2023 •

edited

		All varieties of shared functions may call unshared functions.

		Composite queries, though composable, can only be called externally (from a frontend) and cannot be initiated from an actor.

		```

		because none of the local updates to `state` are visible to any of the callers or callees.

feat: composite queries #4003

feat: composite queries #4003

Conversation

crusso commented May 25, 2023 • edited

Composite query functions

github-actions bot commented Jun 12, 2023 • edited

crusso commented Jun 28, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

crusso Jun 29, 2023 • edited by ggreif

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

luc-blaeser Jun 29, 2023 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

luc-blaeser left a comment

Choose a reason for hiding this comment

crusso commented Jun 29, 2023 • edited

luc-blaeser commented Jun 30, 2023

ggreif Jun 30, 2023 • edited

Choose a reason for hiding this comment

crusso commented May 25, 2023 •

edited

github-actions bot commented Jun 12, 2023 •

edited

crusso Jun 29, 2023 •

edited by ggreif

luc-blaeser Jun 29, 2023 •

edited

crusso commented Jun 29, 2023 •

edited

ggreif Jun 30, 2023 •

edited