CIP-0097? | Representing the Script Context as a SOP term

[michaelpj]

This is the other part split out from #455 that talks about changing the representation of the script context to be a SOP term rather than Data. It depends on CIP-85.

---

(rendered proposal in branch)

[L-as]

What is blocking us from simply making handling Data faster? Why is it so much slower than SOPs?

[MicroProofs]

Now that I understand how SOPs will work in the vm. I'll make some examples comparing and contrasting using data in Aiken vs using sop in Aiken. That way I can show off use case costs. Gonna work on the these examples this week.

[michele-nuzzi]

Do these tests have been performed only on scripts generated using PlutusTX?

[KtorZ]

Given that only PlutusTx (as far as I know?) is trying to translate the script context into some intermediate representation, I'd assume that yes.

So it could be interesting to get some before/after benchmarks for solutions like Plu-ts or Aiken regarding the impact of using SOP vs Data. Though, if the script context still comes in the form of Data from the ledger, I believe that the overhead of turning a Data into a SOP will be still high. (forget that last bit, my mind was still on CIP-0085 for a second; this proposal here is precisely about representing the script context as a SOP)

[michaelpj]

Yes, this is really only for people who actually try to convert the script context into another form. Although I do still think this is desirable - it's just obviously too expensive to be worth it at the moment.

So the numbers here are more meant to indicate how unacceptably high the cost is, not to argue that these are reductions that real users might see (the test cases in question are not real). I can see that's misleading, I'll rephrase.

[MicroProofs]

I'm curious on the current approach to handle maps and lists. Has this been fleshed out anymore?

[MicroProofs]

I like SOP for its fast destructuring, but the problem is the blocker on missing out equals data and serialise_data. As a different compromise how about a new builtin that is similar to case. Say case_constr.

Takes in a constant of type data that is a constr and any term. Then it applies all field arguments from constr plus the constr index to that term. Then that single term can handle constr index and whatever fields it has applies. The current approach is to unconstr_data then fst_pair to check index, then access each field by doing some combination of head_list and tail_list. This is vastly improved by only requiring one builtin to get a constr index and its fields from a constant data that is type constr.

Now I know this step is computationally more expensive than SOP. But since we have a lot of builtins that operate on data, I wonder if operating on data in a more efficient way might be the solution. Even converting script context to SOP does not change that we still need better ways to operate on data because of datum, redeemer, and script context still has data fields in datums, inputs, outputs, and redeemers fields.

Here's an example @michele-nuzzi

[[(builtin caseConstr) (con data #D87A820540)] other_term]

this evaluate to

[[[other_term (con integer 1)] (con data #05)]  (con data #40)]

you could imagine other_term is:

(lam constr_index (force [
    [
      [
        (force (builtin ifThenElse)) 
        [[(builtin equalsInteger) (con integer 0)] constr_index]
      ]
      (delay (con bool True))
    ]

    (delay
        (force [
          [
            [
              (force (builtin ifThenElse)) 
              [[(builtin equalsInteger) (con integer 1)] constr_index]
            ]
            (delay (lam field1
                (lam field2 
                    (con bool True)
                )
            ))
          ]
        ])
    )
  ])
)

EDIT: and perhaps this builtin should be moved to a new CIP, but the more I think about it, even with SOP, the more essential a builtin like this is.

[michele-nuzzi]

not sure case_constr would turn more efficient than manual destructuring, especially for structs with many fields, but if it serves to keep only one representation I'm down to it

[MicroProofs]

Typically I am seeing dApps that have use cases that check values, addresses, and datums of outputs against input. I'll come back and clarify some more in a bit but here's what I got.

Common Cases where I see SOP as more efficient:

• Working with value, i.e. List<Pair<ByteArray, List<Pair<ByteArray, Int>>>>
• Pulling out fields out of a constr like pulling out outputs, inputs, etc.

Common Cases where I see SOP as less inefficient:

• Comparing addresses, like comparing an address that comes from a redeemer, datum, or input address, and comparing it to an output address. In this case especially if the address comes from a datum or redeemer. This is where equals data is really powerful
• Comparing utxo references, i.e. transaction hash plus output index. Again comparing utxo references inputs to the Spend purpose field is much faster thanks to equals data. Also comparing a utxo ref to an output datum or output datum field that holds a utxo ref.

Cases where SOP has no effect:
Comparing redeemer/datum to output datums. This is a pain point where have a builtin like case_constr is nice since datum is still going to be data. I mean that datums and redeemers are given to us as data regardless of SOP on the script context.

[MicroProofs]

not sure case_constr would turn more efficient than manual destructuring, especially for structs with many fields, but if it serves to keep only one representation I'm down to it

It would be more efficient than having to do head and tail list to get each constr field and get the constr index.

[KtorZ]

Suggested change

	CIP: ?
	CIP: 97

[rphair]

(just noted this on Discord but probably better to address here)

@michaelpj would you come to our next CIP meeting - currently scheduled 20 June, 9:30 am UTC - to review this & invite any other Plutus representatives you think would have a stake in that issue? (cc @L-as)

[MicroProofs]

(just noted this on Discord but probably better to address here)

@michaelpj would you come to our next CIP meeting - currently scheduled 20 June, 9:30 am UTC - to review this & invite any other Plutus representatives you think would have a stake in that issue? (cc @L-as)

Could I get an invite @rphair? The discord links don't work for me.

[rphair]

yes @MicroProofs it would be good to have you there 🤓 https://discord.gg/kyaTyzkBqd

[rphair]

p.s. @MicroProofs seems @michaelpj is unavailable for meeting this month so we have bumped the discussion until meeting # 69 (which if regularly scheduled would fall on 04 July).

[L-as]

I don't think we should do this. Being able to hash the script context is highly convenient for evaluating snarks on it, since you can use the hash of the script context as a circuit witness trivially this way. If you can't hash it cheaply, it gets an order of magnitude harder, which is not nice.

The real issue is that the Plutus interpreter is unperformant.

[rphair]

@michaelpj @L-as @MicroProofs @michele-nuzzi this, with other Plutus items, is on the agenda for review early in the CIP editors' meeting Tuesday 04 July 16:00 UTC: https://hackmd.io/@cip-editors/69

[effectfully]

@michaelpj @L-as @MicroProofs @michele-nuzzi this, with other Plutus items, is on the agenda for review early in the CIP editors' meeting Tuesday 04 July 16:00 UTC: https://hackmd.io/@cip-editors/69

Michael is on leave, so I attempted to join instead of him, but it seems I wasn't let in. If you have any questions, I'd be happy to try to answer them, although I'm not sure what medium we should use.

[rphair]

@effectfully: but it seems I wasn't let in

Are you talking about "let in" to Discord? If not, here's the invite link: https://discord.gg/kyaTyzkBqd ... if so, I don't see any problems reported in the #verification channel there... let's confirm you're using Discord first, then try to follow the event link (in less than 2 hours): https://discord.com/events/971785110770831360/1120631775404761139

[effectfully]

Are you talking about "let in" to Discord?

No, I took a meeting from Michael's calendar. Sorry, I should've just asked here. Thank you for the clarification.

I've now joined the Discord server.

let's confirm you're using Discord first, then try to follow the event link (in less than 2 hours): https://discord.com/events/971785110770831360/1120631775404761139

Unfortunately, in 2 hours from that moment (i.e. now in an hour) I'm going to be traveling likely without access to the Internet. I'll try to make it, but chances of that are not high.

[rphair]

Tagging Waiting for Author due to @effectfully / @michaelpj report at last hour's CIP Editors' meeting that Plutus team is not currently sure this solution is preferable to alternatives they are also considering.