RFC: Having a Witnet Data Requests singleton registry

[guidiaz]

Current nuisances and limitations

1. A whole new WitnetPriceFeed contract needs to be deployed everytime a change in the underlying data request is required. No matter if the change is minimal, like adjusting the witnet reward, these are the steps and costs systematically involved on every single chain supporting the modified price feed:

   • Deploy new instance of the WitnetPriceFeed contract: requiring ~2,000,000 gas units.
   • In some cases, a new contract verification process may be required.
   • On the WitnetPriceRouter instance, re-route currency pair to new WintePriceFeed instance: ~175,000 gas units.
   • Eventually, price feed poller bots and the data feeds explorer website will notice about a currency pair being re-routed on the WPR instance, as they need to interact directly with the underlying contract address for certain operations.

2. There's no obvious way to introspect the expected result size of a data request that gets eventully posted to the WitnetRequestBoard contract. At the moment, this limitation prevents the WRB from providing exact calculation of the Witnet base reward for a specifc data request. When assessing possible solutions, let's have in mind that on-the-fly introspecting of new incoming data requests may indeed cost too much gas, though.

3. Badly formed DRs can now pass through the current WRB, as it performs no formal validation at all.

4. The WitnetPriceRouter contract has neither way to verify the result type of the underlying data request for a price feed is actually an integer. While the WPR is owned and managed by the Witnet Foundation, the WPR should whatsoever refrain from accepting data requests returning non-integer values.

5. Web3 apps willing to eavesdrop either the sources, retrieval types (HttpGet/HttpPost/RNG) or witnessing parameters being posted (and eventually solved) to the WitnetRequestBoard have no other option but importing the witnet-requests NPM package, which is only available on Javascript (atm). It would be nice to have some on-chain read-only methods to either disassamble any DR posted to the WRB, or have a place where to look-up for related metadata.

6. There's no obvious way in the witnet-solidity-bridge repo to compose/re-use parameterized data requests, so devs cannot help but engineer their own way onto this.

My proposal

Let's have a WitnetDataRequests registry contract where:

• Anyone can separately register RAD-part bytecodes (ie. DrRad) and any possible combination of quality-of-service parameters (ie. DrQoS).

• Upon registry, a formal validation would be performed by the smart contract, as to make sure no incongruity, incompleteness or rubbish is saved in the Registry.

• Unique identifiers would also be generated (ie. DrRadHash, DrQoSHash) upon registry.

• Bound to each identifier, all metadata extracted during validation would be directly accessible via specific look-up methods (directly reading from storage, w/ no need for on-the-fly calculations).

• Registration of RAD-parts would accept parameterized bytecodes, in which case a list of args (ie. DrRadArgs) would then need to be declared (an offset and a Witnet-supported type for every possible parameter).

• For what it may eventually be worth, the registry would also allow to stamp any combination of pre-registred DrQoS and DrRad bytecodes.

• Similarly to the IWitnetRequest interface, the Registry would also provide:

  • The resulting bytecode of any combination of pre-registred DrQoS and DrRad hashes (either parameterized or not).
  • The SHA256 hash of the any possibly combined bytecode.

Benefits

The utility of this WitnetDataRequests Registry contract would be much augmented if adopted from eventually enhanced WitnetRequestBoard and WitnetPriceRouter contracts.

WitnetRequestBoard

By having an immutable reference to the WF-promoted WDR instance, we'll make sure that:

• Queries posted to the WRB would refer to DRs that should have been previously published, and formally validated, into the WDR registry. This way, there will be no chance for the WRB to ever relay a badly formed DR to the Witnet side-chain.

• The WRB would manage to provide an exact calculation of the minimally acceptable reward for a DR to be relayed into Witnet, based on both the QoS parameters and the result type (and length) of the DR.

• Clients to the WRB would have the chance to post queries refering either a fully-stamped DR (ie. DrHash), a Rad/Qos static combination (ie. DrRadHash, DrQosHash), or even a parameterized Rad/QoS combination (ie. DrRadHash, DrQoSHash, DrArgs).

WitnetPriceRouter

By relaying on the WDR instance refered by the WRB:

• For every currency pair, the WPR would separately keep track of the Radon script, the QoS parameters and the feeding contract that will be used for eventual price udpates. This way, the Radon script or the QoS parameters could get upgraded independently, and without needing to deploy a whole new feeding contract. The upgrade of any of these two would cost like ~50,000 gas units.

• The WPR would reject any attempt to bind a currency pair to a Witnet Radon script not returning a proper integer value.

• As the WitnetPriceFeed would be agnostic to the underlying data request used for updating the price, and by additionally establishing the WitnetPriceFeed to be "clonable", one single instance could be re-used every time a new currency pair is added to the WPR, as long as no improvement in the feeding logic is desired. The cloning of a contract (based on the minimal proxy approach) only takes like ~200,000 gas units on regular EVMs, no matter how complex the cloned contract may be.

[Tommytrg]

In my opinion, the most remarkable point to implementing a Data Request singleton registry with the current state of the bridge is the first one. Regarding the other nuisances and limitations mentioned, I have some questions:

1. What's the cost of being unable to estimate the exact reward on Witnet right now?
2. What kind of validator do you have in mind: something that only checks that all the fields are fulfilled or also validates that the operator pipeline is correct?
3. What could be the cost of implementing that Data Request validator on-chain?
4. Is the implementation of this validator only to prevent human errors? If so, is it worth doing that check on-chain?
5. Why is a problem the inability of the WitnetPriceRouter to verify that the routed request result is an integer?
6. I'm not sure I understand the last point. Do you have a use case in mind to compose DR on-chain?

After reading the proposed solution, my main concern is related to the interaction with it:

7. Do the users need to deal with registering the different parts of a data request? Are they also going to be able to post a data request in the same we are doing it today?
8. If it's also possible to post a request calling _witnetPostrequest(WitnetRequest _request), could that request be split in DrQoS, DR and RAD transparently for the caller?
9. What is the flow to compose and deploy DR? I assume you have to call publishDrQoS(Witnet.DrQoS), publishDrRad(bytes) and publishDrRadTemplate(bb, u[], W..Types[]) but then how do you compose the DR components and deploy a WitnetRequest?

[guidiaz]

5. Why is a problem the inability of the WitnetPriceRouter to verify that the routed request result is an integer?

Remember that WPR is an ownable contract, as some identity needs to be authorized to add/modify routed pfs within a WPR instance.

Then, the WPR accepting pfs is not much of a problem, because if the owner makes a mistake, this could always get amended. However, if we happened to have a WDRR being capable of formally validating the result type of a data request, then the WPR could just take advantange of this feature. Forcing a WPR instance to only accept data requests returning integers would some how:

(i) avoid errors from whoever is owning the WPR instance (not necessarily a human);

and (ii), gurantee users to the WPR instance that it would just not be possible to add just any kind of data requests.

Moreover, if the WDRR happened to be capable of validating sources of a data request (or at least number of sources), a WPR instance could trustlessly commit to accept only data requests with a minimum of data sources. Again, having price feeds with data requests refering just one source needs not to be a problem. But again, the WPR could just be (optionally) improved by leveraging the formal validation capabilities we may end up adding to a WDRR.

[guidiaz]

6. I'm not sure I understand the last point. Do you have a use case in mind to compose DR on-chain?

Think of parameterized data requests. For example, one to count number of reproductions of a youtube video. As a develeper, the source will always be the same, but a content id will have to be provided upon every request. Instead of having to pre- register any possible combination of content ids, you would just need to registered this "parametrized" data request only once, and then use it as many times with any possible content id.

[guidiaz]

7. Do the users need to deal with registering the different parts of a data request? Are they also going to be able to post a data request in the same we are doing it today?

Yes, instead of having to code, compile and deploy a new IWitnetRequest-inheriting contract for every single new data request the user may need to handle, she would just need to call the WDRR passing a pre-compiled bytecode.

I guess we could leave the door open to that if we planned to integrate a WDRR with currently centralized WRB. When moving towards a decentralized bridge, though, I would suggest to let the WRB only accept previously validated data requests on the WDRR.

[guidiaz]

8. If it's also possible to post a request calling _witnetPostrequest(WitnetRequest _request), could that request be split in DrQoS, DR and RAD transparently for the caller?

I guess there would just be new methods substituting/overriding the one you mention. For instance, _witnetPostRequest(bytes32 _drRadHash, bytes32 _drRadQosHash), _witnetPostRequest(bytes32 _drRadHash, Witnet.QoS _drRadQos), and so on.

[guidiaz]

9. What is the flow to compose and deploy DR? I assume you have to call publishDrQoS(Witnet.DrQoS), publishDrRad(bytes) and publishDrRadTemplate(bb, u[], W..Types[]) but then how do you compose the DR components and deploy a WitnetRequest?

The data request bytecode would be generated just like now: offchain. For instance, by using "witnet-request" npm module, or the Sheikah app. As stated before, there would not be any need to deploy WitnetRequest contracts anymore.

Note: This proposal doesn't include the possibility to compose data requests on chain. It could be a future enhancement, but it would be complex, costly, and I personally don't see a practical utility on having this feature. I believe that enabling registration of (precompiled) "parameterized" data requests, would just be enough to cover most of Witnet use-cases to come.