Design options and use cases for Oracles

[lock9]

According to the accepted proposal neo-project/proposals#78

List use cases
Discuss what is the best way to implement it (best design)
Discuss the scenarios we should cover
Discuss security issues we should be concerned
Implementation design

Edit: Implementation design proposals without images will be punished with a thumbsdown (👎 )

[igormcoelho]

@lock9 if you prefer to group here all decisions for oracles, I'll help you.

This last issue I opened has nearly a complete proposal for oracles, in two different execution modes (trusted and non-trusted), and hiding CN IP address, which must be a requirement in my opinion.
#932

The proposal for two different operation types on oracles is here (I'll close that one, to work here): #798

Proposal for TransactionAttributeUsage.Url: #769 (this one was independently proposed by Erik and Shargon in Native NEP-5 big PR). This is already implemented.

I couldn't find the original proposal of oracles, which was by Shargon and Erik, some time ago.

---

@doubiliu is very active there and has given priceless contributions for the design as well, so let's summon here as well ;)

[igormcoelho]

I'll quote my last comment there: (from #932)

@doubiliu interesting analysis.
In fact, like you said, this would indeed affect synchronous process, but we can (and must) do it in a completely asynchronous way (just like transactions). I'll present two options.

• Option A: dettached transaction + tx validated oracle data
  Before some transactions are selected for block, the CN must verify that its oracle data is already processed (remember that the outer ring will send its result over p2p, for every tx they have on mempool). So, when a tx is submitted from a CN, it must already contain the oracle data processed (by the independent outer ring), otherwise it won't even be a candidate for block (will keep waiting on mempool).
  What may happen is that a CN proposes a tx, with validated oracle data (according to the outer ring, passed by p2p), but other CN only have tx (not outer ring p2p oracle data). In this case, these CN would have to do a similar request of RequestTransaction, but like RequestOracleDataForTransaction, and this information would need to be received during block time. At this moment, it's not time to execute the oracle itself, just to receive the same previously validated oracle results that the speaker node already has (flowing through p2p). This only affects oracles that require trusted execution (if not trusted mode, then problem is solved already, as long block proposal includes oracle data). If this oracle data is not received in time (on trusted mode), it's the same effect that the tx hasn't been received in time, thus invalidating the block. We can design a payload that flows together with transactions on p2p, so this increases the odds that, when someone receives the tx, it already receives the validated oracle data (but risks of invalidating blocks on trusted mode due to non-existing oracle data will still be possible).

• Option B: Attached transaction + tx validated oracle data (tx with oracles must be routed by oracle ring)
  Another option is: oracle data could be a pre-requisite as well, meaning that tx with oracle can only flow through p2p when oracle data has already been validated (this would work for both trusted and non-trusted mode). This means that tx with oracle data would be submitted first to the oracle ring, get the oracle data, and then it is automatically routed through regular p2p (this means that every CN that receives this tx, will also already receive the processed oracle data). And since the oracle ring itself will route the transactions, this reduces the risk of someone trying to cancel specific transactions, because they don't like oracle result (like lottery contracts, for example). This attack will still be possible if oracle ring is completely cut-off from p2p, but as long as this ring has direct connection (the one I mentioned to shargon), the attacker will only know the result when it's already being routed through p2p, and even re-routed if lost, so an attack would be quite hard in this sense.

[eryeer]

As far as I know, we are docking some oracle third-party organizations (such as Chain.Link), and then these third-party organizations perhaps will provide oracle data services to Neo in the future. So is there any relation or difference between the oracle design we discussed here and the docking?

[igormcoelho]

@eryeer I'm not aware of the docking you mention, can you clarify? WIll it require an extra attribute transaction field url, like the one we are proposing? Or can it work without it? How can a dapp user be certain that provided onchain oracle service is trusted?

I'll present my "mental challenge" on the Oracle feature. Suppose we want to build a lottery system on Neo, that depends on random.org website, and will give millions of dollars in prize. Can we build it using the third-party provided? This is what I'm trying to (I wanted to) achieve here.

[steven1227]

@eryeer I'm not aware of the docking you mention, can you clarify? WIll it require an extra attribute transaction field url, like the one we are proposing? Or can it work without it? How can a dapp user be certain that provided onchain oracle service is trusted?

I'll present my "mental challenge" on the Oracle feature. Suppose we want to build a lottery system on Neo, that depends on random.org website, and will give millions of dollars in prize. Can we build it using the third-party provided? This is what I'm trying to (I wanted to) achieve here.

Of-course we can use the third-party oracle service. There are no conflicts whether NEO has its own oracle service or there are third-parties.

[longfeiWan9]

@igormcoelho FYI, we are working with ChainLink to build Oracle service on NEO so that projects can use, and Chainlink are working with a lot of big data providers . So do we still need to put so much effort to provide Oracle in NEO by ourselves.

[steven1227]

@eryeer @lock9 @doubiliu @igormcoelho @erikzhang
Before the design, I believe the issue we have to put in the discussion is why we need to build our own Oracle and what is the pros and cons.

[eryeer]

@igormcoelho We can discuss the relevant issues in face-to-face meetings in September.

[EdgeDLT]

@wanglongfei88 @steven1227 I don't think we should be dependent on third party solutions and tokens to offer this functionality.

Chainlink support should expand the options for NEO developers, not replace them. NEO should still offer its own Oracle solution.

[WyattMufson]

I agree with @EdgeDLT on this one. NEO should offer a full solution for developers without relying on the existence of a third party. Having Chainlink support to supplement the built in NEO oracle service would be ideal for developers.

[igormcoelho]

My point is just if ChainLink can provide such a guarantee that I mentioned. If it can, we can adopt it, I'm not against third party solutions.

Let me clarify the challenges I see:

1. CN cannot be exposed. Thats why I'm discussing outer ring oracle nodes to access data relay tx. I dont think CN should even have Internet access, so how could them directly communicate with thirdparty? Couldnt a failure in thirdparty cause all CN to be hacked?
   There's block proposal time issue too.. is it enough time for all CN to request thirdparty to process every tx, in few milliseconds?
2. If its not direct contact of CN with third party, then thirdparty should directly relay all oracle-enabled tx. For example, if someone makes a poker-dapp game move, and doesnt like the random outcome,could it not send the tx and try again? I dont think so... so, all oracle enabled tx should guaranteed to be relayed,or fail by block time expiration.

If that works, or solution is even better, that's good for us,I'm just raising dome concerns on oracle design that werent raised before,and I think they're important.

[lock9]

We need incentives and the features needed for these providers to work with neo. The ability of them being paid using smart contracts.

Like people already mentioned before, we need some kind of delegation, because CN won't access the internet, it will probably count on some sort of proxy to make these requests. We need some sort of 'oracle fees'. Maybe, for these fees, we can even allow them to be paid using nep-5 contracts, like @vncoelho mentioned in #978.

We need to allow Neo to support different providers and make this "profitable" for them (allow Neo users to pay them fees).

[igormcoelho]

We can discuss the relevant issues in face-to-face meetings in September.

Sure @eryeer , we want the best of the best for Neo ;)

[EdgeDLT]

Consider the example set by NeoFS. There is an outer ring of storage nodes paid in GAS (creating incentives for NeoFS network participants and NEO holders), and an inner ring of nodes that handle auditing/accounting.

The same model should work for an oracle network, except network participants would be offering "trustworthy" request data for review instead of providing storage space. The inner ring would be responsible for handling payments/punishment, validating request data, and somehow making the validated information available to CNs.

[igormcoelho]

In fact, I didnt know NeoFS work like that 😂 interesting.. when I say "outer ring", for me is just an "external ring" of nodes, so we wouldnt need internal ones (btw, NeoFS is even more complex than this proposal, congratulations to NSPCC).

[EdgeDLT]

Yep, in this case I guess CNs or contracts act as the 'inner ring' seen in NeoFS.

I just think it is important that NEO provides its own infrastructure to handle transaction routing or to let people participate by running nodes (ideally with incentives powered by GAS). I don't want to see a situation where important features like oracles for contracts are unavailable to NEO developers that do not want to trust a third party project and its network or token.

The way I see third parties like Chainlink fitting into the picture is by connecting as a part of the NEO oracle "external ring." It's up to them to build a product that will convince developers to use those nodes over the native NEO infrastructure. This is what I mean by avoiding dependency; NEO should provide an implementation that these projects can then extend on their own terms.

[igormcoelho]

The only thing I'm afraid is thirdparty not providing Oracle service with one block finality (by third party externally reading and writing on the chain).. block finality is not common, other chains dont have it, so it might be ok for them, but Neo has it and we need to take advantage of it. We need to better discuss it, I dont know the details.

[pappas999]

The concept of a 'baked in' Oracle into the protocol is a great (and ground breaking) idea, and in my opinion if it can be pulled off properly, is an important feature to have for a number of reasons (which Edge touched on). Here's some of my thoughts:

Most platforms aren't suited to have this in the protocol due to their architecture. NEO is suitable for this due to the lower number of consensus nodes coupled with the dBFT consensus mechanism. However as stated above, there are some careful considerations that need to be taken into account re: internet access, block times, security and effect on consensus etc. If it's done, it should be relatively easy for a developer to make use of this Oracle for simple requests, and be able to pay for the request with GAS.

Chainlink has built a great decentralized Oracle network which is quickly becoming the standard for getting off-chain data to all blockchain Smart Contracts. However not all use cases require this level security, decentralization and complexity required to implement.

ie lets say a Smart Contract on NEO pays millions of $ out insurance to farmers based off weather events, then you'll likely want to use multiple sources of data for weather rather than just 1, and have the results aggregated off chain and passed to the Smart Contract. This is where something like Chainlink shines, it gives you confidence in the data due to its ability to pull data from multiple sources & oracles, have it aggregated and with nodes staking tokens and having penalties for if they are found out to be acting bad.

However if you are a simple small startup or developer building a game dApp that needs to connect to a custom web service you've built and deployed on AWS, then it would be better if you could simply call the service from within your Smart Contract without needing to go through an external network, obtain & consume LINK tokens etc.

If NEO can offer both options to developers, I believe it will greatly help adoption from developers, individuals, startups all the way to big enterprises, because it covers all bases from the quick, simple & easy, through to high value scenarios where a very high level of security and decentralization is required

[longfeiWan9]

I like the idea that @pappas999 proposed.

Let Chainlink team to handle the complex Oracle service which takes decentralization, security and efficiency into account, Because that will require lots of effort, expertise and partnerships.

Then NEO could just provide Simple Internet Access to developers or projects without worrying decentralization , Byzantine nodes or malicious users, etc. It is just a simple web access service or resource developers want to use and access. I thought this is also the original propose.

[igormcoelho]

Then NEO could just provide Simple Internet Access to developers or projects

That's what this proposal is about @wanglongfei88 😂 it's challenging to offer such a simple service. My proposal tries to made this as simple as possible, to be honest.

[corck]

A use case that I (us from Novem) would implement right away when we have oracles.

Autoconversion of tokens. At Novem we give 2% discount on purchase of our NNN token if its paid with the NVM token. Both prices fluctuate, NNN tokens relate to the Gold price, NVM is market price on exchanges.
So I would want our NNN purchasers be able to send NVM tokens to my conversion smart contract. The oracle then gets the current price for the NVM (e.g Coinmarketcap API) then the NNN price (from a NOVEM API). It then calculates the exchange rate with the additional discount and converts it right away.

There are a few more use cases like a savings plan, auto portfolio allocations (like having a BTC / Gold portfolio, that auto adjusts to keep a certain exposure to both assets), that comes to my mind, if we can get external pricing.

[doubiliu]

Plan A:
first,we need define 2 transaction type:
Request transaction: Commit=(src,t,q,flag，agg），src represent url，it is a set，SRC={SRC1、SRC2、SRCN},t represent timeout，q represent parameter，flag represent dynamica/static data,
agg represent aggregation contract
Response transaction： Commit=(reqID,answer,log），reqID represent RequestID,answer represent result data，log represent oracle node performance.

[doubiliu]

Whole system will be consist of two components :On-chain components and Off-chain components

[doubiliu]

Work Flow
1.User use client to make a Request transaction and delivery it into p2p network of Neo.
2.Oracle Plugin will store this transaction into a special memory pool(Oracle pool).All transactions in
this pool will not be chosen as candidate transactions by CN nodes before their corresponding Response transactions appear.
3.Oracle node running a Oracle Core application will monitor Oracle pool.When there is a Request transaction
in the pool,Oracle core will catch this transaction and invoke corresponding API to get result.
4.Oracle node obtain data and execute different consensus or aggregation task according the flag of
Request:
i)If flag represent static data,Oracle node will broadcast its result to other oracle nodes and do threshold signatures.
ii)If flag represent dynamic data,Oracle node will broadcast its result to other oracle nodes ,get corresponding aggregation contract from block chain and execute it in a indepent VM.
Theoretically,the aggregation result of each Oracle node is identical.And then,they will do threshold signatures.
5.After the threshold signature or consensus is successfully executed, the data is encapsulated into a Response transaction and deliveried into the Neo's P2P network by Oracle node.
6. Response transaction will be stored into Oracle pool and combined with the Request transaction into a transaction pair by Oracle Plugin.
7.Neo CN nodes will choose transaction pair as candidate transaction to generate block.
8.It is the end of a external API access process.

[doubiliu]

On-chain component
On-chain component consists of Oracle Pool、Oracle Contract Group and Oracle Plugin.
It mainly responsible for request/response cache, data integration, and data persisting on the chain.
Oracle Pool： A memory pool for storing Request/Response transaction.It can be implited by Map + Queue.The key of Map represent Request transaction.And the Value of Map represent Response transaction. The size of Queue will be limited to avoid overflow.
Oracle Plugin： It reads or stores transactions from Oracle pool,and provides them to consensus module.It also can load aggregation contract script on chain.
Oracle Contracts Group：a group of oracle smart contract
Aggregation contract:aggregate data.It can be defined by user.
Reward contract:assign reward to Oracle node according its data deviation value

Off-chain component
Oracle nodes running Oracle core application form Neo's decentralized Oracle chain system.They will monitor block chain,catch request and get data from corresponding external API.And then,they will execute aggregation contract to aggregate data and delivery the result as a Response transaction.

Work flow:

The first stage:

1. Get result data DATAi from external APi.
   2.Randomly generate the encryption key Keyi, and encrypt the hash of the result data DATAi
   to generate Ei
   3.Broadcast MessageType1=（src，Ei）
   4.Do threshold signatures to generate Eblock,
   5.Broadcast MessageType2=（src，Keyi）, dnd decrypt the Eblock by itself, get the decrypted block
   Dblock
   The second stage:
   1.Broadcast MessageType3=（src，DATAi）,and match the hash in Dblock.
   2.Collect enough DATAi to get the Block
   3.Execute aggregation contract to get PreAnswer,PreAnswer=Agg(DATA0,DATA1,...,DATAi).
   4.Do threshold signatures to generate Answer.

[doubiliu]

Advantages: 1) Synchronization
Disadvantages: 1) Failure to execute the transaction will result in Oracle nodes working in vain and not getting money.
2) Neo nodes need to be changed to add Oracle transaction memory pool
3) There is a large number of unavailable transaction attack risks, breaking through Oracle memory pool and Oracle node load

[doubiliu]

We also design a traditional plan

[doubiliu]

Plan B:
Architectural Overview
Whole system will consist of two components :On-chain components and Off-chain components

[doubiliu]

[doubiliu]

Work Flow
1.User use User Contract to invoke Task Contract to make a Request.
Oracle node running a Oracle Core application will monitor Task Contract to catch the Request and invoke corresponding external API to get result.
2.Oracle node obtain data and execute different consensus or aggregation task according the flag of
Request:
i)If flag represent static data,Oracle node will broadcast its result to other oracle nodes and do threshold signatures.
ii)If flag represent dynamic data,Oracle node will broadcast its result to other oracle nodes ,get corresponding aggregation contract from block chain and execute it in a indepent VM.
Theoretically,the aggregation result of each Oracle node is identical.And then,they will do threshold signatures.
3.After the threshold signature or consensus is successfully executed, the data will be encapsulated into a Response.Oracle node will invoke Task Contract to record result.
4.It is the end of a external API access process.

[doubiliu]

It is a asynchronous model
Advantages: 1) Oracle can get a certain amount of money from the transaction chain whether the access fails or not.
2) Neo nodes need not be changed.
Disadvantages: 1) Even if the transaction is successful, it does not necessarily mean that it can be accessed successfully.

[doubiliu]

And we also need to design a Incentive model

[doubiliu]

My proposal is only a draft, and there are still many details to discuss.

[doubiliu]

Maybe we can discuss them on face-to-face meeting next week

[igormcoelho]

I like the two approaches @doubiliu, let's discuss them soon, and make a fine report for the community ;)

[doubiliu]

@igormcoelho I am looking forward😄

[EdgeDLT]

Both are good concepts to discuss. I think we should also discuss penalties in addition to incentives; it is important to have a way to keep nodes honest. Perhaps a quantity of NEO that should be staked by Oracle nodes or something.

[shargon]

We will present our proposal soon, completely different to the proposal neo-project/proposals#78

[shargon]

I think we should also discuss penalties in addition to incentives; it is important to have a way to keep nodes honest

Maybe the Oracle is honest but the server is cheating... Very hard to decide this... everytime you should think in random.org

[doubiliu]

@shargon yeah, it is very important.We collect some key trust problems of designing a distributed Oracle system. And we have design a incentives proposal.We need discuss carefully

[igormcoelho]

@shargon the text you made on proposals is still very good, perhaps this should be re-proposed on specification repo... as soon as we agree on basic oracle features.

[shargon]

The problem with this text is that was proposed for delegate all the work to the current CN and we thought now that we need other nodes, Oracle nodes. without touch the current consensus mechanism.

[vncoelho]

It was great discussions yesterday. Thanks to @shargon and @belane and @neo-project/ngd-shanghai for teaching the options.

Congratulations for the great job and all the design.

I agree with the comment of @igormcoelho here #967 (comment) (OPTION A), which is also the proposal @shargon and @belane are supporting, as well as a need of the guys from NEO-FS.

The incentives are still not clear but sure some good direction will soon appear.

As emphasized, the block finality is a good thing we have and should be used in our favor, providing a one-step on-chain solution with off-chain information sharing.

[vncoelho]

Following this line of reasoning,

I think that 2 consensus are not necessary for the Oracles to work properly.

Perhaps, the following flow could be considered:

• Users can propose to be an oracle node;
• For being a valid Oracle Node it should have 2f+1 votes from the CN;
• Oracle Pending TXs goes to the set of oracle nodes;
• Speaker collects responses and as soon as an agreement of 2f+1 from the available oracles is available it proposes the TX into the next block (along with the hash it agrees for that information);
• Backups keep the same validation scheme.

[pappas999]

What you guys are proposing sounds great, but it is extremely similar to what Chainlink have been developing for the past 2 years: Decentralized Oracle network, consisting of on and off chain parts, staking, penalties, threshold signatures, consensus/aggregation on results etc. If this is the case, wouldn't it make more sense to just integrate to that, and develop some form of atomic swap (GAS for LINK) for NEO Smart Contracts to be able to use and directly pay for data requests with GAS? Only suggesting this because their team has been in development on all this stuff for 2 years, so it requires great effort to achieve.

Having said that, if you do want to replicate that sort of functionality and have it baked into the NEO protocol that's also great, it may pay to have a read of the chainlink white paper for some ideas (or check out their github)

I have a question though. From the description of option A), it sounds like any Oracle Node can take a request from the Oracle pool and request data from the API and generate the response back? Or will it be like Chainlink where specific Oracle nodes are setup to obtain data from specific APIs? eg Alice has a node that obtains weather data from weather.com, Bob has an Oracle Node that gets data from accuweather.com. If this is a more 'general' Oracle implementation where any Oracle can obtain the results from the APIs due to them all being public and open etc, then how do we determine which Oracle Node in the available pool gets to make the request (and get the reward)?

[eryeer]

@pappas999 Our design is the former one, every Oracle node will request the same public API. The API url is specified by transaction. Any oracle node that requests the API and get the consistent result approved by consensus should be considered to contribute to the result of the request, and they should be rewarded equally.

[pappas999]

@pappas999 Our design is the former one, every Oracle node will request the same public API. The API url is specified by transaction. Any oracle node that requests the API and get the consistent result approved by consensus should be considered to contribute to the result of the request, and they should be rewarded equally.

Thanks, that makes sense now. So the use case for the NEO Oracle network will be publicly accessible static or dynamic web services and URLs that return JSON data etc which can then be parsed and aggregated into a result that will be hashed and stored in the block.

I have some thoughts to add for incentives/penalties and considerations:

Incentives:
The obvious way to incentivize Oracle nodes is for them to be paid in GAS for fulfilling requests. There are multiple payment structures/schemes possible:
1 - static fee per request, eg 1 GAS per request, split among all Oracle nodes that contribute to the response
2 - Let each Oracle node define their required fee, then ensure data requestor/Smart Contract has enough GAS for the request for x Oracle nodes to be able to fulfill their request. Perhaps x can be requested by the Smart Contract, giving them a choice in how much decentralization/aggregation they want. ie for something small and cheap, x can be 1 and only 1 Oracle node is used, or for higher value contracts that require better security and decentralization, x can be 20 so 20 nodes used to validate data.
3 - Have the fee calculated based on multiple things, a) how many oracle nodes are used (as per point 2 above), as well as b) size of the response/data obtained.
4 - Other?

Also, seeing as the URLs being accessed are public, there should be good incentive for NEO holders to be Oracle nodes if they wish. This plays in well to the token economics of the NEO ecosystem, and would allow them to collect more GAS from Oracle transaction fees. The only potential downside to this, is you don't want NEO holders with poor servers/internet connections being critical Oracle nodes in the network. Chainlink is implementing a 'marketplace' to combat this, where nodes will have reputations, and to be verified as a high quality data provider, you need to show solid uptime and response times for a certain length of time. Perhaps Consensus Nodes or NEO holders can vote in/out Oracle nodes somehow based on their performance.

Penalties:
This would require a certain amount of NEO tokens to be used/staked by Oracle nodes as collateral to ensure they give back correct responses. If all Oracle nodes are to be treated equally, then perhaps a static amount of NEO can be specified as needing to be staked? Then governance needs to be defined as to what constitutes 'bad behavior', and what the process is of how an Oracle node gets its stake taken away.

data points/URLs: Need to be careful how many Oracle nodes will be accessing these URLs at a given time. Eg Will the URL be able to handle getting 100 GET requests within a couple seconds? I'm not an expert on web servers and requests/scaling so this point may be not applicable.

[doubiliu]

@pappas999 I like point2 in Incentives.So I want to add concept of group.
Group
Any node that enables Oracle functionality can become a potential Oracle service provider in Neo network. These nodes can be aggregated into one group according to certain rules and wishes, and the group provides Oracle services to users. The most authoritative group is a general-purpose group voted by the Neo consensus node, which can provide a more reliable Oracle service.
User can decide which group to use. It is recommended to use generic groups, but in some special cases, such as special authorization, restricted access, etc., other groups can be used, but they are closer to centralization.

[doubiliu]

And Penalties is very difficult.Since there is not any evidence can be trusted to judge a node's behavior

[joshuaellul]

Would you be interested to see how this approach can help?
https://joshua-ellul.medium.com/contrary-to-widespread-misconception-blockchain-dlts-and-smart-contracts-can-make-calls-to-94d864415ca7

Only 1 node needs to make an oracle call. The rest can verify it, if the oracle has signed it's response. In our work above we show that external calls from Blockchain systems are possible, contrary to widespread misconception.