Oracle policy contract (#1445)

* first commit * format * Simplify code * add UT and add some feature * Fix bug * Add summary and fix bug * format * format * little change * little change * little change * change Fee & Fix bug * Optimize * add config class * FiX UT * Format * Fix UT * Fix format and Optimize * Add some UT * Fix bug and add UT * Update src/neo/SmartContract/Native/Oracle/OraclePolicyContract.cs Co-Authored-By: Luchuan <luchuan@ngd.neo.org> * little change * format * Update src/neo/SmartContract/Native/Oracle/OraclePolicyContract.cs Co-Authored-By: Shargon <shargon@gmail.com> * add check * change validator * Add double initialization check * add UT * Add UT * change UT * fill UT * up ut and fix NeoToken.GetValidators * remove console * reset NEO.getValidators * fix GetOracleValidators * Rename variable * Update src/neo/SmartContract/Native/Oracle/OracleContract.cs Co-Authored-By: Erik Zhang <erik@neo.org> * Update src/neo/SmartContract/Native/Oracle/OracleContract.cs Co-Authored-By: Erik Zhang <erik@neo.org> * Fix bug * change logic * Fix name * change to const * Add double initialization check and fix ut * optimize Co-authored-by: Shargon <shargon@gmail.com> Co-authored-by: Luchuan <luchuan@ngd.neo.org> Co-authored-by: Luchuan <luchuan@neo.org> Co-authored-by: Erik Zhang <erik@neo.org> Oracle Syscall (#1399) * Plugins from List to array * Move false to init * Fix UT * Refactor * UT * Change HTTP to HTTPS * UT Syscall * Remove Version * Remove HTTP from the syscall name * Check schema * Rename * Update src/neo/Oracle/OracleRequest.cs Co-Authored-By: Erik Zhang <erik@neo.org> * Fix abstract * Rename * Remove Body * Change order of members * Rename Neo.Oracle.Get Co-authored-by: Vitor Nazário Coelho <vncoelho@gmail.com> Co-authored-by: erikzhang <erik@neo.org> Co-authored-by: Belane <srbelane@gmail.com> Oracle Syscall (#1399) * Plugins from List to array * Move false to init * Fix UT * Refactor * UT * Change HTTP to HTTPS * UT Syscall * Remove Version * Remove HTTP from the syscall name * Check schema * Rename * Update src/neo/Oracle/OracleRequest.cs Co-Authored-By: Erik Zhang <erik@neo.org> * Fix abstract * Rename * Remove Body * Change order of members * Rename Neo.Oracle.Get Co-authored-by: Vitor Nazário Coelho <vncoelho@gmail.com> Co-authored-by: erikzhang <erik@neo.org> Co-authored-by: Belane <srbelane@gmail.com> Rename OracleResult->OracleResponse (#1516) Removed hash from the response Oracle Syscall (#1399) * Plugins from List to array * Move false to init * Fix UT * Refactor * UT * Change HTTP to HTTPS * UT Syscall * Remove Version * Remove HTTP from the syscall name * Check schema * Rename * Update src/neo/Oracle/OracleRequest.cs Co-Authored-By: Erik Zhang <erik@neo.org> * Fix abstract * Rename * Remove Body * Change order of members * Rename Neo.Oracle.Get Co-authored-by: Vitor Nazário Coelho <vncoelho@gmail.com> Co-authored-by: erikzhang <erik@neo.org> Co-authored-by: Belane <srbelane@gmail.com> Rename OracleResult->OracleResponse (#1516) Removed hash from the response Oracle Service (#1517) * Oracle Service * Fix fee * Optimize load configuration * Fix akka message * Optimize * dotnet format * Fix sort * Oracle Service * Fix fee * Optimize load configuration * Fix akka message * Optimize * dotnet format * Fix sort * Apply rename * Small changes * Add UT * Fix Start Stop * dotnet-format * Migrate to HTTPS test * Change AKKA message * Tommo recomendations * Add comunication between OracleTx and UserTx * Advance of signature * Fix UT * dotnet-format * Advance of signature Fix UT dotnet-format * Fix UT * Remove duplica method * Reduce timeout time * Add log for fix UT * Add filters * Change port to 443 * Oracle Service * Fix fee * Optimize load configuration * Fix akka message * Optimize * dotnet format * Fix sort * Oracle Service * Fix fee * Optimize load configuration * Fix akka message * Optimize * dotnet format * Fix sort * Apply rename * Small changes * Add UT * Fix Start Stop * dotnet-format * Migrate to HTTPS test * Change AKKA message * Tommo recomendations * Add comunication between OracleTx and UserTx * Advance of signature * Fix UT * dotnet-format * Fix UT * Remove duplica method * Reduce timeout time * Add log for fix UT * Add filters * Change port to 443 * Merge changes * Change SSL Protocol * Merge remote-tracking branch 'neo-project/oracle-service' into oracle-service * Add UT certificate * Clean code * Add UT fix GetOracleMultiSigAddress (#1538) * fix GetOracleMultiSigAddress * First lowercase Co-authored-by: Shargon <shargon@gmail.com> Oracle: Tx comunication (#1540) * Tx comunication * Some fixes Transaction Version (#1539) Oracle: Allow wallets to add assert for the oracle response (#1541) * Add assert for each oracle request * Change to goto * Improve wallet asserts * Optimize * Fix URL type * dotnet format * Oracle: Tx comunication (#1540) * Tx comunication * Some fixes * Optimize OracleExecutionCache * Add assert for each oracle request * Change to goto * Improve wallet asserts * Optimize * Fix URL type * dotnet format * Optimize OracleExecutionCache * MakeTransaction UT Oracle: Set the right Tx version (#1542) * Set the right Tx version * Remove Error types from TX and VM * Neo.Oracle.Get now only return the result or null Oracle: P2P Response Signature Message (#1553) * Oracle Response Signature P2P * caches * rename private fields * remove comments * remove more comments and fix size * Clean code Co-authored-by: Shargon <shargon@gmail.com> Oracle: OracleService (#1555) * Start pool * Prepare for neo-node * Fix double call * Prepare pool * Fix * Join oracle pool with oracle service * dotnet-format * Try to fix UT * UT pass * Fix UT * Fix p2p message * Unify collections * Relay p2p message * Clean code and fixes * Send tx to OracleService * RequestTx will wait for ResponseTx * Rename * Remove supervisor * Allow to put request, and response in the same block * Check the sender of OracleResponses * Remove OracleResponse message in OracleService * Organize code * Fix typo * Remove task count TODO * Remove Thread-safe TODO * Remove TODO * ResponseItem changes * Read the oracle contract on my response Receive oraclePayload * Clean code * Add Stop message * Save only one response for PublicKey/RequestTx Improve sorting response pool * Improve sort * Group sort methods in the same region * Ask for the request TX if I don't have it * Reorder code * Rename * First oracle TX Oracle: Remove some TODOs (#1574) * Remove some todos * Remove one method * Reuse ManualWitness * dotnet format * Allow both tx in the same block without change order or fee * Fix p2p message serialization * Fix p2p * Fixes and IInventory * Fix akka message * Fix ut Fix conflicts Remove double call
neo-project · Apr 17, 2020 · 3a98905 · 3a98905
1 parent 0604b24
commit 3a98905
Show file tree

Hide file tree

Showing 50 changed files with 4,388 additions and 97 deletions.
diff --git a/.gitignore b/.gitignore
@@ -192,6 +192,7 @@ ClientBin/
 *.dbmdl
 *.dbproj.schemaview
 *.pfx
+!UT-cert.pfx
 *.publishsettings
 node_modules/
 orleans.codegen.cs

diff --git a/src/neo/Consensus/ConsensusContext.cs b/src/neo/Consensus/ConsensusContext.cs
@@ -283,7 +283,7 @@ public ConsensusPayload MakePrepareRequest()
             Span<byte> buffer = stackalloc byte[sizeof(ulong)];
             random.NextBytes(buffer);
             Block.ConsensusData.Nonce = BitConverter.ToUInt64(buffer);
-            EnsureMaxBlockSize(Blockchain.Singleton.MemPool.GetSortedVerifiedTransactions());
+            EnsureMaxBlockSize(Blockchain.Singleton.MemPool.GetSortedVerifiedTransactions(Snapshot));
             Block.Timestamp = Math.Max(TimeProvider.Current.UtcNow.ToTimestampMS(), PrevHeader.Timestamp + 1);
 
             return PreparationPayloads[MyIndex] = MakeSignedPayload(new PrepareRequest

diff --git a/src/neo/Consensus/ConsensusService.cs b/src/neo/Consensus/ConsensusService.cs
@@ -443,7 +443,7 @@ private void OnPrepareRequestReceived(ConsensusPayload payload, PrepareRequest m
                 return;
             }
 
-            Dictionary<UInt256, Transaction> mempoolVerified = Blockchain.Singleton.MemPool.GetVerifiedTransactions().ToDictionary(p => p.Hash);
+            Dictionary<UInt256, Transaction> mempoolVerified = Blockchain.Singleton.MemPool.GetVerifiedTransactions(context.Snapshot).ToDictionary(p => p.Hash);
             List<Transaction> unverified = new List<Transaction>();
             foreach (UInt256 hash in context.TransactionHashes)
             {

diff --git a/src/neo/Ledger/Blockchain.cs b/src/neo/Ledger/Blockchain.cs
@@ -64,7 +64,7 @@ public class RelayResult { public IInventory Inventory; public VerifyResult Resu
         private uint stored_header_count = 0;
         private readonly Dictionary<UInt256, Block> block_cache = new Dictionary<UInt256, Block>();
         private readonly Dictionary<uint, LinkedList<Block>> block_cache_unverified = new Dictionary<uint, LinkedList<Block>>();
-        internal readonly RelayCache ConsensusRelayCache = new RelayCache(100);
+        internal readonly RelayCache RelayCache = new RelayCache(100);
         private SnapshotView currentSnapshot;
 
         public IStore Store { get; }
@@ -301,6 +301,7 @@ private void OnInventory(IInventory inventory, bool relay = true)
                     Block block => OnNewBlock(block),
                     Transaction transaction => OnNewTransaction(transaction),
                     ConsensusPayload payload => OnNewConsensus(payload),
+                    OraclePayload payload => OnNewOracle(payload),
                     _ => VerifyResult.Unknown
                 }
             };
@@ -393,7 +394,15 @@ private VerifyResult OnNewConsensus(ConsensusPayload payload)
         {
             if (!payload.Verify(currentSnapshot)) return VerifyResult.Invalid;
             system.Consensus?.Tell(payload);
-            ConsensusRelayCache.Add(payload);
+            RelayCache.Add(payload);
+            return VerifyResult.Succeed;
+        }
+
+        private VerifyResult OnNewOracle(OraclePayload payload)
+        {
+            if (!payload.Verify(currentSnapshot)) return VerifyResult.Invalid;
+            system.Oracle?.Tell(payload);
+            RelayCache.Add(payload);
             return VerifyResult.Succeed;
         }
 
@@ -425,6 +434,14 @@ private VerifyResult OnNewTransaction(Transaction transaction)
             VerifyResult reason = transaction.Verify(currentSnapshot, MemPool.SendersFeeMonitor.GetSenderFee(transaction.Sender));
             if (reason != VerifyResult.Succeed) return reason;
             if (!MemPool.TryAdd(transaction.Hash, transaction)) return VerifyResult.OutOfMemory;
+
+            if (transaction.Version == TransactionVersion.OracleRequest)
+            {
+                // Oracle Service only need the OracleRequests
+
+                system.Oracle?.Tell(transaction);
+            }
+
             return VerifyResult.Succeed;
         }
 
@@ -463,6 +480,9 @@ protected override void OnReceive(object message)
                 case ConsensusPayload payload:
                     OnInventory(payload);
                     break;
+                case OraclePayload oracle:
+                    OnInventory(oracle);
+                    break;
                 case Idle _:
                     if (MemPool.ReVerifyTopUnverifiedTransactionsIfNeeded(MaxTxToReverifyPerIdle, currentSnapshot))
                         Self.Tell(Idle.Instance, ActorRefs.NoSender);

diff --git a/src/neo/Ledger/MemoryPool.cs b/src/neo/Ledger/MemoryPool.cs
@@ -27,7 +27,6 @@ public class MemoryPool : IReadOnlyCollection<Transaction>
 
         private readonly NeoSystem _system;
 
-        //
         /// <summary>
         /// Guarantees consistency of the pool data structures.
         ///
@@ -169,17 +168,35 @@ public IEnumerator<Transaction> GetEnumerator()
 
         IEnumerator IEnumerable.GetEnumerator() => GetEnumerator();
 
-        public IEnumerable<Transaction> GetVerifiedTransactions()
+        public IEnumerable<Transaction> GetVerifiedTransactions(StoreView snapshot)
         {
+            Transaction[] ret;
+            var oracle = new PoolItem.DelayState();
+
             _txRwLock.EnterReadLock();
             try
             {
-                return _unsortedTransactions.Select(p => p.Value.Tx).ToArray();
+                ret = _unsortedTransactions
+                    .Where(u => u.Value.IsReady(snapshot, oracle))
+                    .Select(p => p.Value.Tx)
+                    .ToArray();
             }
             finally
             {
                 _txRwLock.ExitReadLock();
             }
+
+            // Fetch transactions
+
+            foreach (var tx in ret)
+            {
+                yield return tx;
+            }
+            foreach (var delayed in oracle.Delayed)
+            {
+                if (oracle.Allowed.Contains(delayed.Hash))
+                    yield return delayed;
+            }
         }
 
         public void GetVerifiedAndUnverifiedTransactions(out IEnumerable<Transaction> verifiedTransactions,
@@ -197,17 +214,36 @@ public IEnumerable<Transaction> GetVerifiedTransactions()
             }
         }
 
-        public IEnumerable<Transaction> GetSortedVerifiedTransactions()
+        public IEnumerable<Transaction> GetSortedVerifiedTransactions(StoreView snapshot)
         {
+            Transaction[] ret;
+            var oracle = new PoolItem.DelayState();
+
             _txRwLock.EnterReadLock();
             try
             {
-                return _sortedTransactions.Reverse().Select(p => p.Tx).ToArray();
+                ret = _sortedTransactions
+                    .Reverse()
+                    .Where(u => u.IsReady(snapshot, oracle))
+                    .Select(p => p.Tx)
+                    .ToArray();
             }
             finally
             {
                 _txRwLock.ExitReadLock();
             }
+
+            // Fetch transactions
+
+            foreach (var tx in ret)
+            {
+                yield return tx;
+            }
+            foreach (var delayed in oracle.Delayed)
+            {
+                if (oracle.Allowed.Contains(delayed.Hash))
+                    yield return delayed;
+            }
         }
 
         [MethodImpl(MethodImplOptions.AggressiveInlining)]
@@ -239,7 +275,7 @@ private PoolItem GetLowestFeeTransaction(out Dictionary<UInt256, PoolItem> unsor
             }
             finally
             {
-                unsortedTxPool = Object.ReferenceEquals(sortedPool, _unverifiedSortedTransactions)
+                unsortedTxPool = ReferenceEquals(sortedPool, _unverifiedSortedTransactions)
                    ? _unverifiedTransactions : _unsortedTransactions;
             }
         }

diff --git a/src/neo/Ledger/PoolItem.cs b/src/neo/Ledger/PoolItem.cs
@@ -1,5 +1,8 @@
 using Neo.Network.P2P.Payloads;
+using Neo.Persistence;
+using Neo.SmartContract.Native;
 using System;
+using System.Collections.Generic;
 
 namespace Neo.Ledger
 {
@@ -33,6 +36,52 @@ internal PoolItem(Transaction tx)
             LastBroadcastTimestamp = Timestamp;
         }
 
+        internal class DelayState
+        {
+            public HashSet<UInt256> Allowed = new HashSet<UInt256>();
+            public HashSet<Transaction> Delayed = new HashSet<Transaction>();
+        }
+
+        public bool IsReady(StoreView snapshot, DelayState state)
+        {
+            switch (Tx.Version)
+            {
+                case TransactionVersion.OracleRequest:
+                    {
+                        if (state.Allowed.Remove(Tx.Hash))
+                        {
+                            // The response was already fetched, we can put request and response in the same block
+
+                            return true;
+                        }
+                        else
+                        {
+                            if (NativeContract.Oracle.ContainsResponse(snapshot, Tx.Hash))
+                            {
+                                // The response it's waiting to be consumed (block+n)
+
+                                return true;
+                            }
+                            else
+                            {
+                                // If the response it's in the pool it's located after the request
+                                // We save the request in order to put after the response
+
+                                state.Delayed.Add(Tx);
+                                return false;
+                            }
+                        }
+                    }
+                case TransactionVersion.OracleResponse:
+                    {
+                        state.Allowed.Add(Tx.OracleRequestTx);
+                        break;
+                    }
+            }
+
+            return true;
+        }
+
         public int CompareTo(Transaction otherTx)
         {
             if (otherTx == null) return 1;

diff --git a/src/neo/Ledger/SortedBlockingCollection.cs b/src/neo/Ledger/SortedBlockingCollection.cs
@@ -0,0 +1,84 @@
+using System.Collections.Concurrent;
+using System.Collections.Generic;
+using System.Threading;
+
+namespace Neo.Ledger
+{
+    public class SortedBlockingCollection<TKey, TValue>
+    {
+        /// <summary>
+        /// _oracleTasks will consume from this pool
+        /// </summary>
+        private readonly BlockingCollection<TValue> _asyncPool = new BlockingCollection<TValue>();
+
+        /// <summary>
+        /// Queue
+        /// </summary>
+        private readonly SortedConcurrentDictionary<TKey, TValue> _queue;
+
+        /// <summary>
+        /// Constructor
+        /// </summary>
+        /// <param name="comparer">Comparer</param>
+        /// <param name="capacity">Capacity</param>
+        public SortedBlockingCollection(IComparer<KeyValuePair<TKey, TValue>> comparer, int capacity)
+        {
+            _queue = new SortedConcurrentDictionary<TKey, TValue>(comparer, capacity);
+        }
+
+        /// <summary>
+        /// Add entry
+        /// </summary>
+        /// <param name="key">Key</param>
+        /// <param name="value">Value</param>
+        public void Add(TKey key, TValue value)
+        {
+            if (_queue.TryAdd(key, value) && _asyncPool.Count <= 0)
+            {
+                Pop();
+            }
+        }
+
+        /// <summary>
+        /// Clear
+        /// </summary>
+        public void Clear()
+        {
+            _queue.Clear();
+
+            while (_asyncPool.Count > 0)
+            {
+                _asyncPool.TryTake(out _);
+            }
+        }
+
+        /// <summary>
+        /// Get consuming enumerable
+        /// </summary>
+        /// <param name="token">Token</param>
+        public IEnumerable<TValue> GetConsumingEnumerable(CancellationToken token)
+        {
+            foreach (var entry in _asyncPool.GetConsumingEnumerable(token))
+            {
+                // Prepare other item in _asyncPool
+
+                Pop();
+
+                // Iterate items
+
+                yield return entry;
+            }
+        }
+
+        /// <summary>
+        /// Move one item from the sorted queue to _asyncPool, this will ensure that the threads process the entries according to the priority
+        /// </summary>
+        private void Pop()
+        {
+            if (_queue.TryPop(out var entry))
+            {
+                _asyncPool.Add(entry);
+            }
+        }
+    }
+}